[Part 3] Your first steps in Computer Vision on AWS: deploying models for an ASL recognition app

This is the third and final part of the series "Your first steps in Computer Vision on AWS". If you are just starting, this tutorial will guide you step by step from the initial dataset to a deployed app.

You can read the first part on ASL dataset + preprocessing and the second part on a custom CNN training.

What we are trying to build

While most ML tutorials end with model training and validation, we will take things one step further and deploy our models in an app. This is the most interesting and exciting part! We will make our models live!

Here is what the AWS architecture will look like:

Here's a quick overview of the solution:

The best way to test models is to take photos using a mobile phone. A simple static, serverless ASL Alphabet Recognition web application is hosted on S3 and accessed via a CloudFront URL. To call the models for predictions, we will use API Gateway. A Lambda event function will preprocess the uploaded photo and send it to a serverless SageMaker inference endpoint. The endpoint will be created using a container stored in ECR.

Most of the AWS infrastructure will be created automatically using CloudFormation (not shown on the architecture image). This will make the setup reproducible. However, we will first need to create the ECR repositories, build and push the SageMaker and Lambda container images, and upload the model archive to S3. The website files will also be uploaded to S3 manually after the stack is deployed.

❓ Why serverless endpoint?
Our use case will have idle periods between traffic spurts and can tolerate cold starts. Serverless endpoints automatically launch compute resources and scale them in and out (even down to 0) depending on traffic, eliminating the need to choose instance types or manage scaling policies. With a pay-per-use model, Serverless Inference is a cost-effective option in our infrequent traffic pattern. The serverless endpoint supports a maximum memory size of 6144 MB (6 GB), which is more than enough for our three models.

❓ What would be the app cost?
I spent less than $5 on the application. The serverless endpoint was running for around a week, but the cost was minimal. I spent more on the ml.g5.xlarge instance in the previous part.

Prepare models for inference

⚠️ For this step, you can use your local environment or a SageMaker notebook instance (ml.t3.medium is more than enough).

Models trained in the previous part are ready: put raw.keras, filtered.keras, and edges.keras into models/
You will also need to add the class_names.json file to the models/ directory (copy/download it from the GitHub repo).
It contains the exact class mapping used during training.

From inside models/ directory run the command:

tar -czvf asl-models.tar.gz \ raw.keras \ filtered.keras \ edges.keras \ class_names.json 

You will get the archive asl-models.tar.gz. Upload it to S3:

aws s3 cp asl-models.tar.gz \ s3://YOUR-BUCKET/asl-app/models/asl-models.tar.gz 

⚠️ Don't forget to replace the placeholder YOUR-BUCKET and put your S3 bucket name!

Once the archive has been uploaded to S3, we will use its S3 URI in CloudFormation.

s3://YOUR-BUCKET/asl-app/models/asl-models.tar.gz 

Build and push SageMaker inference container to ECR

⚠️ For this step, you can use your local environment with AWS CLI (How to install/update AWS CLI?) or a SageMaker notebook instance (ml.t3.medium is more than enough).

⚠️ Copy/download inference.py, requirements.txt, and Dockerfile files from the GitHub repo. They should sit inside the sagemaker directory (same structure as in the repo).

The code in inference.py is pretty simple: its main job is to handle inference for all three models.

Now, create an ECR repository:

aws ecr create-repository \ --repository-name asl-sagemaker-inference 

Get your account ID:

aws sts get-caller-identity \ --query Account \ --output text 

Set variables:

export AWS_REGION=YOUR-REGION export AWS_ACCOUNT_ID=$(aws sts get-caller-identity \ --query Account \ --output text) export ECR_URI=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com 

Login:

aws ecr get-login-password \ --region $AWS_REGION \ | docker login \ --username AWS \ --password-stdin $ECR_URI 

Build and tag:

cd sagemaker docker build \ -t asl-sagemaker-inference:latest . docker tag \ asl-sagemaker-inference:latest \ $ECR_URI/asl-sagemaker-inference:latest 

And finally push:

docker push \ $ECR_URI/asl-sagemaker-inference:latest 

⚠️ Make sure that your role or identity has the required permissions to do the job. Required actions: ecr:InitiateLayerUpload, ecr:UploadLayerPart, ecr:CompleteLayerUpload, ecr:BatchCheckLayerAvailability, and ecr:PutImage.

You will use in CloudFormation:
ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com/asl-sagemaker-inference:latest

Build Lambda image

The main task of the Lambda function will be to accept an image from a phone, crop it to the center at 200×200 pixels, do the required preprocessing for the Gaussian-filtered model and Canny edge model and finally obtain predictions from all three models by invoking the SageMaker endpoint. For each model, it will get the top 3 predictions with their probabilities.

⚠️ For this step, you can use your local environment with AWS CLI (How to install/update AWS CLI?) or a SageMaker notebook instance (ml.t3.medium is more than enough).

⚠️ Copy/download lambda_function.py, requirements.txt, and Dockerfile files from the GitHub repo. They should sit inside the lambda directory (same structure as in the repo).

Then, create a new ECR repository:

aws ecr create-repository \ --repository-name asl-preprocessing-lambda 

Get your account ID and set variables:

aws sts get-caller-identity \ --query Account \ --output text export AWS_REGION=YOUR-REGION export AWS_ACCOUNT_ID=$(aws sts get-caller-identity \ --query Account \ --output text) export ECR_URI=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com 

Login:

aws ecr get-login-password \ --region $AWS_REGION \ | docker login \ --username AWS \ --password-stdin $ECR_URI 

Build, tag, and push:

cd lambda docker buildx build \ --platform linux/amd64 \ --provenance=false \ -t asl-preprocessing-lambda:latest \ . docker tag asl-preprocessing-lambda:latest \ $ECR_URI/asl-preprocessing-lambda:latest docker push $ECR_URI/asl-preprocessing-lambda:latest 

You will use the Lambda image in CloudFormation:
ACCOUNT.dkr.ecr.REGION.amazonaws.com/asl-preprocessing-lambda:latest

Deploy the CloudFormation stack

⚠️ For this step, you can use your local environment with AWS CLI (How to install/update AWS CLI?) or a SageMaker notebook instance (ml.t3.medium is more than enough).

⚠️ Copy/download the cloudformation.yaml file from the GitHub repo.

This CloudFormation stack will create all the resources required for the serverless ASL Alphabet Recognition web application: CloudFront, S3 bucket, API Gateway, Lambda event function, SageMaker Serverless endpoint. As shown in the architecture diagram, CloudFront will route requests to /api/* to API Gateway, giving you a single public URL for the application.
Before deploying the stack, you will need to retrieve three values:

Let's deploy the stack:

aws cloudformation deploy \ --template-file cloudformation.yaml \ --stack-name asl-recognizer \ --capabilities CAPABILITY_IAM \ --parameter-overrides \ ModelS3Uri=s3://YOUR-BUCKET/asl-app/models/asl-models.tar.gz \ SageMakerImageUri=ACCOUNT.dkr.ecr.REGION.amazonaws.com/asl-sagemaker-inference:latest \ LambdaImageUri=ACCOUNT.dkr.ecr.REGION.amazonaws.com/asl-preprocessing-lambda:latest 

⚠️ Make sure that your role or identity has the required permissions to successfully deploy your CloudFormation stack. Required actions: cloudformation:DescribeStacks, s3:CreateBucket, s3:ListAllMyBuckets, s3:GetBucketLocation, and apigateway:POST.

It will take around 10-15 mins to complete. A good thing is that you can monitor stack events in the AWS CloudFormation Management Console:

Upload the website

⚠️ For this step, you can use your local environment with AWS CLI (How to install/update AWS CLI?) or a SageMaker notebook instance (ml.t3.medium is more than enough).

⚠️ Copy/download app.js, index.html, and styles.css files from the GitHub repo. They should sit inside the frontend directory (same structure as in the repo).

After CloudFormation finishes, retrieve the bucket:

aws cloudformation describe-stacks \ --stack-name asl-recognizer \ --query \ "Stacks[0].Outputs[?OutputKey=='FrontendBucket'].OutputValue" \ --output text 

The command will return a bucket name similar to asl-recognizer-frontend-12000000. Use this value to upload your website pages from the frontend directory to S3:

aws s3 sync frontend/ s3://asl-recognizer-frontend-12000000/ 

After this, retrieve the CloudFront URL:

aws cloudformation describe-stacks \ --stack-name asl-recognizer \ --query \ "Stacks[0].Outputs[?OutputKey=='WebsiteURL'].OutputValue" \ --output text 

You'll get the CloudFront URL (e.g. https://d1234567890abc.cloudfront.net). Paste it in the browser to access your application.
As you can see, the web application's UI is very simple. Its main purpose is to let us test the models and get predictions:

Again, the provided GitHub repo has all the required files for models inference preparation, SageMaker inference container creation, Lambda image creation, as well as website pages and the CloudFormation stack deployment. The repo structure is:

asl-web-app/ │ ├── models/ │ ├── raw.keras │ ├── filtered.keras │ ├── edges.keras │ └── class_names.json │ ├── sagemaker/ │ ├── inference.py │ ├── Dockerfile │ └── requirements.txt │ ├── lambda/ │ ├── lambda_function.py │ ├── Dockerfile │ └── requirements.txt │ ├── frontend/ │ ├── index.html │ ├── app.js │ └── styles.css │ └── cloudformation.yaml 

Test the app

The most exciting part begins! Use your phone to take photos and make predictions.
First, I tried several images from the asl_alphabet_test folder, which contains additional images provided alongside the training dataset.
All three models performed reasonably well, producing the correct class predictions:

However, you may notice that the models' predictions on your own photos are not always correct. I tried to show a 'W': the model trained on raw RGB returned 'R', Gaussian-filtered and Canny edge models predicted 'K':

All our models achieved more than 90% accuracy on our held-out test split. They also performed well on additional images from Kaggle's separate asl_alphabet_test folder, which were not included in the training, validation, or test splits. But when facing real-world images, models started to act weird.

This reveals an important lesson about ML models and their applications outside the collected dataset. There is a gap between a toy dataset and real images; and we didn't fully account for it. If you look closer at training images, they all share a similar background, same skin tone, same hand size. Lightning and focus do vary, but not drastically. What if the person signing in ASL has a darker skin tone? What if we use a green/blue/orange background?

In this project, we got too excited and proceeded with training using the data we had. And we are not alone. If you check the Kaggle dataset notebooks, many practitioners achieved almost 100% accuracy. The model works perfectly on the provided images, but its ability to generalize to real-world scenarios is not guaranteed.

📖 For the future, we need to be aware of data cascades: issues in data are 'cascaded' down the line, and the key strategy is to detect such data issues and work on them early on, before any model is trained. (Useful read: "Everyone wants to do the model work, not the data work": Data Cascades in High-Stakes AI).

📖 It is very possible that our models relied on shortcut learning (read Shortcut learning in deep neural networks). One way to investigate this is to use explainable AI methods, such as Grad-CAM. It can help visualize which parts of an image influence a CNN's predictions.

Conclusion

In this part, we deployed three CNN models to a single serverless SageMaker inference endpoint. We also used CloudFormation to provision the AWS resources needed to build a simple ASL Alphabet Recognition web application. Next, we tested the app with the test dataset images, as well as with our custom images. Finally, we found a gap between a toy Kaggle dataset and real-world conditions. The ability to overcome this gap is what distinguishes a simple proof of concept (POC) from a production-ready ML solution.