Overview
This is my first step into the Computer Vision MLOps Engineering field. I wanted to explore an MLOps subject that I might see as a senior Computer Vision or MLOps engineer:
Process a large set of images from COCO 2017 validation set(5000 images) as cheaply and quickly as possible on AWS. Images land in S3, queue up in SQS, and workers on ECS run them through YOLOv8m in batches, writing results to S3 and DynamoDB. The project is about throughput, cost and reliability for bulk work, not per-request latency.
Source Code
Goals
- Find the best batch size: the batch size on a T4 GPU (g4dn.xlarge) that gives the most images per second and the lowest cost per 1,000 images, and the point where bigger batches stop helping.
- Quantify GPU vs. CPU for bulk work: how much cheaper and faster a 5,000-image job is on a GPU worker than on a CPU worker.
- Measure an optimized runtime: what ONNX Runtime adds on top of batching, without losing accuracy.
- Build a pipeline you'd trust in production: it scales with queue depth, isolates bad images in a dead-letter queue, and can be re-run without processing any image twice.
Architecture Diagram
Headline results
- Best setup: pipelined PyTorch FP16 at batch 32 processes 85.7 images/s for $0.0017 per 1,000 images on one
g4dn.xlarge. Its mAP50-95 is 0.5021, against 0.5026 for FP32 (the published YOLOv8m value is 0.502). It's the GPU service's default. - GPU vs. CPU: about 30× faster and 20× cheaper per image than an unoptimized CPU baseline at a similar hourly price (
c6i.2xlarge: 2.8 images/s, $0.0339 per 1,000). - Pipelining beat batch size. Larger batches took FP32 from 6 to 27 images/s, but the GPU was still only 49% busy. Overlapping download/decode, inference and writes in separate threads doubled that to 52 images/s with the GPU 93% busy.
- TensorRT is the open issue. It has the fastest model execution (5.4 ms per image vs. 10 ms for PyTorch FP16) but the slowest end-to-end FP16 result (53.7 images/s), because image loading slows down while it runs.
Inference Pipeline Setups
One step at a time: the GPU (green) is busy only during each batch's infer.
Pipelined: each step runs in its own thread, so the GPU goes straight from one batch to the
next.
Results
| Setup | Precision | Batch size | Images/sec | Time for N images | Cost per 1,000 images | Avg GPU utilization | mAP50-95 | Failed images |
|---|---|---|---|---|---|---|---|---|
ECS workers, CPU (c6i.2xlarge, PyTorch, unoptimized) | FP32 | 1 | 2.8 | 29 min 57 s (5,000) | $0.0339 | — | 0.5026 | 0 |
ECS workers, GPU (g4dn.xlarge), PyTorch | FP32 | 1 | 6.0 | 14 min 00 s (5,000) | $0.0245 | 11% | 0.5026 | 0 |
ECS workers, GPU (g4dn.xlarge), PyTorch | FP32 | 8 | 18.3 | 4 min 33 s (5,000) | $0.0080 | 39% | 0.5026 | 0 |
ECS workers, GPU (g4dn.xlarge), PyTorch | FP32 | 16 | 22.4 | 3 min 43 s (5,000) | $0.0065 | 44% | 0.5026 | 0 |
ECS workers, GPU (g4dn.xlarge), PyTorch | FP32 | 32 | 25.5 | 3 min 16 s (5,000) | $0.0057 | 46% | 0.5026 | 0 |
ECS workers, GPU (g4dn.xlarge), PyTorch | FP32 | 64 | 26.9 | 3 min 06 s (5,000) | $0.0054 | 49% | 0.5026 | 0 |
ECS workers, GPU (g4dn.xlarge), PyTorch, pipelined | FP32 | 16 | 46.8 | 1 min 47 s (5,000) | $0.0031 | 94% | 0.5026 | 0 |
ECS workers, GPU (g4dn.xlarge), PyTorch, pipelined | FP32 | 32 | 51.8 | 1 min 37 s (5,000) | $0.0028 | 93% | 0.5026 | 0 |
ECS workers, GPU (g4dn.xlarge), PyTorch, pipelined | FP32 | 64 | 48.5 | 1 min 43 s (5,000) | $0.0030 | 92% | 0.5026 | 0 |
ECS workers, GPU (g4dn.xlarge), PyTorch, pipelined | FP16 | 32 | 85.7 | 58 s (5,000) | $0.0017 | 83% | 0.5021 | 0 |
ECS workers, GPU (g4dn.xlarge), ONNX Runtime (CUDA), pipelined | FP16 | 32 | 81.8 | 1 min 01 s (5,000) | $0.0018 | 81% | 0.5021 | 0 |
ECS workers, GPU (g4dn.xlarge), ONNX Runtime (TensorRT), pipelined | FP16 | 32 | 53.7 | 1 min 33 s (5,000) | $0.0027 | 28% | 0.5022 | 0 |







