NVIDIA Dynamo on Amazon SageMaker HyperPod: Distributed Inference and Its Trade-offs

AWS's headline number is a 2.05× throughput jump from splitting prompt processing and token generation apart. The fine print matters more than the headline — and it's actually the interesting part.

NVIDIA Dynamo, the open-source distributed inference framework, now runs natively on SageMaker HyperPod Inference, with HyperPod supplying EFA networking, GPU scheduling, and node recovery. Dynamo supports vLLM, SGLang, and TensorRT-LLM, and the integration lets you run aggregated or disaggregated serving, workers on one instance or across several, with KV-aware routing deciding where requests land.

The 2.05× number is real — and narrowly scoped

AWS's disaggregation benchmark (Qwen3-235B-A22B-FP8 on 16 H200s, 4,000-token prompts, 200-token outputs) hit 207 output tokens per second per GPU disaggregated versus 101 aggregated. But read the method: the two numbers came from different concurrency levels (80 vs. 32), and only points meeting a P90 time-to-first-token under one second and median generation of at least 50 tokens per second per user were kept. That's a legitimate comparison under latency constraints — it is not a same-concurrency shootout, and it is definitely not a promise your workload gets 2×.

The quieter benchmark is the more honest one

A second test put Qwen3.5-397B across two instances and kept 99.1% of single-node throughput. Crossing the node boundary cost almost nothing — which is genuinely good news for multi-node serving. The catch: that model fits on a single instance, so this is a controlled baseline, not proof that a model which needs two instances (like GLM-5.3, whose FP8 weights alone exceed one ml.p5.48xlarge's 640 GB of GPU memory) will behave the same way.

Disaggregation doesn't delete the sizing problem, it moves it

Separate prefill and decode pools mean independent scaling and tuning — but both pools load the full model weights, and the KV cache has to travel between them over EFA. You still have to size and coordinate everything; you've just traded one hard problem for a different hard problem with better knobs.

What's still missing

No cost comparison anywhere in the evidence. No quantified recovery times or availability outcomes, even though node replacement is pitched as part of the operational value. The integration is a real step forward for teams serving big models — but the economics and the failure-mode numbers are still yours to measure.

Go deeper

Read the full analysis on Agentic Affairs: NVIDIA Dynamo on Amazon SageMaker HyperPod: Distributed Inference and Its Trade-offs

Originally published on Agentic Affairs (https://agentic-affairs.com)