Python Deep: Asyncio Internals and FastAPI at Scale

Deep Part 11 ended with a promise: the traced gateway we built there still had to survive real traffic, and the thing that kills a FastAPI service in production is rarely the model call. It is a blocking call sitting in the wrong place, a thread pool that quietly fills up, or a fan-out with no limit. This part opens the black box. We froze the event loop on purpose, measured uvloop against the standard loop, and benchmarked FastAPI with one worker and four, so we can show numbers instead of repeating the usual advice.

Our position, stated plainly before the numbers: most FastAPI slowness is not FastAPI. It is Python code that blocks the one thread doing all the work. uvloop and extra workers are the last 15 percent, not the first fix. If your endpoint waits on a database or an LLM API, the loop choice barely matters. If your endpoint does real work on the loop thread, no loop choice will save it.

This is the final part of the deep series, and it leans on the earlier parts. Part 2 of the original series covered asyncio for AI engineers at a practical level. Deep Part 4 measured inference latency, and Deep Part 10 built the MCP server that the Deep 11 gateway now fronts. Everything here runs on Python 3.12.3, FastAPI 0.162, uvicorn 0.54, and uvloop 0.23, on a small shared virtual machine with 4 vCPUs. The absolute request rates are modest because the load client and the server shared the same machine. The ratios between configurations are the point, and we say so wherever a number depends on the rig.

The event loop is one thread with a to-do list

Asyncio runs one event loop on one thread. The loop keeps a queue of ready callbacks and a set of file descriptors it watches with epoll on Linux. When a coroutine awaits something slow, it hands control back to the loop, and the loop runs the next ready callback. When the slow thing finishes, its callback goes back in the queue. Nothing ever runs in parallel inside one loop. The magic is only interleaving: while one request waits on the network, another request uses the CPU.

That single fact explains every failure mode in this article. A coroutine that never awaits holds the thread, and every other request waits behind it. A coroutine that awaits properly costs almost nothing while it waits. The loop itself is cheap. Creating 10,000 empty tasks took 0.077 seconds on the standard loop in our test, and 0.053 seconds on uvloop. Tasks are not the bottleneck. Blocking is.

Experiment 1: one blocking call freezes every request

We ran a heartbeat task that wakes every 50 milliseconds and records how late it was. Then we ran a CPU bound function for 1.5 seconds in two ways: called directly inside a coroutine, and pushed to the default executor with run_in_executor. The code is short enough to read in full.

 async def heartbeat(gaps): while True: t = time.monotonic() await asyncio.sleep(0.05) gaps.append(time.monotonic() - t) def cpu_burn(seconds=1.5): end = time.monotonic() + seconds x = 0 while time.monotonic() ConfigEndpointRequests per secondp50 latencyp99 latency 1 worker, stdlib loop/fast2,74917.1 ms26.8 ms 1 worker, stdlib loop/io90952.0 ms77.2 ms 1 worker, stdlib loop/sync-io69064.9 ms128.6 ms 1 worker, uvloop/fast3,18813.5 ms27.3 ms 1 worker, uvloop/io87752.9 ms69.4 ms 1 worker, uvloop/sync-io70364.5 ms141.6 ms 4 workers, stdlib loop/fast1,16242.1 ms54.8 ms 4 workers, stdlib loop/io53593.0 ms105.5 ms 4 workers, stdlib loop/sync-io51793.0 ms116.8 ms 4 workers, uvloop/fast3,1699.1 ms71.1 ms 4 workers, uvloop/io90052.0 ms64.7 ms 4 workers, uvloop/sync-io79456.6 ms84.0 ms Three findings survive a hard read of that table. First, uvloop helped exactly where theory says it should: the /fast endpoint with one worker went from 2,749 to 3,188 requests per second, a 16 percent gain, with p50 latency falling from 17.1 to 13.5 milliseconds. On the /io endpoint, which spends its life waiting, uvloop changed nothing: 909 versus 877 requests per second is noise. Second, async def beat plain def on the waiting endpoint, and the reason is the thread pool. The async /io endpoint served 909 requests per second at 52.0 milliseconds p50, which is the 50 millisecond wait plus about 2 milliseconds of overhead. The sync /sync-io endpoint served 690 requests per second at 64.9 milliseconds p50. At 50 concurrent requests against a 40 thread pool, roughly 10 requests queue for a free thread on every round, and that queueing is the extra 13 milliseconds. The p99 gap is wider: 77.2 against 128.6 milliseconds. The rule writes itself: if the work is waiting, use async def and a truly async client. If the work is a synchronous library, use def and accept that the pool is a shared resource with a 40 thread ceiling. Third, four workers bought nothing on this rig, and one configuration got visibly worse. The 4 worker stdlib run served /fast at 1,162 requests per second against 2,749 for one worker, and /io at 535 against 909. That is not a finding about workers; it is a finding about our test rig. Client and server shared the same small machine, so four server processes plus the load client fought for the same cores, and the extra processes added scheduling overhead without adding capacity. The 4 worker uvloop run recovered to roughly the 1 worker numbers, which tells us the dip was contention, not a property of workers. On real hardware, with the load coming from elsewhere, workers equal to CPU count is still the right default, because one loop uses one core and a second process is the only way to use a second core. We report the dip because hiding it would be dishonest, and because it is a useful warning: benchmark workers on hardware that looks like production, not on the same box as the load generator. ## Our production checklist for a FastAPI service **Run one worker per CPU core.** Start uvicorn with workers set to the core count, behind a reverse proxy, and let the operating system balance connections. One worker leaves cores idle. More workers than cores adds context switching without adding throughput. **Turn on uvloop on Linux.** It is one flag, uvicorn main:app --loop uvloop, and it buys a real 10 to 20 percent on thin endpoints. Do not expect it to fix a slow model call. Measure before and after with your own traffic shape, the way we did here. **Never call time.sleep in an async endpoint.** It freezes the loop exactly like Experiment 1. The async equivalent is await asyncio.sleep, and the real fix for a slow synchronous call is an async client or run_in_executor. **Cap every fan-out with a semaphore.** Pick a number your downstream can handle, 10 or 20 is a sane start, and gather through it. Unbounded fan-out is the fastest way to turn one slow dependency into an outage. **Give the thread pool a budget and watch it.** Forty threads is the default, and it is shared by every plain def endpoint plus every run_in_executor call. If p99 on a sync endpoint climbs while p50 stays flat, the pool is queuing. Either convert the call to a real async client or raise the limiter deliberately, knowing each thread costs memory. **Pool database connections and set timeouts.** An async driver with a pool of 10 to 20 connections behaves like the semaphore experiment: bounded, predictable, and honest about overload. A timeout on every downstream call turns a hung dependency into a fast error instead of a hung worker. The observability from [Deep Part 11](/blog/llmops-deep-observability-cost-control-eval-driven-deploys) is what shows you these numbers in production: trace the request, record the pool wait, and the bottleneck names itself. ## What is Next in This Series This is Deep Part 12, the last part of the deep series as planned. Across twelve parts we went from attention internals to tokenization, training, inference, context windows, RAG chunking, hybrid search, evals, LangGraph, MCP, LLMOps, and now the Python runtime underneath all of it. The series is complete, and the next step is not another part but the work itself: take the traced gateway from Deep Part 11, apply the checklist above to it, and measure it under your own traffic. A p95 problem is only visible once the traces exist, and a blocking call is only visible once you measure loop lag the way Experiment 1 does. If you are reading the deep series in order, the full run is: [Deep Part 1 on attention](/blog/transformers-from-scratch-attention), [Deep Part 2 on tokenization](/blog/tokenization-bpe-from-scratch-vocab-size), [Deep Part 3 on training and scaling laws](/blog/training-llms-pretraining-loss-curves-scaling-laws), [Deep Part 4 on inference and KV cache](/blog/llm-inference-kv-cache-sampling-temperature-measured), [Deep Part 5 on context windows and RoPE](/blog/context-windows-rope-long-context-why-models-forget), [Deep Part 6 on RAG chunking](/blog/rag-chunking-strategies-scale), [Deep Part 7 on hybrid search and reranking](/blog/hybrid-search-reranking-measured), [Deep Part 8 on evals](/blog/evals-deep-beyond-ragas-eval-harness), [Deep Part 9 on LangGraph](/blog/langgraph-deep-state-memory-multi-agent-patterns), [Deep Part 10 on MCP](/blog/mcp-deep-tool-design-security-production-hardening), and [Deep Part 11 on LLMOps](/blog/llmops-deep-observability-cost-control-eval-driven-deploys). The original eight part series that this deep series revisits starts with [Part 1, the AI engineer roadmap](/blog/ai-engineer-roadmap-2026), and ends with [Part 8 on shipping AI to production](/blog/llmops-shipping-ai-to-production). --- *Originally published on [LogicStack](https://logicstack.org/blog/python-deep-asyncio-internals-fastapi-at-scale). Run the code examples right in your browser with our free interactive [Python](https://logicstack.org/python-playground) and [SQL](https://logicstack.org/sql-playground) playgrounds — no setup, no sign-up.*