asyncio.gather Kept Running: 1,761 LLM Calls Finished After My 500

My summarizer endpoint returned a 500, and the user saw an error toast. Then the event loop kept working on that dead request for another 40 seconds, sending Claude calls whose answers nobody would ever read.

The cause was one line: await asyncio.gather(...). Python's asyncio.gather does not cancel the other tasks when one of them raises. Your code gets the exception right away. The rest of the work keeps running in the background, and inside a long-lived server that work can keep starting new API calls.

Over 9 days, 1,761 LLM calls finished after their parent request had already failed. Here's how I found it, a 30-line repro you can run, and the fix.

TL;DR

• asyncio.gather with the default return_exceptions=False passes the first exception to the caller without cancelling the siblings. The Python docs say so; I hadn't read that paragraph.
• Add a Semaphore and it gets worse: tasks still waiting on the semaphore keep acquiring it and start new calls after the failure.
• Later exceptions from those orphans are dropped. No traceback, no "Task exception was never retrieved" warning.
• asyncio.run() cancels leftover tasks on exit, so the bug hides in scripts and tests and only shows up in servers like FastAPI/uvicorn.
• Fix: asyncio.TaskGroup (Python 3.11+) cancels the siblings and waits for them before it raises. Calls started after a failure went from 1,522 to 0.

The setup

The pipeline is boring on purpose. A user uploads a document. I split it into roughly 40 chunks, summarize each one with a Claude call, validate the JSON with Pydantic, and merge the results.

sem = asyncio.Semaphore(8) async def summarize_chunk(chunk: str) -> ChunkSummary: async with sem: resp = await client.messages.create( model=MODEL, max_tokens=600, messages=[{"role": "user", "content": PROMPT.format(chunk=chunk)}], ) return ChunkSummary.model_validate_json(resp.content[0].text) @app.post("/summarize") async def summarize(doc: Doc): chunks = split(doc.text) results = await asyncio.gather(*(summarize_chunk(c) for c in chunks)) return merge(results) 

Concurrency is capped at 8 so I don't hit rate limits. If any chunk fails validation, the whole request fails, since a summary with a hole in the middle is worse than none. That was the design, anyway.

How did I notice orphaned LLM calls?

The numbers didn't match. My app logs one line per chunk summary it actually uses. The provider's usage console counts every call. Over 9 days my log said 43,451 calls were used and the console said 45,212. A 3.9% gap is too large to be rounding and too small to show up as an outage.

So I added a log line when each call starts, tagged with the parent request ID, and another one in the endpoint's error path recording when the 500 went out. Then I joined the two.

1,522 calls began after the request was already dead. That's not "a few in-flight calls got unlucky." The batch kept running at full speed with no one waiting for it.

And the frontend retried failed uploads twice. Most of my failures were deterministic (the same chunk produced the same bad JSON each time), so one broken document turned into three full batches, two of them pure waste.

Why doesn't asyncio.gather cancel the other tasks?

Because it was designed that way. When return_exceptions=False, the first exception goes straight to whoever awaits gather(), and the other awaitables "won't be cancelled and will continue to run." Gather cancels its children only when gather itself is cancelled from outside, for example by a timeout around it. A child raising doesn't count.

That gives you the worst mix of behaviors: fail-fast for the caller, keep-going for the work. You get the error immediately and you still pay for everything.

The semaphore is what made this expensive. Without it, all 40 calls would already be in flight when one failed, and you'd waste at most what was already running. With it, 32 tasks are sitting in sem.acquire() when the failure lands. Nothing tells them to stop, so as each running call finishes they take a slot and fire a new request.

Can I reproduce it without an API key?

Yes. This swaps the API for asyncio.sleep(0.5) and makes chunk 2 fail:

import asyncio async def work(i, sem, started, finished): async with sem: started.append(i) await asyncio.sleep(0.5) # pretend this is an LLM call if i == 2: raise ValueError("bad chunk") finished.append(i) async def main(): sem = asyncio.Semaphore(8) started, finished = [], [] try: await asyncio.gather(*(work(i, sem, started, finished) for i in range(40))) except ValueError: print("at failure:", len(started), "started,", len(finished), "finished") await asyncio.sleep(3) # a server's event loop keeps living print("3s later: ", len(started), "started,", len(finished), "finished") asyncio.run(main()) 

On Python 3.11 on my machine:

at failure: 16 started, 7 finished 3s later: 40 started, 39 finished 

When the caller saw the exception, 16 calls had started. Three seconds later all 40 had started and 39 had finished. 24 calls began after the caller had already given up.

Remove the await asyncio.sleep(3) and the orphans disappear, because asyncio.run() cancels any leftover tasks when main() returns. That's why my unit tests were green. In tests the event loop shuts down right after the failure. In uvicorn it never shuts down.

Where did the other errors go?

They were dropped. If two chunks fail, you see the first exception and gather quietly marks the second as retrieved. I checked by making tasks 1 and 5 both raise:

caught fail 1 done 

There was no traceback for task 5 and no "Task exception was never retrieved" warning. In production this meant that out of the 1,761 late calls, I had no log line for the ones that failed too. My error rate per chunk was undercounted by exactly the cases I most wanted to see.

How do you cancel remaining tasks when one fails?

Use asyncio.TaskGroup (Python 3.11+). When a child raises, the group cancels every other child, waits for those cancellations to finish, and then raises an ExceptionGroup that contains every real failure, not only the first one.

@app.post("/summarize") async def summarize(doc: Doc): chunks = split(doc.text) try: async with asyncio.TaskGroup() as tg: tasks = [tg.create_task(summarize_chunk(c)) for c in chunks] except* ValidationError as eg: for e in eg.exceptions: log.warning("chunk failed: %s", e) raise HTTPException(502, "summary failed") return merge([t.result() for t in tasks]) 

Same repro, swapping gather for a TaskGroup:

at failure: 11 started, 7 finished 3s later: 11 started, 7 finished 

Three wave-two tasks got the semaphore in the few event-loop ticks before the cancellation arrived, and they were cancelled during their "call." Nothing started after the caller saw the error, because the TaskGroup doesn't raise until its children are done.

Stuck on 3.10 or older? Do it by hand:

tasks = [asyncio.create_task(summarize_chunk(c)) for c in chunks] try: results = await asyncio.gather(*tasks) except Exception: for t in tasks: t.cancel() await asyncio.gather(*tasks, return_exceptions=True) # wait, collect, log raise 

The second gather matters. Without it you raise while cancellations are still pending, and the semaphore can hand out a slot in the meantime.

What changed after the fix?

Here are 7 days on the TaskGroup version, same traffic pattern:

The remaining 0.4% is the cancelled in-flight calls. My client hung up on them, but from my side I can't tell how much of each one the provider had already generated. I count them as possibly billed and moved on.

I also made the frontend stop retrying on 502. A deterministic bad chunk doesn't get better on attempt three.

Should you use return_exceptions=True instead?

Only if partial results are what you want. return_exceptions=True is honest: every task runs, every exception shows up as a value, and you choose what to do with it. For a search fan-out where 37 out of 40 results is fine, that's the right tool.

The problem is the default. Pick one on purpose:

• All or nothing: TaskGroup. Fails fast and actually stops the work.
• Best effort: gather(..., return_exceptions=True). Runs everything and reports everything.
• Default gather: fails fast for you and keeps going for the provider bill. In a long-running server I can't think of a case where I want that.

So does asyncio.gather cancel tasks when one fails?

No. With the default return_exceptions=False, asyncio.gather passes the first exception to the caller and leaves every other task running. In a long-lived server those orphans keep making requests, and tasks waiting on a semaphore will start brand-new LLM calls after your endpoint has already returned an error. Later exceptions are dropped without a warning, and asyncio.run() hides the whole thing in scripts by cancelling leftovers at exit. If one failure should stop the batch, use asyncio.TaskGroup on Python 3.11+, or cancel and await the tasks yourself. In my pipeline that took calls started after a failure from 1,522 to 0.

Written by the developer behind Preterview, an interview prep platform.