The request log said 1,593,543 ms.
That's one request taking 26 and a half minutes. For those 26 minutes, our API logged nothing else. Not an error, not a warning, not a single other request. The health check failed, the proxy in front of it started refusing every call, and then at 6:41 UTC everything came back as if nothing had happened.
We found out because a site build failed.
1. The clue was in the wrong place
Late Friday night (Arizona time), the build of our public site failed. The build reads our live calendar of deprecation dates, and that read timed out. That's not normal. The calendar is one of the cheapest things we serve.
The API's own log had the rest. No crash, no deploy. The process never died. It just stopped answering for 26 minutes, from 06:15 to 06:41 UTC, and in the middle of that window sat one request with a 1.6 million millisecond duration.
The request was a check of one Python package: pypi:sentry@23.7.1. Sentry's own server package, an old version, which a census we were running had nominated for a look.
2. What that check does
Since can tell you which AI model a library sends when you don't pick one. (new ChatOpenAI() with no model sends whatever the library defaults to, and sometimes that model is retired.) To answer, we read the package's code and look for classes that hold a model id as a default a caller could change.
Part of that is inheritance. If MyChat extends BaseChat and BaseChat holds the default, MyChat holds it too. So for each class we walk up its bases, to a depth of six, asking "does anything up there hold a default?"
That works fine until you meet Django-style code.
3. The tree that wasn't a tree
Sentry has hundreds of classes with the same name. Every Django model and serializer can carry an inner class called Meta. There are lots of classes called Serializer. And many of them have bases that are also called Meta or Serializer.
Our walk looked bases up by name. A name like Meta doesn't point to one class. It points to every declaration of Meta in the package. So at each level, every base named Meta fanned out into every class named Meta, and each of those fanned out again.
That's not a walk up a tree. It's an exponential blowup wearing a tree's clothes.
Here is a toy version you can run. Forty classes called Meta, each subclassing Meta:
import time from functools import lru_cache # A toy codebase: 40 classes named "Meta", each one subclassing "Meta". # A lookup by name can't tell them apart, so every "Meta" base # could be any of the 40. classes = {"Meta": [["Meta"] for _ in range(40)]} def holds(name, depth, seen=None): seen = seen if seen is not None else [0] seen[0] += 1 if depth == 0: return False for bases in classes.get(name, []): # every declaration of the name for base in bases: if holds(base, depth - 1, seen): return True return False @lru_cache(maxsize=None) def holds_memo(name, depth): if depth == 0: return False return any(holds_memo(b, depth - 1) for bases in classes.get(name, []) for b in bases) for depth in (3, 4, 5): seen = [0] t = time.perf_counter() holds("Meta", depth, seen) print(f"depth {depth}: {seen[0]:>11,} visits, {time.perf_counter() - t:.2f}s") t = time.perf_counter() holds_memo("Meta", 6) print(f"memo, depth 6: {holds_memo.cache_info().currsize} visits, {time.perf_counter() - t:.4f}s") On my laptop (Python 3.9):
depth 3: 65,641 visits, 0.01s depth 4: 2,625,641 visits, 0.36s depth 5: 105,025,641 visits, 16.58s memo, depth 6: 7 visits, 0.0001s Every extra level multiplies the work by 40. Depth six, the one we actually used, would be another 40 times on top of that 16 seconds. The memoized version does seven lookups, because there are only seven distinct questions: "does Meta hold a default, with this much depth left?"
Our real fix is the same idea: remember the answer for each name and depth, and clear it each round. Same answers as before. The Sentry read, reproduced locally at 364 to 550 seconds, now takes 3.6 seconds.
4. The part that actually stung
The slow walk explains why the check was slow. It doesn't explain why one slow check took down the whole API.
That part was the timeout. The read had one. It was even checked: inside the loop, the code asked "has my signal been aborted?" on every turn.
But in an event loop (JavaScript's, Python's asyncio, any of them), a timer can only fire when the thread gets a chance to run something else. The loop never gave it one. No await, no yield, just synchronous work, millions of times. So the timer that would have aborted the read sat in the queue, waiting for a turn that never came, while the loop kept asking "has my timer fired yet?" and the answer kept being no.
It's a bit like checking your phone for a text while holding it face down in a drawer.
A timeout that can't run is a comment. Ours looked exactly like working code, and in every normal case it was, because normal reads finish long before their deadline.
So there are two fixes, not one:
- Make the work smaller. The memo turned an exponential walk into a linear one.
- Make the work stoppable. The reader's loops now hand the thread back as they go, so a timer can fire, a health check can answer, and the read can actually be cancelled. A read that runs past its budget is stored as "too large to read in the time one read has" instead of hanging around.
The first fix solves this package. The second fix solves the next package we haven't met yet.
5. Making the next one loud
The worst part of the 26 minutes wasn't the slowness. It was that we learned about it from an unrelated build. So every process now runs a tiny one-second timer. If it fires five seconds late or more, something held the thread, and that gets logged at error level with how long, and raised as an alert.
It's the dumbest possible check, and that's why I like it. It doesn't need to know what went wrong, only that the thread wasn't free. (It also can't catch a thread held forever, since then the timer never fires at all. The health check covers that case.)
And the site build that started all this now retries its calendar read three times, ten seconds apart, instead of failing on the first slow answer.
What I'd tell myself before
- "Look it up by name" is a graph, not a tree, the moment two things can share a name. Memoize anything that walks it.
- A deadline check inside a synchronous loop is decoration. If the loop never yields, nothing else runs, including the thing that would stop it.
- Measure "the thread was busy" directly. Errors tell you what failed. A late timer tells you that nothing else got to run, which is the failure you can't see.
The toy above fits on one screen. The real one took a package with hundreds of Meta classes to find.
What's the longest a single request has ever held your service, and how did you find out?
I'm Nabil. I build Since, which tracks what your code depends on and tells you before it breaks.

