Some bugs don't survive an afternoon. Others survive three days, because you keep looking in the one place where you already know every line.

This is a story about the second kind.

The symptom

A small Python service of mine called an external API. On my laptop, every request took almost exactly five seconds. Not roughly five, but 5.0, 5.0, 5.1. On a colleague's machine, the same code with the same dependencies returned in 200 milliseconds.

A delay that round should have told me something. Real slowness is noisy. A perfectly constant delay means a timeout is firing somewhere. I noticed the number and moved on.

Day one: the obvious suspect

I opened my own code and started reading. I added timing logs around every call. I ran cProfile, and it did show most of the time inside socket.getaddrinfo. I read that as "the network is slow" and went back to my own functions, because those were the ones I could change.

I rewrote the retry logic. I moved the request into a thread. Behavior shifted slightly each time, which felt like progress. It wasn't. A change in behavior is not a change in cause.

Day two: the slightly less obvious suspect

When my code came up clean, I moved one step out. I upgraded requests, then swapped it for httpx. I recreated the virtual environment from scratch. I pinned older versions of urllib3.

Same five seconds. By the end of the day I was rewriting code that had no evidence against it, just because it was the code I could touch.

Day three: asking a different question

The question I had been asking was: "What in my code could cause this?"

The question that finally helped was: "What is the full list of things that could cause this, and which of them have I not looked at?"

I wrote it down as layers, from closest to me to furthest away:

  1. My code. The logic I wrote.
  2. My dependencies. Libraries, their versions, their defaults.
  3. Runtime and configuration. Interpreter, environment variables, flags.
  4. The operating system. Resolver, resource limits, drivers, services.
  5. The outside world. Router, network, the remote server.

I had spent two and a half days on layers 1 and 2. Then I did the cheapest possible test: I tethered my laptop to my phone. The requests took 200 milliseconds.

Same machine, same code, different network. The bug didn't follow the code. That ruled out layers 1 to 3 in one minute, after three days of ignoring that option.

I measured the phases with curl:

curl -s -o /dev/null -w "dns: %{time_namelookup}s total: %{time_total}s\n" https://api.example.com

The output was dns: 5.0s. All the time went into name resolution. Then came the detail that explained why I'd been fooled:

time dig api.example.com # instant time getent ahosts api.example.com # 5 seconds

dig was fast, but getent was slow. dig uses its own resolver logic, while Python, curl and almost everything else go through the system's getaddrinfo. That function sends the IPv4 and IPv6 lookups in parallel, and my router was dropping one of them. The system resolver waited out its default five-second timeout and then carried on. The tool I would naturally reach for to test DNS was the one that hid the problem.

The fix was one line in /etc/resolv.conf:

options single-request-reopen

It makes the resolver send the two queries separately. The delay disappeared. Finding it took three days. Fixing it took ten seconds.

Why I missed it

It wasn't a lack of skill. It was a habit: I searched where I was most comfortable, not where the evidence pointed. The profiler had literally named getaddrinfo on day one. I read the evidence and dismissed it because it pointed outside my code.

There's also a quieter reason. Debugging your own code feels productive even when it isn't, because you can always find something to improve. Debugging the environment feels like admitting you don't control the situation.

What I do now

  • Write down the symptom without a cause. One sentence, only what is observable. "Every request takes exactly 5 seconds" already contains half the answer.
  • List the layers before touching anything. Even a rough list of five items changes where you look first.
  • Check the cheapest layer first, not the most familiar. A second network, a second machine or a clean container takes two minutes. Rewriting a function takes two hours.
  • Ask "does this reproduce somewhere else?" If the bug doesn't follow the code, it isn't in the code.
  • Treat suspiciously round numbers as timeouts. 5.0 seconds, 30 seconds, 60 seconds: look for the setting that equals that number.
  • Set a time box. If one layer has had two hours and produced no evidence, move outward.

The takeaway

Most of what this bug taught me wasn't about DNS. It was about order of search. The question "where could this be?" is worth more than any single debugging trick.

If you've lost days to a bug that turned out to be somewhere you weren't looking, I'd like to hear which layer it was.