Lots of things broke this week. The server broke lots of times. I expected the causes to be obvious. They were not, and the fastest fixes came from reading what the system was already showing me instead of guessing.
The setup: an Ubuntu VM running nginx in front of two small Python backends, with a firewall and my own certificates. I broke each layer on purpose.
The path
browser | DNS: name -> IP v router + NAT private source rewritten (MASQUERADE) | v firewall [3] default-deny, IPv4 only | v nginx :443 [2] TLS ends here | [1] proxy_pass, retry on timeout |-> backend :3001 |-> backend :3002 (round robin, max_fails=2) 1. Break the proxy [1]
I made one backend slow and expected nginx to hide it. Instead there was no answer, just a very long pause, and then I was given the 504.
nginx accepted the connection and waited for the backend to answer. The backend never did, so after proxy_read_timeout (3 s in my test) nginx gave up. Because timeout is in proxy_next_upstream, it then retried on the second backend and waited again. Two timeouts, 3 s apart, so the 504 arrived after 6 s instead of 3. The error log showed two upstream timed out (110) lines 3 s apart.
2. Break the certificate [2]
To tell them apart, I ignored the exit code and looked at the human-readable error text and the TLS handshake details that curl and OpenSSL print before they give up. Exit code 60 is the problem; the error string and the certificate chain dump are the diagnosis.
The three certificate failures I've seen:
| Failure | s_client "Verify return code" | What it means |
|---|---|---|
| Untrusted or incomplete chain (cannot obtain local issuer certificate) | 20 / 21 | the issuer is not in the client trust store |
| Self-signed certificate | 18 | the certificate signed itself |
| Hostname mismatch | 62 | the name in the certificate does not match the name I asked for |
3. Break the firewall [3]
I assumed the firewall rule was the only thing blocking the connection, so removing it should have made curl fail. It still worked.
Forcing each address family told me why: curl -4 timed out (exit 28) and curl -6 worked (exit 0). iptables filters IPv4 only, and I had written no IPv6 rules, so the connection simply went around the firewall. A real host needs ip6tables rules too, and the habit I took from this is to test both address families after any firewall change.
What I would tell my past self
Verify the thing you're about to change is actually the thing in the path: check counters and state before you touch config.
The habit that would have saved the most time: look at the evidence the system is already emitting before forming a theory. iptables -L -v counters, conntrack -L, curl -v, s_client -showcerts — the system was telling me the answer the whole time. I was reading the result (exit code, "still worked") and guessing at the cause instead of reading the mechanism.
What "broken" taught me that working never did: a green checkmark hides which component is actually doing the work. Failure is the only thing that makes the path visible. The long silence, the identical exit code across three different causes — those are the moments the system reveals its real makeup. Working systems let you keep your wrong model forever.
When something surprises you, it's going to be the data. Chase the mechanism, not the outcome: counters, state, and raw output first; theory second.