In three days the site dropped off the network five times, the longest for five hours. The server itself ran without a single error the whole time: no reboots, no crashes, stats kept coming in. Only the road from it to the internet kept disappearing. Here is how I tracked that down and what I changed so that next time the answer is ready on its own.
The setup
A small server that runs my portfolio site and an SSH honeypot for bots. The site lives in Docker behind a reverse proxy; next to it a watchdog made of bash and systemd timers checks every ten minutes that everything works and sends a push notification if something breaks. An external monitor checks the site from outside every five minutes.
"0 problems" while the site is down
I was not the first to learn about the outages: an external monitor was. It emails me when the site stops answering. At the same time the server's own check was writing "0 problems" to its log every ten minutes.
It was not wrong. It checks what is inside: whether the data is fresh, containers and services are running, there is disk space left. Inside, everything was fine. That is the first lesson of this story: a check from inside sees the server, not the path to it. A visitor does not care that the processes are alive if nobody can reach them.
What the logs showed
First I ruled out the obvious: the server had not rebooted in six days, no container had restarted, the kernel log had no out-of-memory or disk errors. Docker, however, kept logging the same error at the right minutes: it could not ask DNS for the address of anything, from the certificate authority to my own domains. Timeout, timeout, timeout.
To see where the outages begin and end, it is enough to bucket these errors into ten-minute slots:
journalctl -u docker --since today -o short-iso --no-pager | grep 'failed to query external DNS' | cut -c1-15 | uniq -c The picture came out neat: silence, a half-hour burst, silence again, then an hour and a half of solid errors. The bursts matched the monitor's emails to the minute. The server could neither reach outside nor accept connections from outside, so the whole link was gone.
Teaching the watchdog to notice a drop
The first fix is obvious: also check the outside connection. Now the watchdog asks DNS and tries to connect to a couple of public addresses every ten minutes.
The tricky part is elsewhere: while the link is down there is no way to report it, a push simply will not leave the server. So the watchdog remembers when the first check failed, and when the link comes back it sends one message, "offline from … to …", and prepares a draft write-up by itself. The day turns red on the uptime bar right away, without waiting for me to write the text.
A false alarm
Two days later the watchdog caught three drops. I was about to write them up, but first decided to check whether visitors had actually noticed. The best witness here is the site's own access log: if external requests kept arriving, the site was reachable, whatever the watchdog thought.
python3 -c ' import sys, json, datetime, ipaddress, collections c = collections.Counter() for line in sys.stdin: e = json.loads(line) if ipaddress.ip_address(e["request"]["remote_ip"]).is_global: c[datetime.datetime.fromtimestamp(e["ts"]).strftime("%H:%M")[:4] + "0"] += 1 print(sorted(c.items()))' < access.log The script only counts requests from public addresses: internal ones, such as checks from the local network, do not count.
Two drops were confirmed: not a single request came in from outside during those hours. The first one, ten minutes long, turned out to be false. Only DNS was failing, and visitors were getting in just fine. The watchdog saw that the server could not reach outside and concluded that the site was down.
That is the second lesson: "the server cannot reach outside" and "the site is down" are different events, and both need checking. Now, before writing a draft, the watchdog looks at the access log itself. If requests came in during at least half of the gap, it only sends a push saying the site was reachable all along, and the uptime bar stays green.
Where exactly it breaks
What was left was to find out whose side the drop is on. That takes two addresses: the gateway the server uses to reach the network, and the first provider hop behind it. The gateway comes from the routing table, and the first provider hop can be found with a simple trick: send a ping with a time-to-live of 2.
ping -n -c1 -t 2 1.1.1.1 The packet dies at the second hop, and that hop answers "Time to live exceeded", giving away its address. While the link is up, the watchdog remembers both addresses once a day. During an outage it pings them every ten minutes and writes a verdict into the draft:
- the gateway did not answer: the fault is on our side;
- the gateway answered but the provider hop did not: the line to the provider failed;
- both answered: the problem is further out in the provider's network, or DNS only.
With a verdict like that you can go to support not with "it sometimes doesn't work" but with exact times and a location.
An outside view that works during an outage
The final touch: a status page that opens even when the site is down. It lives on GitHub Pages, checks the site every five minutes from GitHub Actions and sits on the subdomain status.ogurexxx.ru.
There was a trap here too. The site has HSTS enabled for all subdomains: the browser opens them only over HTTPS with a valid certificate and does not let you click through a warning. Until the certificate for the new subdomain is issued, the page simply does not open, even if DNS is set up correctly. And some monitoring services only issue a certificate for your own domain on a paid plan.
How much downtime can I afford
After this story the incidents page got a reliability table: number of outages, total downtime, mean time to recover (MTTR) and mean time between failures (MTBF). And an error budget, SRE-style. The target is 99.5% availability per month, which is about 3 hours 40 minutes of downtime.
November burned through that budget in its first week: the two outages on 5 November alone took more than five hours. It is an unpleasant number, but a useful one. While budget is left, I can experiment freely. Once it is gone, there is one priority, finding the cause of the drops, and new features wait.
What I took away
- A check from inside and a check from outside answer different questions. You need both.
- During an outage the server cannot report it by itself. So the report has to arrive after recovery, with the facts: how long it lasted and where it broke.
- Before writing a write-up, ask a witness the outage did not touch. The access log honestly showed which drop was real.
- Five drops in three days is no longer chance but a fault. Now I have the data to find it.
Write-ups of each outage are on the Incidents page, the current state is at status.ogurexxx.ru.
This post was first published in Russian on Habr.
