The Short Version
- An alert called "local" was checking the public internet.
- Two replicas turned into one line, and my CI check stayed green.
- A monitoring chart was never installed, so my rules had nothing to measure.
- One config mistake silently dropped about 870 of 935 Postgres series.
- What I do now so a green alert actually means something.
The Pager Goes Off
So. It was August and an alert told me a local service was dead and to go and check the containers. I checked. Every one of them was idle, nothing was wrong, and the probe watching from London, which should have disagreed with the alert, was green the whole time. I work on a small cost-tracking product called Plutus, so I spend a lot of my time telling other people why their numbers moved, and here was my own monitoring giving me a confident, wrong answer.
Where Was It Actually Looking?
Honestly, the answer was embarrassing. The rule was called Local Health Probe Failing and I'd named it after what I wanted it to do, not what it did. It pointed at the public URL, which leaves the box, goes out to Cloudflare and comes straight back in. What had really happened was about half an hour of packet loss on the box's outbound traffic, with TLS handshakes going from 0.045 seconds to 5.5. So I wrote a proper loopback probe, with redirects switched off so it can never wander back out through Cloudflare by accident.
Two Pods, One Line
This is the one that still annoys me. My collector config pinned the same container label on every backend pod, so my two replicas produced identical label sets and collapsed into a single series. Every backend rate, ratio and latency rule quietly started measuring one pod. The CI check that's meant to catch this stayed green the whole time, because it compared the alert rules against the collector config and both had the same wrong idea about how the pods were labelled. Now there's a script that runs every rule against live Grafana and fails if any of them returns nothing, which is the only version of that check that looks at what's really there.
The Collector I Never Installed
This one only turned up because I asked an AI agent to look at live Grafana. I'd re-pointed the alert rules and both dashboards at the new Kubernetes labels, and written the config for the collector chart that produces them, but the automation that actually installs that chart was never updated. So every app metric in production still carried the old labels, and every rule and panel selecting on the new ones matched zero series. Nothing paged, because a rule with nothing to measure doesn't fail, it just sits there green. That's why the check from the last section exists.
870 Missing Series
The last one was a single line of config. I switched on a few extra Postgres collectors in the exporter, and that setting, as far as I can tell from the data, replaces the default set instead of adding to it, so a list of only the four extras dropped about 870 of 935 Postgres series. I worked it out from the data afterwards, because nothing errored. I fixed the list, spelled the defaults out, and added an alert that fires when those series go missing, with "no data" set to alert, which is the opposite of what most of my rules do.
Here's the exporter's series count over that night. It falls off a cliff and sits flat at the bottom for hours, with no errors anywhere.
This is the list as it stands now. The first version had only the "optional" line.
enabled_collectors = [ // defaults "database", "locks", "replication", "replication_slot", "roles", "stat_bgwriter", "stat_database", "stat_progress_vacuum", "stat_user_tables", "statio_user_tables", "wal", // optional "database_wraparound", "long_running_transactions", "postmaster", "stat_checkpointer", ] And the guard alert. It counts two series that should always exist, and the last line is the point.
expr = "count(pg_stat_user_tables_n_dead_tup) + count(pg_stat_database_numbackends)" op = "lt" threshold = 1 for = "15m" no_data_state = "Alerting" Why Does an AI Make This Matter More?
Because these days I point an AI agent at production and ask what's going on, which is how the missing collector turned up. With good telemetry it reads the logs, runs the queries and finds the pod in about the time it takes to make a cup of tea. With telemetry that has holes in it, it tells you everything's fine just as confidently, because an empty query result can mean the system is healthy or can mean nobody ever wired the thing up. So I wrote the traps down next to the queries, for me and for the agent.
What I'd Do Again
- Run every alert rule against the real system and fail the build if one returns no series. That check would have caught the collector and the replicas.
- Add a guard alert for missing series, and set "no data" to alert on it.
- Write "down" and "unhealthy" as two rules, because a dead component stops reporting and the "unhealthy" rule goes quiet.
- Keep alert rules in version control, so a threshold change gets reviewed like any other code.
- Write the traps down next to the queries.
(Yes, the title is a bit dramatic. The alerts weren't lying so much as badly briefed, which I suppose makes me the liar.)
