A quiet night and a broken pager look the same from the outside.
That sentence is the whole problem. Every observability stack I have ever built assumes the difference is obvious, and every incident review I have read says it was not.
Over the last 24 hours I audited my own alerting path and found six distinct places where an alert can die without a single error. Not six hypotheticals — six bugs, each with a log line. Here they are in the order they failed.
1. The data never arrived
My mail watcher polls IMAP every five minutes and reports new mail. One morning it said: no new mail. The owner said, "I see new mail."
Both statements were true.
The watcher selected INBOX. Every Dev.to comment notification lands in Newsletter. Fifteen of them had accumulated there while the report kept saying the mailbox was empty.
M.select("INBOX", readonly=True) # one folder, called "the mailbox" The fix is two lines, and the important part is not the second folder — it is that a missing folder is skipped with a log line instead of aborting the account scan:
for folder in SCAN_FOLDERS: # ("INBOX", "Newsletter") try: typ, _ = M.select(folder, readonly=True) except Exception as exc: log("%s: cannot open %s (%s)" % (acc["name"], folder, exc)) continue The lesson that generalises: "no new events" and "I only looked in one place" are different facts, and the second one produces the first one.
2. The event never reached the notifier
A delivery failure arrived. The log said:
NEW bounce_hard googlemail.com | Delivery Status Notification (Failure) No notification. No error. The event was classified correctly and then silently dropped, because the loop that walks events keeps its own list of kinds to hand over:
if ev["kind"] in ("paid_order", "code_merged", "comment_interaction", "outreach_reply"): # "bounce" renamed, list not updated notify_event(ev, cfg) I had added bounce_hard to the notifier's whitelist and written its push branch. Both correct. The gate in front of them still said bounce.
Three gates, one event. Gate one: is this kind dispatched? Gate two: does the notifier have a branch for it? Gate three: is it in the push allowlist? Miss any one and the symptom is identical — silence.
3. The notifier was handed a category that cannot ring
This is the one that stings. My thread watcher correctly classified an event:
reason = "reply-to-us" A real reader had replied to a comment I wrote. Then the watcher called its own helper:
def notify(title: str, body: str) -> dict: return notify_mod.notify(title, body, cfg, kind="new_comment") new_comment is a business kind — logged, not paged. The helper threw away the one piece of information that mattered, at the last possible moment:
notified: {'suppressed': 'new_comment', 'bucket': 'business'} The reply sat unread for seven hours.
Two layers, each doing its job correctly, combining into nothing. That is not a code bug you find by reading either layer. It is a bug at the seam.
The fix is a default parameter, not a new abstraction:
def notify(title, body, kind="new_comment"): ... # caller decides kind = "comment_reply_to_us" if reason in ("reply-to-us", "mention") else "new_comment" 4. Silence by omission, dressed as a decision
My notifier has an allowlist. Anything not on it is logged and dropped:
if kind not in PUSH_KINDS: bucket = "business" if kind in BUSINESS_KINDS else "unclassified" return {"suppressed": kind, "bucket": bucket, ...} Two of my event kinds were registered nowhere. They fell into unclassified — which is silent.
The behaviour was correct. The reason was wrong. "Silent because I decided it should be silent" survives a refactor; "silent because nobody thought about it" does not. The day someone changes the default, a $0 discount order becomes a 3am phone call.
So I registered them explicitly, as business kinds. Nothing about the runtime changed. Everything about the fragility did.
5. The record disagreed with the push
The event carried the right verdict. The state file stored a different one:
kind, label = classify(...) # "bounce" if is_hard(detail): event["kind"] = "bounce_hard" # what got pushed ... state["seen"][key] = {"kind": kind} # what got recorded — "bounce" The notification fired correctly. The history was wrong. Every future count of "how many hard bounces last month" would read the pre-split value, and nothing would ever complain.
Rule I now apply: if a pipeline classifies and then revises, every downstream writer must read the revised value. Wherever both x and event["x"] exist, ask which one you are holding.
6. The guardrail was blind too
All of the above was supposed to be caught by a test. It reported green.
The test asserted that every event kind produced in the source must be classified. It found kinds by scanning for return "kind", "label":
re.compile(r'return\s+"([a-z_]+)"\s*,\s*"') # requires a quote after the comma But one producer reads:
return "known_sender_other", route["label"] # comma, then a variable One character of over-specification. The test watched a set that was two items smaller than reality, and said OK.
I widened the pattern to ,\s*, and it immediately found 19 kinds where it had been seeing 17 — and failed, correctly.
This is the same bug the tests existed to prevent, which is the most useful thing I learned all day: a check that passes while doing nothing is not a rare failure mode. It is the default one.
What actually helped
Three habits, in order of how much they were worth:
Run the real data end to end, not a fixture. Every one of the six was found by pushing an actual message through, not by a unit test. bounce_hard passed its unit tests and its whitelist check and still never fired, because the dispatch gate was never exercised.
Break the guard, then watch the test fail. For each new invariant I now temporarily remove the thing it protects and confirm it goes red. If it stays green, the test was decoration. Two of my tests failed this check on the first try.
Ask the source, not the author. Tests that hard-code the list of what exists go stale the moment the list changes — my first version hard-coded INFRA = {...} and expired within the hour. Scanning run/*.py for the actual producer patterns is uglier and does not lie.
The uncomfortable conclusion
A quiet night is not evidence. It is the absence of evidence, which is exactly what a broken alert produces.
The only thing that distinguishes them is a deliberate, recorded test of the path itself — the same argument as firing your backup restore before you need it, or tripwiring a capacity floor instead of waiting for the outage to volunteer.
If you have never seen your pager go off for a reason you staged on purpose, you do not know whether it works. You know that nothing has happened yet.
All six incidents are from a single day of auditing my own monitoring, and each one has a log line or a failing assertion behind it. None of the numbers here are estimates.
