Alert after 1 failed check or 3? Tuning uptime alerts so on-call stops ignoring them

The uptime monitor fired 14 times last month. Twelve were a single failed check from one region that cleared on the next run. By the time a real outage came, the on-call engineer had muted the channel. Users noticed first.

Alert fatigue doesn't come from too many outages. It comes from alerts that are wrong often enough that people stop trusting them.

TL;DR

• A single failed check is a signal, not an outage. Confirm before you page.
• Confirm in two dimensions: time (consecutive failures) and place (more than one location).
• Work backwards from how long an outage is allowed to go unnoticed, then pick the interval and threshold.
• My opinion: "page on first failure" is almost never right for websites. Two consecutive failures from two regions is a better default.

Why single checks fail when nothing is wrong

One failed HTTP check can come from:

• a packet lost on the monitoring provider's own network
• a DNS resolver hiccup in one location
• a slow garbage-collection pause that pushed one response over the timeout
• a deploy that took one instance out for a few seconds
• a regional route flap that has nothing to do with you

None of those need a human at 3 AM.

Dimension 1: consecutive failures

Require N failures in a row before alerting. The cost is detection delay:

worst-case detection time ≈ interval × N + timeout 

Notice the last row. If you want 3 confirmations without slower detection, check more often rather than alerting on one failure.

Dimension 2: multiple locations

A failure from one region and success from three others usually means a network path problem, not your site. A failure from most regions is the real thing.

A simple rule that works well:

page if: failures in >= 2 of 4 regions, for >= 2 consecutive checks warn if: failures in 1 region, for >= 3 consecutive checks 

The warning still matters. A single region failing for ten minutes might be a real routing problem affecting real users in that region. It just isn't worth waking someone for.

Start from your budget, not from a default

Ask: how long can the site be down before someone must know? If your target is 99.9% monthly, the whole month's budget is about 43 minutes. The breakdown in What three nines really buys you shows how fast that goes.

If you burn 3 minutes of that just detecting each outage, that's fine. If you burn 15, you'll miss your target on the second incident. Pick interval and threshold so detection is a small fraction of the budget.

Tune the timeout, not just the threshold

A 30-second timeout means a hung server takes 30 seconds to count as one failure. A 3-second timeout on a page that normally takes 2.5 seconds means constant false positives.

Measure your real latency first:

for i in $(seq 1 20); do curl -sS -o /dev/null -w '%{time_total}\n' --max-time 30 https://example.com/ sleep 3 done | sort -n | awk '{a[NR]=$1} END {print "p50", a[int(NR*0.5)], "p95", a[int(NR*0.95)], "max", a[NR]}' 

Set the timeout to something like 3 to 4 times your p95. Slow enough to tolerate a bad moment, fast enough to catch a hang.

A tiny confirm-before-alert loop

If you're rolling your own check, the confirmation logic is a few lines:

import time, urllib.request URL, TIMEOUT, INTERVAL, REQUIRED = "https://example.com/", 10, 30, 2 def ok(): try: with urllib.request.urlopen(URL, timeout=TIMEOUT) as r: return r.status == 200 except Exception: return False failures = 0 alerted = False while True: if ok(): if alerted: print("RECOVERED") # send recovery notice  failures, alerted = 0, False else: failures += 1 if failures >= REQUIRED and not alerted: print(f"ALERT: {failures} consecutive failures") # page here  alerted = True time.sleep(INTERVAL) 

Two details matter more than they look: alert once per incident (not every interval), and always send a recovery notice. People trust alerts that close themselves.

Before you page, double-check from outside

When an alert fires, the first question is "is it really down, or is it my monitor?". A quick check from an independent service such as upordown.online answers that in seconds. And if the check itself is a ping or a hardcoded /health route, revisit why ping is not an uptime check and the 7 health check mistakes first. Confirmation logic can't fix a check that measures the wrong thing.

Route by severity

Not every alert should go to the same place:

Certificate expiry is a great example of an alert that should never page. It's predictable weeks ahead. If you don't track it yet, How to check when an SSL certificate expires covers it.

My take

The goal of an uptime alert isn't to catch every blip. It's to be right every time it pages. An alert that is right 95% of the time gets acknowledged. One that's right 20% of the time gets muted, and a muted alert is worse than none, because everyone assumes someone is watching.

Over to you

What's your current rule: first failure, N consecutive, or multi-region quorum? And has anyone found a good way to alert on partial outages, like one region or one ISP, without drowning in noise?