It's 2 a.m. Your origin server runs out of disk, panics, and never comes back. The on-call engineer is asleep. And here's the part that makes it worse: your DNS keeps answering, cheerfully and correctly, with the IP address of a machine that no longer exists. Every recursive resolver on the planet now holds a valid, authoritative answer that points real users straight into a black hole.

DNS doesn't monitor anything. It's a directory service — it answers from the zone you configured, and it has no idea whether the IP inside that answer still accepts connections. DNS failover exists to close exactly this gap: instead of always returning the same address, the record watches its own targets and rewrites the answer the moment one of them stops responding.

This is a practical tour of how that mechanism works, where it genuinely helps, where it doesn't, and how to set it up so it actually fires when you need it — not three hours after your users have already tweeted about it.

The gap: a directory with no pulse check

The most common instinct when a server is flaky is to add a second A record and call it a day. Two IPs, traffic splits between them, problem solved — right?

Not quite. Multiple A records give you round-robin distribution, but every value in that set is treated as equally alive. If one of the two servers dies, resolvers still hand out its address roughly half the time. Your outage goes from "site down" to "site down for 50% of visitors, randomly" — which is arguably more confusing to debug, because half your team will be unable to reproduce it.

The missing ingredient is health awareness. Nothing in the DNS protocol checks whether an endpoint is reachable before answering with it. That has to be layered on top by whoever operates the authoritative servers, and that layer is what we call failover.

What DNS failover actually is

A failover record looks almost identical to an ordinary multi-value record — same hostname, multiple A or AAAA values — but the values have roles:

  • One (or more) primary targets, serving traffic under normal conditions.
  • One or more backup targets, idle until needed.

Behind the record sits a probe that continuously checks each target. As long as the primaries answer, the record serves them — nothing changes for your users. When a primary stops passing the probe, the record quietly drops it from the answer and serves the backup instead. When the primary recovers, it's promoted back. From the outside, it looks like your hostname simply refuses to die.

It's worth being precise about what this is not: it's not load balancing at the HTTP level, it doesn't drain sessions, and it won't rescue a misconfigured application. It operates one layer below all of that — it decides which IP gets handed out, nothing more. Used for what it's good at, though, it turns your most common failure mode (an unreachable endpoint) into a non-event.

The health check is the whole game

Everything about failover quality comes down to the probe. Three design decisions matter most.

1. What to probe. A TCP probe that connects to your service port (80, 443, 5432, whatever the endpoint actually serves) is the workhorse: cheap, unambiguous, and hard to fool. An endpoint that accepts the TCP handshake is at least running; one that refuses or times out is not. (Application-level checks — "is it returning a healthy HTTP status?" — are more truthful but heavier, and many platforms don't offer them, so know which kind you're buying.)

2. How often. Probing every second burns resources and gains little; probing every ten minutes means users suffer for ten minutes before the switch. For most production setups, a probe on the order of once a minute is the sweet spot — fast enough that a real outage costs you a minute, slow enough to stay out of the way. Serious implementations let you tighten the interval on paid plans when the stakes justify it.

3. Debouncing — the part everyone skips. Networks blip. A single dropped probe proves nothing; switching answers on the first failure is how you build a system that flaps — bouncing traffic between primary and backup on transient hiccups, resetting caches each time. The fix is to require consecutive failures before acting: a common design is two failed rounds in a row before the target is declared down. Two-round debounce plus a minute-scale probe interval means the worst case for a genuine hard failure is a couple of minutes of impact — and, critically, transient noise costs you nothing.

Failover vs. round-robin vs. weighted

The same "one hostname, several values" record can behave in very different ways depending on the scheduling policy:

  • Round-robin: rotate through all values. Simple, adds no capacity awareness, and — the key point — completely blind to health. A dead member keeps receiving its share of traffic.
  • Weighted: send a configurable share to each value. Great for canary rollouts and uneven hardware; still no health awareness on its own.
  • Failover: active/passive with health checks; backups only appear when primaries fail. This is the only policy in the family that reacts to a dead endpoint.
  • Source-hash style policies (session stickiness by client IP) solve a different problem entirely — keeping a given visitor on the same endpoint — and are orthogonal to the availability question.

If you take one thing from this section: round-robin and weighted answers distribute traffic; failover is the one that removes traffic from something that's dying. If your goal is surviving a dead server, you need the third one.

When everything is down: never answer empty

Here's a corner case that separates careful implementations from careless ones. Suppose the primary is down, failover switches to the backup — and then the backup dies too. Now what?

A naive implementation returns an empty answer: the name exists, but there's nothing to give you. That's the worst possible behavior, because empty answers invite negative caching — resolvers remember "no answer here" for the record's TTL, so even after a server recovers, a chunk of the internet keeps believing your hostname is broken. A total outage of the endpoint becomes a total outage of the name, with a built-in lag on recovery.

The better design, and what you should look for in any provider's docs: when every target in a failover set is down, keep serving the last known good answer. It's wrong about the world, but it's the most useful kind of wrong — the moment anything recovers, clients start reaching it immediately, with no negative-cache tax on top.

Detection is half the battle: alert while it happens

Failover answers the question "what do users get while a server is down?" It doesn't answer "when do I find out?" — for that you need the monitoring half of the platform to shout at you. Two things make alerting actually usable:

  • The same debounce logic applied to notifications: several failed probes and a short aggregation window before an alert fires, so a single lost packet doesn't page anyone at dinner.
  • Channels you actually read. Email is table stakes; the useful ones are IM hooks — Telegram and DingTalk-style webhooks land the alert where your team already lives, which is the difference between a two-minute response and a two-hour one.

A solid DNS monitoring setup pairs downtime detection with the same probe infrastructure the failover uses, so what the record does and what the alert says never disagree.

A setup checklist that actually works

  1. Give the backup something real to do. Failover switches traffic, not state. If the backup serves a stale copy or a bare maintenance page, decide deliberately whether that's acceptable — a branded "we're briefly down" page beats a connection timeout, but be honest with yourself about which it is.
  2. Probe the port the service actually uses, not a neighboring one. A web server that has fallen over into a state where 443 is closed but 22 still answers should count as down.
  3. Set the interval and debounce consciously. Minute-scale probing with two-round failure confirmation covers most setups; tighten only if your recovery-time objective genuinely demands it.
  4. Keep the TTL on failover records short. Failover changes the answer; a long TTL delays the change reaching users. A few minutes is the right neighborhood here — these records are exactly the ones that shouldn't be cached for hours.
  5. Verify the backup is reachable before you need it, not during the incident. A free port checker run from outside your network tells you whether the standby actually accepts traffic on the right port — firewalls and security groups have a talent for silently eating failover traffic.
  6. Drill it. Once, deliberately, in a quiet window: stop the primary, watch the probe fail twice, confirm the backup is being served, restart, confirm the switch-back. A failover mechanism that has never been observed firing is a hypothesis, not a feature.

Three gotchas people learn the hard way

  • TCP-up ≠ application-healthy. The process may accept connections and still serve HTTP 500s. Failover fixes unreachable; it cannot see unhappy. That's what application monitoring is for — know which layer you're covered on.
  • Hair-trigger switching costs more than it saves. Aggressive single-failure switching feels safe and produces chronic flapping. The debounce is not a bug in your platform's confidence; it's the feature that keeps the system stable.
  • Recovery gets forgotten. Many people test the switch to backup and never test the return. If the primary comes back into rotation half-configured, you've engineered a countdown timer to a second outage. The drill in step 6 covers both directions — do both.

Wrapping up

None of this requires exotic infrastructure: a failover-capable record, a sane probe with debouncing, an alert channel your team reads, and one rehearsal. What it buys you is a specific and very common kind of 2 a.m. — the dead endpoint — being demoted from "wake everyone up" to "a notification you read with breakfast."

If you'd rather not assemble the pieces yourself, this is squarely managed-DNS territory. MatchDNS is an authoritative DNS hosting platform whose failover records probe targets over TCP with two-round failure confirmation before switching — and when every endpoint in a set is down, it keeps serving the last known answer rather than an empty one, so a total outage doesn't compound into broken negative caching. The free tier is enough to wire it up for a couple of domains and run the drill yourself.