Why your cloud monitoring is blind to IPv6 blackholes (and how to fix it)

TL;DR: your checker probably takes the first address the resolver hands back, and on a dual-stack host that is the AAAA record. If IPv6 is a blackhole — a route that exists and drops packets — every check waits for its full timeout and then reports the target as down. The target answered in 281 ms on IPv4 the whole time. This post is the ten-second experiment that proves it, the two different shapes of IPv6 failure that everybody merges into one word, and a two-file CLI that tells them apart.

The dashboard was green. The site was green. A monitor reported timeout of 48048ms exceeded and paged somebody at 3am, and the site had been answering requests the entire time.

That number is not a network fault. It is a timeout setting faithfully reporting how long it waited. Which means the interesting question is not "why did it take 48 seconds" but "why did anything wait at all, and who decided the target was the one to blame?"

Here is the answer, and you can reproduce it in a container in about ten seconds.

The ten-second experiment

Take a hostname that resolves to a healthy IPv4 address and an IPv6 address that swallows packets. I used a documentation prefix for the AAAA side, which is the cleanest way to make a blackhole on purpose:

printf '127.0.0.1\tlocalhost\n104.20.23.154\tdemo.internal\n2001:db8::1\tdemo.internal\n' > /tmp/hosts.demo 

Now ask the resolver what it thinks, and then ask a checker that takes the first answer:

$ python3 -c "import socket; [print(' ', 'IPv6' if i[0]==socket.AF_INET6 else 'IPv4', i[4][0]) for i in socket.getaddrinfo('demo.internal',443,proto=socket.IPPROTO_TCP)]" IPv6 2001:db8::1 IPv4 104.20.23.154 

IPv6 comes first. That is not a bug and it is not this host being odd — RFC 6724 says to prefer the larger scope and longest matching prefix, and identical scopes break the tie in favour of IPv6. Any resolver worth using does this. Which means the very common line below is a coin flip:

addr = socket.getaddrinfo(host, 443, proto=socket.IPPROTO_TCP)[0] # <- the bug 

A check built on that line gives this answer, on this host, at this moment:

$ python3 -c "..." DOWN (timeout) after 10.0s 

And the same target, measured properly:

$ alertpath --check-host demo.internal --timeout 10 demo.internal:443 -> IPV6-BLACKHOLE  vantage : localhost summary : IPv6 blackholed while IPv4 works for demo.internal A : 104.20.23.154 AAAA : 2001:db8::1 IPv4 104.20.23.154 ok 281ms IPv6 2001:db8::1 timeout 10007ms no response before 10s deadline 

281 milliseconds, versus a ten-second report that says DOWN. Both are true statements about the same hostname at the same instant. One of them is a measurement. The other is a measurement of the wrong thing, wearing the target's name.

Note what the failure costs you: the timeout is not wrong, it is unattributed. Nothing in DOWN (timeout) says whether the far side failed to answer or our own packets failed to arrive. Those have opposite owners, and the alert has one word for both.

The two shapes of IPv6 failure, which are not the same thing

This is the part I got wrong first, and it is the part that matters for how you fix it.

I tested three vantages I actually had access to. A homelab host with working IPv6, a cloud VM in the Asia-Pacific region, and a default Docker bridge network.

Two of those rows look similar in a log and are completely different animals:

No route. The kernel answers immediately: ENETUNREACH, in zero milliseconds. Your check fails fast, fails loudly, and falls back to IPv4 if you wrote it to fall back. Annoying, but honest. A container with no IPv6 route is this case, which is why most people never see the dangerous one.

Blackhole. A route exists. Packets leave. Nothing comes back, and nothing tells you so. The connect call does not fail — it waits. This is the only one that produces a false green, and it is the one that costs you a night. Measured above: the full deadline, every time, with no distinguishing text.

Here is why that distinction is not academic. If you see "IPv6 doesn't work" and you have the first shape, you conclude you have no IPv6 and move on — correctly. If you have the second shape, you also conclude you have no IPv6, and you move on, and meanwhile the resolver is handing IPv6 to the front of the list, so roughly half your connects begin by waiting for a deadline that will never be met.

Same conclusion, opposite risk. One of these is a config note. The other is why your p95 went up and nobody could find out why.

Containers make this worse, in a specific and boring way

A default Docker bridge network has no IPv6 route, so you land in the loud failure shape — measured at 0 ms, Network is unreachable. That is genuinely the safe case, and it is why containerised probes usually survive this.

But two things change the picture:

1. The container still resolves AAAA records, because DNS works over IPv4. So getaddrinfo still hands back an IPv6 address first, and any code that trusts the first entry still tries it. On a bridge network it fails instantly; on --network host, or in a VPC with IPv6 enabled, the same code meets a route that may or may not drop.
2. Whether your container has IPv6 is a property of the network it is attached to, not of the image, not of the application, and not of your intent. The same probe image that behaves on one network silently mis-attributes on another. That is the definition of a configuration that fails in the direction you cannot see.

The uncomfortable generalisation: a check's behaviour is a property of where it runs. Anything that reports a verdict without recording where it stood has produced a claim nobody can check.

"Just raise the timeout" is the wrong repair

When a check times out against a healthy site, the instinct is to raise the timeout so it "has time". That makes the false report slower. It does not make it truer, and it makes the page arrive later than it otherwise would.

The timeout is not the defect. The defect is that one string covers two situations with different owners:

Connection refused is the one people get wrong most often. It is an answer. It means the target is alive and declined, which is a state no amount of probe repair will change. Reporting a refusal as an outage is both wrong and expensive, because it sends somebody to look at the wrong machine.

How to fix it

Three rules, in order of how much they buy you.

1. Never merge the outcomes. Keep "connected", "answered no", and "said nothing" as three separate values all the way through, from the socket to the alert label. The moment they share a variable, every downstream consumer inherits the ambiguity.

2. Give each outcome its own exit code. A monitor reads a code, not a sentence. Codes are what let you say "exit 3 should not wake anybody up" — because exit 3 means the probe is blind, not that the service is dead.

3. Measure from a second vantage. This is the only honest way to answer "is it the target or the path?". Silence from one vantage while the other reaches the target is a path problem, and it names the broken path. Silence from everywhere points at the target, with the caveat that a shared upstream produces the identical picture — so say so rather than pretending the question is closed.

One measurement can never separate the two. That is not a limitation of a tool, it is a property of what a single vantage knows.

The tool

I extracted this from the audit tooling I maintain and packaged it as two files with no dependencies, because the machine where things are broken is frequently a machine where you cannot install anything:

https://github.com/dafili/alertpath-cli

git clone https://github.com/dafili/alertpath-cli cd alertpath-cli python3 -m alertpath --check-host your-service.example.com 

Exit codes are the interface:

Exit 3 and 5 are the interesting ones, because they are the codes that tell you not to page. Any checker that cannot express "the instrument is blind" will eventually use "the patient is dead" to say it.

Second vantage, with nothing installed on the far side — it ships the same source over ssh and runs it there, so both measurements come from the same code:

$ alertpath --check-host example.com --via probe-eu --compare example.com:443 -> OK vantage : localhost example.com:443 -> OK vantage : ssh:probe-eu compare -> OK  reached : localhost, ssh:probe-eu 

And when the far side cannot be reached at all, it refuses to conclude anything:

compare -> INCOMPLETE  No measurement from ssh:probe-eu, so there is nothing to compare against. This says nothing about the target -- fix the vantage and run again. 

I shipped that wrong the first time. Falling back to "the local vantage says OK, so everything is fine" is the same defect one level up: a missing measurement is not agreement, it is an unfinished experiment. A tool built to catch premature verdicts has to be able to catch its own.

What I am not claiming

I measured three vantages and one deliberate blackhole. I did not measure AWS or GCP — I do not have accounts there, and inventing a number for them would make this post worth exactly as much as the DOWN (timeout) it is complaining about.

The mechanism does not depend on the provider, though. It depends on three things being true, and you can check all three on your own infrastructure in under a minute:

1. Does your target publish AAAA records? python3 -c "import socket;print([i[4][0] for i in socket.getaddrinfo('your-host',443,family=socket.AF_INET6,type=socket.SOCK_STREAM)])"
2. Does the vantage your monitor runs from have a working IPv6 route to it?
3. Which family does your resolver hand back first?

If (1) is yes, (2) is no-or-unsure, and (3) is IPv6, you have the exact configuration that produced a 48-second timeout against a site that was up. You do not need AWS to have this problem. You need a dual-stack hostname and one probe standing in the wrong place.

I build audit tooling for Linux servers, and this is the smallest piece of a pattern I keep meeting in different costumes: a check that reports success while the thing it was supposed to verify is broken. The checker works. The answer is about a different question. The report names the wrong party.

alertpath is MIT and free, and it is meant to be enough on its own. The paid tool I maintain applies the same idea to nine more checks on a single server — disk, inodes, backup freshness, systemd units, memory, load, time sync, cron — without an agent: https://vaultcheck.dafili.cn.

If you find a case where these outcomes still collapse into one, that is a bug and I want the exit code: open an issue.

Reproducing everything in this post

# the blackhole demo (needs IPv6 routing on the host, e.g. --network host) printf '127.0.0.1\tlocalhost\n104.20.23.154\tdemo.internal\n2001:db8::1\tdemo.internal\n' > /tmp/hosts.demo docker run --rm --network host -w /src -v /tmp/hosts.demo:/etc/hosts:ro \ -v "$PWD":/src:ro python:3.11-slim \ python3 -m alertpath --check-host demo.internal --timeout 10 # the "no route" shape, for contrast docker run --rm -w /src -v "$PWD":/src:ro python:3.11-slim \ python3 -m alertpath --check-host example.com 

Two container networks, one tool, and two different answers: 3 (the probe is blind) on the host network, 0 (degraded but honest) on the bridge. Nothing about the image changed. Only where it stood did.