I run DownForReal, a hosted version of Uptime Kuma. Disclosure up front: it is my product, it costs $10 a month, and the fork is public on GitHub under the MIT licence. This post is about one piece of plumbing in it, the part that checks a down alert from three regions before the alert goes out.

The problem is one vantage point

Uptime Kuma checks from wherever it is installed. That is the only view it has. If the network path between the Kuma box and your site breaks for a minute, Kuma sees a dead site and sends the alert. Your site was fine. You still got paged.

This is the oldest open request in the project. Issue #1915 asked for multi-location checks in 2021 and was closed as not planned. The usual workaround is a second Kuma install on another network, synced by hand. I wanted the check inside the alerting path instead.

How the vote works

Every customer gets their own Kuma instance with its own database, running in a container with hard limits (512 MB of memory, half a CPU). Nothing about checking changes. The instance monitors exactly the way stock Kuma does.

The change sits in the notification path:

  1. The instance reports a monitor down and fires its webhook.
  2. A small adapter service receives the webhook. Kuma 2.x sends its own heartbeat payload shape, so the adapter translates it into the confirm service's input: which monitor, which URL, which tenant.
  3. The confirm service asks three probe servers to check the same URL at the same moment. The probes sit in New York, Frankfurt, and Singapore.
  4. The alert goes out only if at least 2 of the 3 probes also get a failure.
  5. If the vote fails to confirm, the alert is suppressed, and the suppression is written back into the customer's own instance as a log entry they can read. Suppressing silently would make the service impossible to audit, so we do not do it.

A real outage looks different from a local network blip under this test. If the site is actually down, all three probes fail and the alert fires, a few seconds later than stock Kuma would send it. If one region has a routing problem, one probe fails and two succeed, and nobody gets paged.

The fleet that runs it

Five small servers, all DigitalOcean, $54 a month total:

  • One control host: billing, provisioning, the confirm service, the adapter, and the status page mirrors.
  • One tenant host: the customer containers.
  • Three probe hosts, one per region, each doing a single job: answer health checks and run confirmation checks on request.

Status pages get copied off the tenant host to the control host on a schedule. A status page that lives on the same box as the outage is decoration, so the mirror has to live somewhere else. There is also a meta monitor watching the fleet itself, with its dead man's switch heartbeat going to a service outside this fleet, because a monitoring system that fails quietly is worse than one that fails loudly.

What it costs to run the vote

The probes are the main added cost over a plain hosted Kuma instance. Three small droplets at $6 a month each run around the clock whether any alert ever fires or not. The confirm service itself is a few hundred lines of Node with no dependencies beyond the standard library. At 100 customers our measured planning figure is about $2 per customer per month all in, against the $10 price. The full cost breakdown, with the server math shown, is in the cost article on the site.

Limits, stated plainly

  • The vote adds seconds to a real alert. Confirmation takes as long as the slowest probe check.
  • Two of three probes agreeing is a quorum, not proof. A target can be down for one region only, and the vote will call that up.
  • One container per customer costs more than a shared multi-tenant design and will not scale past a few hundred customers without a rebuild.
  • The probes live in three regions on one cloud provider. A provider-wide incident can take the vote out with the fleet, which is why the meta monitor's heartbeat leaves the fleet entirely.

The exit is part of the design

The fork is public, the data is a SQLite file inside the customer's own instance, and anyone can take both and self-host the whole thing. If DownForReal shut down tomorrow, every customer could be running their own copy the same day. Hosted monitoring asks for a lot of trust. Portability is how I would rather earn it.

If you self-host Kuma today, I am interested in how you handle the false page problem. A second install, a heartbeat trick, or just living with it?