Originally published on the AWS Builder Center: https://builder.aws.com/content/3Ly81UTUan33wAxL0t5mZCVgCTl/implementing-an-slo-and-an-error-budget-you-can-actually-gate-on-for-dollar0
Repo: https://github.com/PradeepKandepaneni/golden-path-slo

Most SLOs live in a Google Doc. Someone writes "99.9% availability" in a reliability review, everyone nods, and nothing downstream ever checks whether the number is true. An SLO you don't measure isn't an objective, it's a wish.

So I built the smallest version of a real one. The service from the earlier golden-path work, now instrumented, with an SLO definition, a check that measures it, an error-budget reading, and CI that fails if the gate can't catch a violation. It runs as plain binaries on free infrastructure, so you can clone it and watch an SLO pass and then break. Repo: github.com/PradeepKandepaneni/golden-path-slo. It's the fourth in a series: ship it, survive a zone outage, secure the supply chain, and now measure it against a promise.

The two signals, and nothing more

An availability-and-latency SLO needs exactly two things from the service: how many requests succeeded, and how long they took. Everything else is decoration. So the instrumentation is deliberately small, a request counter by status class and a latency histogram, exposed at /metrics in Prometheus text format using only the Go standard library:

http_requests_total{status="2xx"} 767
http_requests_total{status="5xx"} 33
http_request_duration_seconds_bucket{le="0.025"} 800
http_request_duration_seconds_count 800

Real systems use OpenTelemetry or the Prometheus client library feeding a collector, and in production you should. But reaching for a framework first tends to turn "implement an SLO" into "stand up an observability stack," and the SLO never gets written. The signals are simple; the discipline is in what you do with them.

Measure the promise where the user is

Here's a choice that matters more than it looks. My SLO check, slocheck, measures success and latency from the client's side, by probing the service and timing the responses, not by reading the server's own counters.

That's deliberate. An SLO is a promise to whoever calls the service. If the server thinks it returned 200s quickly but a connection pool or a slow proxy made the caller wait, the server's counters will tell you a comfortable lie. You measure where the user is. The server's /metrics are still exposed, scraping and dashboards still matter, but the pass/fail verdict comes from the client's experience, because that's what the promise was about.

slocheck drives a burst of traffic, records every result, and computes the two numbers the SLO is made of:

── SLO report ──────────────────────────────
requests : 500
availability : 100.000% (target 99.000%)
p95 latency : 11.0 ms (objective ≤ 150 ms)
error budget consumed : 0.0% of the 1.00% allowance
────────────────────────────────────────────
SLO MET
The error budget is the useful part

Availability targets get the attention, but the error budget is what changes behaviour. If your availability target is 99%, then 1% of requests are allowed to fail. That 1% is a budget, and it's a resource you spend.

The reading that matters isn't "are we at 99%," it's "how much of the budget have we burned." slocheck reports exactly that. Here's the same service with an 8% failure rate injected:

── SLO report ──────────────────────────────
requests : 500
availability : 91.750% (target 99.000%)
p95 latency : 10.9 ms (objective ≤ 150 ms)
error budget consumed : 825.0% of the 1.00% allowance
────────────────────────────────────────────
SLO VIOLATED: availability 91.750% below target 99.000%

825% of the budget consumed. Not "we dipped below target", we spent eight times the entire allowance. That framing is what makes an error budget a decision tool: a healthy budget means you have room to ship risky changes fast; a burned one means reliability work comes before features. The number turns an abstract reliability argument into an accounting one.

Prove the gate catches a violation

A reliability check you've only ever watched pass is a check you're trusting on faith. So the CI pipeline runs slocheck twice: once against a healthy scenario, which must pass, and once against a deliberately degraded one, where the job fails unless the gate reports the violation:

yaml

  • name: Prove the gate CATCHES an SLO violation (must be caught) run: | if ./bin/slocheck -n 500 -delay-ms 10 -fail 0.08 \ -target-availability 0.99 -latency-objective-ms 150; then echo "FAIL: slocheck passed a run that breached the SLO." exit 1 fi echo "Gate correctly caught the SLO violation."

This is the same pattern that runs through the whole series, a bad pod the cluster must reject, a known-vulnerable image the scanner must block, and now an SLO breach the check must catch. Testing that your guardrail fires is as important as the guardrail itself. The whole pipeline runs as plain binaries in the runner, no cluster, no external tools, no dependencies, so it's fast and there's almost nothing to break.

What this deliberately doesn't do

A measured burst is not a 28-day window. Real SLOs are evaluated over a rolling window against real production traffic, often a month. slocheck reads a compressed burst. The mechanics are right, the timescale is a demo.

Stdlib metrics are not an observability platform. For anything real you want OpenTelemetry, a collector, and a time-series database so the error budget is computed continuously, not in a one-shot probe. The exposition format here is compatible with that world; the repo just isn't that world.

The budget informs the decision; it doesn't make it. A burned budget is a signal to slow down and invest in reliability. Whether you freeze features, page someone, or accept the risk is a human call. Wiring CI to hard-block every deploy the instant a budget dips is how you get teams that resent the SLO and route around it.

Takeaways

An unmeasured SLO is a wish. The gap between "we have a 99.9% SLO" and a check that actually computes it against observed traffic is the entire difference between reliability theatre and reliability engineering.

Measure the promise from the caller's side. Server-side counters can hide latency and failures the user actually sees. The SLO is about their experience, so measure there.

Lead with the error budget, not the target. "How much budget have we burned" is a better operational question than "are we above the line," because it tells you how much risk you can still afford to take.

The repo is public and runs for $0: github.com/PradeepKandepaneni/golden-path-slo. Clone it, break the SLO, and watch the budget burn.