Synthetic Checks & SLOs for the Paths Nobody Clicks

A client's status page showed 99.98% uptime for the quarter. In the same quarter, their password reset flow was broken for 11 days. The homepage returned 200, the API health endpoint returned 200, and the load balancer was happy. But a queue worker that sent reset emails had been pinned to an old mail driver after a config change, and the reset page itself rendered fine; it just never sent anything. Nobody alerted because nobody clicks "forgot password" often enough for error rates to move. We found out from a support ticket written in capital letters.

I wrote this with help from AI tools and checked every script, config and query myself. I am a Principal Software Engineer in Karachi with 9+ years across PHP/Laravel, Vue, Node and .NET microservices, and at Loop Origin we run and maintain production servers for SaaS clients. This is Part 6, the finale of the series Silent Failures in Production SaaS. This last part is the safety net under the earlier five: actively exercising the paths real users rarely touch and measuring them against an objective.

📚 Series Navigation — Silent Failures in Production SaaS

Part 1: Cron Jobs That Never Ran: Heartbeats & Dead-Man Switches for SaaS

Part 2: Webhooks That Stopped Arriving: Gap Detection & Reconciliation

Part 3: Email That Never Landed: SPF/DKIM/DMARC Drift & Bounce Pipelines

Part 4: Config & Feature Flags That Quietly Diverged Across Environments

Part 5: Backups That Never Restored: Automated Restore Drills

Part 6 (This Post, Finale): Synthetic Checks & SLOs for the Paths Nobody Clicks

Related earlier posts: Production Observability on a Budget · Observability for Async Work: Correlation IDs & Job Traces

How do you catch failures on paths real users rarely use?

Direct answer: Run synthetic checks: scheduled scripts that perform the real user journey end to end (request a password reset, receive the email, follow the link, export an invoice) using a dedicated test tenant. Record each run as pass or fail with timing, define an SLO per journey, and alert on error-budget burn rather than on single failures or plain uptime.

Why uptime monitoring misses the important failures

Uptime checks answer one question: does a URL respond? Low-traffic paths fail in ways that never touch that question. Here is how the three common layers compare on the failures from this series:

Failure	Uptime ping	Error-rate alert on real traffic	Synthetic journey check
Password reset email never sent	Misses (page renders)	Misses (no exception, few users)	Catches (no email within 2 min)
SSO login broken for one IdP	Misses	Misses until a big customer logs in	Catches (test IdP login fails)
Invoice PDF export times out	Misses	Maybe, if volume is high	Catches (export > 30 s)
Webhook endpoint returns 200 but drops data	Misses	Misses	Catches (round-trip record not found)
Signup blocked by a new CAPTCHA config	Misses	Misses (users just leave)	Catches (signup step fails)
Homepage down	Catches	Catches	Catches

The bottom row is the only one uptime monitoring is designed for, and it is also the failure you would notice anyway.

Step 1: pick the journeys, not the endpoints

I list journeys by asking two questions: if this broke for a week, would it cost money or trust? And does it get fewer than a few hundred uses a day? Anything that answers yes to both gets a synthetic check. For a typical B2B SaaS that is:

Signup and email verification
Password reset end to end
SSO login for each identity provider type you support
Invoice or report export
Inbound webhook round trip
Plan upgrade on a test payment method
Account deletion or data export request (often a legal obligation)

Use a dedicated synthetic tenant with clearly labelled data, excluded from analytics and billing. Never point checks at real customer accounts.

Step 2: write the check as a real browser journey

For flows that involve a UI, I use Playwright. This is a trimmed version of our password reset check. The test mailbox is a catch-all on a subdomain we control, read through the mail provider's inbound API:

// synthetics/password-reset.spec.ts import { test, expect } from '@playwright/test'; import { waitForEmail } from './mailbox'; test('password reset end to end', async ({ page }) => { const email = process.env.SYNTH_USER_EMAIL!; const started = Date.now(); await page.goto(`${process.env.APP_URL}/forgot-password`); await page.getByLabel('Email').fill(email); await page.getByRole('button', { name: /send reset link/i }).click(); await expect(page.getByText(/we have emailed/i)).toBeVisible(); const msg = await waitForEmail({ to: email, subject: /reset/i, sinceMs: started, timeoutMs: 120_000 }); const link = msg.html.match(/href="([^"]+reset-password[^"]+)"/)?.[1]; expect(link, 'reset link present in email').toBeTruthy(); await page.goto(link!); await page.getByLabel('Password', { exact: true }).fill(process.env.SYNTH_USER_PASSWORD!); await page.getByLabel('Confirm Password').fill(process.env.SYNTH_USER_PASSWORD!); await page.getByRole('button', { name: /reset password/i }).click(); await expect(page).toHaveURL(/dashboard|login/); });

The email step is what would have caught the 11-day outage. A check that stopped at "we have emailed you" would have passed every single run.

For API-only journeys, a plain script that posts a signed test event and polls until the record exists is enough.

Step 3: record results as metrics, not just alerts

Each run writes one row: journey, pass or fail, duration, timestamp. In Laravel I keep a tiny table and a scheduled command, but pushing to Prometheus via a Pushgateway works just as well:

// app/Console/Commands/RunSynthetics.php (handle method, trimmed) foreach (config('synthetics.journeys') as $journey => $cmd) { $start = microtime(true); $result = Process::timeout(180)->run($cmd); SyntheticRun::create([ 'journey' => $journey, 'ok' => $result->successful(), 'duration_ms' => (int) ((microtime(true) - $start) * 1000), 'output' => Str::limit($result->errorOutput() ?: $result->output(), 2000), ]); }

// routes/console.php Schedule::command('synthetics:run') ->everyFiveMinutes() ->withoutOverlapping() ->onOneServer() ->pingOnSuccess(config('services.heartbeats.synthetics'));

Note the heartbeat. As Part 1 argued, a monitor that silently stops running is the worst silent failure of all, so the synthetic runner itself reports in.

Step 4: define SLOs per journey and alert on burn rate

A single failed run is often noise: a slow mail provider, a deploy in progress. Paging on every failure trains people to ignore pages. Instead I set an SLO per journey and alert when the error budget burns too fast.

Example: password reset should succeed in 99% of synthetic runs over 30 days. At one run every five minutes, that is 8,640 runs and an error budget of about 86 failures. The query for the current failure ratio:

SELECT journey, 1 - AVG(ok::int) AS failure_ratio_1h, PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY duration_ms) AS p95_ms FROM synthetic_runs WHERE created_at > now() - interval '1 hour' GROUP BY journey;

Alerting rules I use as defaults, borrowed from the multi-window burn-rate approach in Google's SRE workbook:

Alert	Condition	Meaning	Action
Fast burn	Failure ratio over 1 h > 14.4× budget rate, confirmed over 5 min	Budget gone in about 2 days	Page on-call
Slow burn	Failure ratio over 6 h > 6× budget rate, confirmed over 30 min	Budget gone in about 5 days	Ticket, fix this week
Latency	p95 duration > 2× baseline for 1 h	Degrading, not broken	Ticket
Silence	No synthetic heartbeat for 15 min	Monitoring is down	Page on-call

With a 99% target the budget rate is 1%, so "fast burn" means more than about 14% of runs failing in the last hour. Two consecutive failed runs of a five-minute check will usually trip it, which is the behaviour you want for a fully broken path.

Step 5: keep the checks honest

Synthetic checks rot like any other code. Run them against production from outside your network so DNS, TLS and CDN problems show up, keep the synthetic user's credentials in your secret manager, fix or delete flaky checks instead of muting them, and add a check for every incident where a customer told you first.

We set up and run these as part of our server setup and maintenance work. If you want help choosing journeys and SLO targets for your own product, get in touch with us.

Diagnostic checklist

[ ] You have a written list of low-traffic, high-value journeys.
[ ] Each journey has a synthetic check that completes the whole flow, including email or webhook steps.
[ ] Checks use a dedicated, labelled synthetic tenant excluded from analytics and billing.
[ ] Checks run against production from outside your network at least every 5–15 minutes.
[ ] Every run is stored with pass or fail and duration.
[ ] Each journey has an SLO and burn-rate alerts instead of single-failure pages.
[ ] The synthetic runner has its own heartbeat; silence pages someone.
[ ] Flaky checks are fixed or deleted within a week, never muted indefinitely.
[ ] Every "customer told us first" incident produces a new check.

FAQ

Aren't synthetic checks just end-to-end tests?

They share tooling, but the purpose differs. E2E tests run before deploy against a test environment. Synthetic checks run continuously against production to catch what changes after deploy: config, third parties, certificates and infrastructure.

Won't synthetic traffic pollute my metrics?

Only if you let it. Tag the synthetic tenant, exclude it in analytics and billing, and add a request header your logs can filter on.

How many journeys should I start with?

Three to five. Password reset, signup and your main export or integration path cover most of the silent failures I see.

What SLO target should I pick?

Start lower than you think, often 99% for synthetic journeys, then tighten once a month of data shows what is realistic. A target you constantly miss gets ignored.

Can I use a hosted service instead of building this?

Yes. Checkly, Datadog Synthetics, Grafana Synthetic Monitoring and similar tools handle scheduling and storage. The important work, choosing journeys and writing the full flow including email, is the same.

Series wrap-up

That closes Silent Failures in Production SaaS. Cron jobs, webhooks, email, config drift, backups and now unclicked paths: the common thread is that a system with no errors is not the same as a system that works. Give each critical behaviour a positive signal, alert when that signal goes quiet, and measure it against a target someone agreed to.