The perf job failed on a change that couldn't have made anything slower. I reran it and it passed. It happened again on the next one.
That is the moment a performance budget stops being useful: nobody trusts a red build that goes green when you click "re-run". I could have loosened the budget. Instead I kept it and made each number say whether the app or the machine was slow. This is how the setup works.
What gets measured
The app is an issue tracker that keeps a local copy of a workspace in IndexedDB and syncs over WebSockets. Two things must feel instant: reopening a big workspace, and seeing someone else's edit.
So the perf test fills a workspace with 2,000 issues, using the same seed script I use in development (sign-up over HTTP, then mutations through the sync gateway). Then it times two things, five runs each:
- Open from cache. Reload the page, with the copy already in IndexedDB, and read how long until the issue list is on screen.
- Edit propagation. Two browsers, two people, same team. One renames an issue the other is looking at. Time from the Enter key until the new title is in the other page.
It fails when the median of the five goes over the budget. Not the worst run and not the mean: one slow run on a shared runner shouldn't fail the build, and a real regression moves all five.
The perf tests run in their own CI job with one Playwright worker and no retries, so nothing else shares the runner while they time things, and a slow run is a failure rather than a flake.
Measuring from inside the page
A Playwright locator poll adds round trips to a number that is a few hundred milliseconds. So the app records a user timing mark when the list has painted (simplified):
export function markIssueListShown(count: number): void { if (performance.getEntriesByName(MARK_ISSUE_LIST_SHOWN).length > 0) return; // After the next frame, so the mark means painted rather than merely committed. requestAnimationFrame(() => { performance.mark(MARK_ISSUE_LIST_SHOWN, { detail: { count } }); }); } On a reload, the mark's startTime is the time since navigation start, which is exactly "how long opening took". The test just reads it. One warm-up reload comes first, uncounted, to fill the HTTP cache and the service worker.
For propagation, both browsers run on one machine, so Date.now() is the same clock in both. The reader page waits with a MutationObserver and resolves with the time the title shows up. The writer stamps its own Enter keydown from inside the page, again so Playwright's round trip isn't counted:
await writer.page.evaluate(() => { document.addEventListener( "keydown", () => { window.__savedAt = Date.now(); }, { capture: true, once: true }, ); }); await titleField.press("Enter"); The numbers and why
The first CI run on GitHub's runners gave a median of 536 ms to open and 14 ms to propagate. My laptop: about 650 ms and 15 ms.
The budgets are 1,500 ms and 100 ms. Opening gets about 3x its measured time: loose enough for a noisy runner, tight enough that opening taking three times as long fails. Propagation is so quick that runner noise dominates it, so it gets a looser multiple, and catches a change that makes it several times slower.
Every run writes its numbers to the job summary, so when they show a lot of room, I tighten. The first budgets were a guess, and the first CI run is what set them. Later, when rendering only the rows near the screen took a cached open from about 1,100 ms to about 340 ms on my machine, the same commit tightened the bulk-edit budgets rather than keeping the slack.
The comment at the top of the budgets file ends with the rule: never loosen a budget to get a change through; find out what got slower. It's in the repository's contributor guidelines too, so an AI coding agent working there reads it as well. A budget that moves whenever it's inconvenient is a number nobody believes.
The machine probe
The rule says don't loosen the budget, so I needed something that could tell me, after the fact, that the runner was the problem.
machine-probe.mjs is a separate Node process that shares nothing with the app but the machine. It does two things:
- It wakes every 10 ms on a timer and records how late it was. A timer that wakes 80 ms late means the CPU was taken away.
- Every 100 ms it writes 4 KB to a file and calls
sync, and times that. A slow synced write means the disk stalled.
Anything over 50 ms counts as a stall, and is appended as a line of JSON with a timestamp:
let woke = performance.now(); setInterval(() => { const now = performance.now(); const late = now - woke - WAKE_MS; woke = now; if (late > STALL_MS) note("timer", late); }, WAKE_MS); The spec starts the probe before the measurements and stops it after, then matches stalls to each measurement's time window. Each result line in the log and the job summary ends with the machine's story, in the form machine: 3 stalls, woke up to 140 ms late, wrote to disk in up to 90 ms, or machine: no stalls.
Over budget with stalls: look at the runner. Over budget with no stalls: look at the app. The probe doesn't change the verdict. A run over budget still fails. It only changes what I do next.
What went wrong along the way
The edit test was timing the handshake. After reloading the writer's page, it started editing as soon as the title field showed. But that field renders from the local copy, before the page's WebSocket is up, and an edit made then waits for the socket. So the first run also timed the connection setup. Now the test waits for the sync badge to say "Live" before it starts the clock. That fix and the probe landed in the same change, and the budgets stayed as they were.
A stall is noted when it ends. One that spans the end of a measurement is written after the measurement is over. So the test reads the stalls file only once the probe has exited, and the probe finishes its write in progress and its notes before it quits. It also never lets two disk writes pile up, because a slow one is exactly what it is looking for.
The job also keeps Playwright's trace of a failed run for seven days, so a red run can be opened rather than guessed at.
Was it worth it?
Mostly yes. I don't have a statistical story: five runs and a median is crude, and the probe is a hint, not proof. A stall on a runner doesn't prove the app is innocent, and no stall doesn't prove it's guilty. But a red build now comes with a line about the machine next to it, and re-running it is no longer my only move.
How do you keep timing tests honest on shared CI runners? Do you retry, widen the budget, or do something else?
The test and probe are open source: apps/web/perf/machine-probe.mjs in the tracker's repository.
I drafted this with an AI assistant (Claude), which also wrote much of the code it describes. I checked the facts and numbers against the code.
