Your Playwright suite is lying to you (and retries are how)

`
Every team I've talked to about end-to-end tests has the same ritual.

The PR goes red. Someone opens the CI log, sees Timeout 30000ms exceeded, shrugs, and clicks Re-run failed jobs. It goes green. Merge.

Do that enough times and two things happen. First, people stop reading red builds at all. Second, the day a real regression shows up as a timeout, it gets the same shrug and the same re-run — and if it's intermittent, it merges.

The suite isn't failing. It's lying. A red check stops meaning "you broke something" and starts meaning "something, somewhere, was slow." I spent the last few months building tooling to make that verdict honest again, and this post is what I learned.

Retries hide the thing you need to see

Playwright's retries: 2 is the obvious fix, and it does make CI greener. But look at what it does to your signal:

• A test that fails, then passes on retry, is reported as flaky in the HTML report — and passed by the exit code. CI only reads the exit code.
• Nobody aggregates those flaky passes across runs. The test that has flaked on 30% of runs this month looks identical to the one that flaked once.
• A real race condition your users will hit 1 in 5 times looks exactly like a flake. Retries launder it.

So you have two bad options: retries off (every flake blocks a merge, people learn to ignore red) or retries on (every intermittent bug ships, people never see red). What I actually wanted was a third rule:

Block the merge only on a failure you can reproduce. Everything else gets reported, counted and owned — but doesn't block.

Rule 1: rerun once, block only what reproduces

The core idea is boring on purpose. Run the specs a PR touches. If something fails, run those same specs again. Then:

The important part is that a flake isn't forgotten — it's recorded with its error text and counted. It just doesn't get to hold your merge hostage.

I wired this into an open-source CLI called QAFind (MIT). In CI it's one command:

bash
npx qafind verify-merge --run --base origin/main --head HEAD


--run picks the affected Playwright specs from the git diff, runs them with the JSON reporter, reruns the failures once, and exits 1 only if a regression reproduced. If you already produce Playwright JSON reports in your pipeline, you can hand it both reports instead (--results first.json --rerun-results rerun.json).

I tested it on a tiny demo suite where the same assertion fails in two specs. In one it fails twice; in the other it fails then passes:

text
regression.spec.ts failed → failed 1 reproduced regression exit 1
flake.spec.ts failed → passed 1 flake, 0 regressions exit 0


Same error message. Different verdict. The only thing that changed the outcome was whether it reproduced — which is exactly the question a human asks before clicking re-run.

I also ran it against a third-party open-source Playwright suite I don't own (a page-object example that drives a public demo shop). Its add-to-cart spec failed on both runs, so the verdict was "reproduced regression, merge blocked." No special setup for that repo — which brings me to the second rule.

Rule 2: don't make me rewrite my specs

Every testing tool I evaluated wanted me to adopt its runner, its DSL or its cloud. I have hundreds of specs. I'm not porting them.

So the attach step wraps your config instead of replacing it:

bash
npx qafind attach .
npx playwright test --config .qafind/playwright.qafind.config.ts


attach generates a wrapper config that imports your playwright.config.ts, keeps your reporters, and appends one more reporter that writes a small JSON manifest per run. Your original config and specs are read, never written. In CI you change one flag.

The fiddly part, if you ever try this yourself: Playwright resolves paths relative to the config file that was loaded. Put the wrapper in a subfolder and suddenly testDir, outputDir, globalSetup, storageState and webServer.cwd all point to the wrong place and Playwright cheerfully finds zero tests. The wrapper has to rebase every one of those. That one cost me a weekend.

Rule 3: when it flakes, say why — and say how sure you are

Once each run leaves a manifest, you can aggregate across runs:

bash
npx qafind flake .qafind/reports --markdown


text
| Test | Runs | Flakes | Cause | Quarantine |
| cart fixture returns totals | 3 | 1 | network_fixture | yes |
| login selector exists | 3 | 1 | selector_drift | yes |
| slow recommendations endpoint | 3 | 1 | timing | yes |
| known checkout total mismatch | 3 | 3 | app_race | no |


Look at the last row. It failed on every run. That's not a flake — it's a stable failure, so it's never recommended for quarantine. A tool that auto-quarantines anything red is just a slower way to delete your tests.

I want to be upfront about how the "Cause" column works, because this is where tools overclaim. It's deterministic matching on the recorded error text: net::ERR_ → network, a strict-mode locator violation → selector drift, a bare timeout → timing, an expect mismatch → app race. If you run with --determinism, it also records a per-test request ledger, which can confirm a network cause (a transport failure was really recorded) or contradict it (all requests succeeded, so it wasn't the network).

On a seeded synthetic benchmark of 400 tests × 5 runs, that's 87.8% accurate on error text alone and 90.3% with the request ledger. Synthetic means I know the right answer by construction — it measures the classifier, not your suite. It's a recommendation for a human, not proof of causality. Please verify before you quarantine anything.

Bonus: make the flakes go away instead of labelling them

Labelling flakes is good. Not having them is better. The two biggest sources I see are the network and the clock, so attach --determinism can:

• record every HTTP request/response per test, then replay them byte-for-byte on later runs — and fail loudly on a missing fixture rather than silently hitting the network;
• freeze the browser clock before navigation, so "3 minutes ago" and end-of-month date logic stop flipping.

Again with no spec changes: a generated tsconfig path alias points @playwright/test at a shim that adds auto-use fixtures. Your import { test } from '@playwright/test' keeps working. (Limits: WebSockets and service workers aren't recorded yet.)

What I'd tell my past self

1. The exit code is the product. Your HTML report can be beautiful; CI only reads one bit. Decide precisely what flips it.
2. "Passed on retry" is data, not success. Keep it, count it, give it an owner.
3. Reproduction is the cheapest honest test of a failure. One rerun answers most "is this real?" questions before a human looks.
4. Never quarantine a test that fails every time. That's a bug report.
5. Say how sure you are. A cause label with no confidence attached will eventually be wrong in a way that costs someone a day.

Try it

The CLI is MIT and runs entirely locally — no account needed for attach, flake or verify-merge:

bash
npx qafind attach .
npx playwright test --config .qafind/playwright.qafind.config.ts
npx qafind flake .qafind/reports --markdown


Repo: github.com/katesaikishore/QAFind. It's Playwright-only today. If you run it on your suite, I'd love to hear where the cause labels are wrong — real-suite misclassifications are the most useful bug reports I can get.

How does your team handle flaky e2e tests today — retries, quarantine, or just vibes? Tell me in the comments.`