Playwright 1.65: A Filter That Matches Nothing Now Fails Every Shard

Playwright 1.65 has one line in its notable changes that reads like a small tidy-up: a sharded run now fails with "No tests found" when no tests match across all shards. If your CI builds a --grep or a path filter per pull request and fans it out over a shard matrix, that line decides whether a pull request that touches no tests is green or red.

I ran the same suite on 1.64.1 and 1.65.1 to see exactly which cases changed. The harness and raw results are in the PoC repo.

The setup

• 32 spec files with one test each. The tests use no browser.
• fullyParallel: true and workers: 1, so Playwright can put each test on any shard.
• @playwright/test 1.64.1 and 1.65.1, on Node 24 and macOS.
• 0, 1, 2, 4, 8 and 16 selected tests, picked by tag.
• 2, 4, 8 and 16 shards. Every shard is its own process, as it is in a CI matrix.
• Each cell with and without --pass-with-no-tests, with Playwright's own split and with playwright-shard-balancer in the config.

That is 1,464 runs. I recorded the exit code of every run and the number of tests it ran. A second script runs four real filters that can match nothing.

The numbers

Zero selected tests, failed shards out of all shards:

On 1.64.1, sharding hid the error. The unsharded run failed with "No tests found" and the same filter with --shard passed on every shard. On 1.65.1 the sharded run agrees with the unsharded one: every shard fails.

No other exit code changed. A job here is one cell of the matrix with all of its shards. With 1 to 16 selected tests, all 180 jobs had 0 failed shards on both versions, even when 15 of 16 shards had no test to run. An empty shard is still fine. A whole run with no tests is not.

Which filters hit it

The same check with four filters, as shards 1/2 and 2/2, without the flag:

For --last-failed, the last run file from the green run was still on disk, as it is when CI restores it from a cache or an artifact.

Two results here are not in the release note:

• --last-failed after a green run still passes when sharded on 1.65.1. It ran 0 tests and exited 0 on both shards. Unsharded, the same command exits 1 with "No tests found" on both versions. The new check did not apply to this flag. Without the last run file, --last-failed ran the whole suite, so this only happens when the file is restored.
• --only-changed with no changed test file exited 0 in every cell, sharded or not, on both versions. It passed before the change, so it says nothing about the new rule.

The balancer

playwright-shard-balancer plans shards through Playwright's preprocess hook, so I checked whether a shard it leaves empty trips the new rule. It does not. Version 0.1.3 gave the same exit code as Playwright's split in every cell: all shards fail on a zero-test selection, and none fail when at least one test is selected. The balancer did plan differently: it left a different number of shards empty in 36 of the paired jobs.

What I'd change in CI

• If a filter can legitimately match nothing, such as a docs-only pull request, add --pass-with-no-tests to the sharded command. It turned every cell green on both versions.
• The flag cannot tell a docs-only pull request from a broken filter. Both select nothing and both pass. If an empty selection should never happen in your pipeline, leave the flag off: on 1.65 that mistake fails the build instead of passing on every shard.
• Check your sharded jobs before you upgrade. A job that was green on 1.64 with no tests to run is red on 1.65.
• Do not use the exit code of a sharded --last-failed job as proof that tests ran. With a restored last run file from a green run, it ran nothing and exited 0.

Originally published at https://garyparker.dev/blog/playwright-165-empty-shards