A visual test that never fails is worse than none. I wanted to know what toHaveScreenshot() in Playwright lets through with its default settings, so I built one small card (heading, price, button, gradient), made eleven small changes to it and compared each change to a baseline under five option sets. The results are not what the option names suggest.

Setup: @playwright/test 1.63.0 with its Chromium headless shell, Node v24.22.3, Linux, run on 2026-11-13. The page is served from a local HTTP server and the font is DejaVu Sans, so nothing depends on a remote site. Viewport is 800 x 500.

The test

Config first:

module.exports = { testDir: './tests', workers: 1, reporter: 'line', snapshotPathTemplate: '{testDir}/__shots__/{arg}{ext}', use: { viewport: { width: 800, height: 500 } }, };

Save the next file as tests/vr.spec.js. It builds the baseline for the whole page and for the .card element, then replays each variant against each option set and prints pass or the number of differing pixels. The last two tests are used further down.

const { test, expect } = require('@playwright/test'); const http = require('http'); const html = (v) => `<!doctype html><meta charset=utf-8><style> body{margin:0;font:16px/1.4 DejaVu Sans,sans-serif;background:#fff} .card{margin:30px;padding:24px;width:420px;border:1px solid #ddd;border-radius:6px} h1{font-size:22px;margin:0 0 8px;color:${v.h1 || '#1a1a1a'};font-weight:${v.weight || 600}} p{margin:0 0 16px;color:#444} button{background:${v.btn || '#0b5fff'};color:#fff;border:0;padding:10px 18px;font:inherit;border-radius:${v.radius || 6}px} .img{width:200px;height:100px;margin-top:16px;background:linear-gradient(90deg,${v.g1 || '#4a90d9'},#e8f0fb)} </style><div class=card style="${v.shift ? 'padding-left:25px;width:419px' : ''}"> <h1>Order summary</h1><p>Total ${v.total || '$1,240.00'}</p> <p>Three items, shipping included.</p><button>Pay now</button><div class=img></div></div>`; const variants = { 'h1 colour +8 levels (#1a1a1a to #222)': { h1: '#222222' }, 'h1 colour +25 levels (#1a1a1a to #333)': { h1: '#333333' }, 'button blue #0b5fff to #0b5ffa': { btn: '#0b5ffa' }, 'button blue #0b5fff to #0a58e0': { btn: '#0a58e0' }, 'button blue to red': { btn: '#d32f2f' }, 'card text shifted 1 px': { shift: 1 }, 'one digit: $1,240.00 to $1,248.00': { total: '$1,248.00' }, 'button radius 6 to 8': { radius: 8 }, 'button radius 6 to 18': { radius: 18 }, 'h1 bold to regular': { weight: 400 }, 'gradient start #4a90d9 to #4a90e0': { g1: '#4a90e0' }, }; const options = { 'default': {}, 'threshold 0': { threshold: 0 }, 'threshold 0.05': { threshold: 0.05 }, 'maxDiffPixels 100': { maxDiffPixels: 100 }, 'maxDiffPixelRatio 0.01': { maxDiffPixelRatio: 0.01 }, }; const targets = { 'page': (page) => page, 'card': (page) => page.locator('.card'), }; let server, port; test.beforeAll(async () => { server = http.createServer((req, res) => { const v = JSON.parse(decodeURIComponent(req.url.slice(1))); res.setHeader('content-type', 'text/html'); res.end(html(v)); }); await new Promise((r) => server.listen(0, '127.0.0.1', r)); port = server.address().port; }); test.afterAll(() => server.close()); const open = (page, v) => page.goto(`http://127.0.0.1:${port}/${encodeURIComponent(JSON.stringify(v))}`); test('baseline', async ({ page }) => { await open(page, {}); for (const [name, target] of Object.entries(targets)) { await expect(target(page)).toHaveScreenshot(name + '.png'); } }); test('matrix', async ({ page }) => { test.setTimeout(180_000); // failing comparisons retry until their timeout for (const [tname, target] of Object.entries(targets)) { console.log(`\n${tname} screenshot | ` + Object.keys(options).join(' | ')); for (const [vname, v] of Object.entries(variants)) { await open(page, v); const cells = []; for (const o of Object.values(options)) { try { await expect(target(page)).toHaveScreenshot(tname + '.png', { ...o, timeout: 3000 }); cells.push('pass'); } catch (e) { const m = /(\d+) pixels/.exec(e.message); cells.push(m ? `FAIL ${m[1]} px` : 'ERROR'); } } console.log(vname + ' | ' + cells.join(' | ')); } } }); test('noise: same page 20 times at threshold 0', async ({ page }) => { for (const [name, target] of Object.entries(targets)) { let fails = 0; for (let i = 0; i < 20; i++) { await open(page, {}); try { await expect(target(page)).toHaveScreenshot(name + '.png', { threshold: 0, maxDiffPixels: 0, timeout: 3000 }); } catch { fails++; } } const box = await page.locator(name === 'card' ? '.card' : 'body').boundingBox(); console.log(`${name}: ${fails}/20 failed; ${name === 'card' ? Math.round(box.width) + 'x' + Math.round(box.height) : '800x500'} px`); } }); test('toHaveCSS sees what pixel thresholds miss', async ({ page }) => { const checks = [ ['button blue #0b5fff to #0a58e0', { btn: '#0a58e0' }, 'background-color', 'rgb(11, 95, 255)'], ['button radius 6 to 8', { radius: 8 }, 'border-radius', '6px'], ]; for (const [name, v, prop, expected] of checks) { await open(page, v); try { await expect(page.locator('button')).toHaveCSS(prop, expected, { timeout: 1000 }); console.log(name + ': pass'); } catch (e) { console.log(name + ': FAIL, got ' + await page.locator('button').evaluate((el, p) => getComputedStyle(el)[p], prop)); } } });

Run it twice, once to write the baselines and once to compare:

npx playwright test -g baseline --update-snapshots npx playwright test -g matrix

What came back

Full-page screenshot, 800 x 500 = 400 000 pixels. "px" is the number of pixels Playwright reported as different.

page screenshot | default | threshold 0 | threshold 0.05 | maxDiffPixels 100 | maxDiffPixelRatio 0.01 h1 colour +8 levels (#1a1a1a to #222) | pass | FAIL 858 px | pass | pass | pass h1 colour +25 levels (#1a1a1a to #333) | pass | FAIL 858 px | FAIL 723 px | pass | pass button blue #0b5fff to #0b5ffa | pass | FAIL 3963 px | pass | pass | pass button blue #0b5fff to #0a58e0 | pass | FAIL 3977 px | pass | pass | pass button blue to red | FAIL 3875 px | FAIL 3984 px | FAIL 3941 px | FAIL 3875 px | pass card text shifted 1 px | FAIL 1954 px | FAIL 19770 px | FAIL 2368 px | FAIL 1954 px | pass one digit: $1,240.00 to $1,248.00 | FAIL 16 px | FAIL 49 px | FAIL 33 px | pass | pass button radius 6 to 8 | pass | pass | pass | pass | pass button radius 6 to 18 | FAIL 170 px | FAIL 170 px | FAIL 170 px | FAIL 170 px | pass h1 bold to regular | FAIL 907 px | FAIL 1224 px | FAIL 1127 px | FAIL 907 px | pass gradient start #4a90d9 to #4a90e0 | pass | FAIL 17034 px | pass | pass | pass

On an earlier version of this fixture (larger price font, so the pixel counts differ a little) I ran the matrix three times and the pass/fail pattern was identical each time. I then ran this version twice, the second time from the code blocks of this article in a fresh directory, and every cell in both tables matched.

Five things stand out.

1. threshold is about colour per pixel, not about how many pixels changed. Playwright's docs describe it as an acceptable perceived colour difference in YIQ space between the same pixel in two images, from 0 (strict) to 1, default 0.2. The blue button changed on about 4 000 pixels, and the default setting passed it, because every one of them moved only a little. A one-digit price change touched 16 pixels, and the default setting failed it, because each of those pixels moved a lot.

2. The default setting passes real colour changes. #0b5fff to #0a58e0 passed. So did text going from #1a1a1a to #333333, and a gradient start moving from #4a90d9 to #4a90e0. Lowering threshold to 0.05 catches the #333 text but still passes the blue button.

3. maxDiffPixelRatio: 0.01 is a ratio of the whole image. On this page that is 4 000 pixels. It passed every change in the table: a red button (3 875 px), all text shifted by a pixel (1 954 px), bold turning regular, a changed digit. The ratio looks small, but the screenshot is big, so most of the page can change.

4. maxDiffPixels: 100 suits noise, not meaning. It passed the digit change (16 px) and the gradient. Pixel counts say little about whether a change matters: a changed digit in a price is tiny in pixels and large in meaning.

5. A change from border-radius: 6px to 8px never fails. Not even at threshold: 0. Playwright's bundled pixelmatch sets includeAA: false (I found it in playwright-core/lib/coreBundle.js), which skips pixels it thinks are anti-aliasing, and the pixels that differ on a small radius change look like anti-aliasing. pixelmatch's README documents includeAA. To check that this was the reason, I compared the two button images with pixelmatch directly:

import { chromium } from '@playwright/test'; import { PNG } from 'pngjs'; import pixelmatch from 'pixelmatch'; const shot = async (browser, radius) => { const page = await browser.newPage({ viewport: { width: 200, height: 80 } }); await page.setContent(`<style>body{margin:20px;font:16px DejaVu Sans} button{background:#0b5fff;color:#fff;border:0;padding:10px 18px;font:inherit;border-radius:${radius}px}</style> <button>Pay now</button>`); const png = PNG.sync.read(await page.screenshot()); await page.close(); return png; }; const browser = await chromium.launch(); const a = await shot(browser, 6); for (const r of [8, 18]) { const b = await shot(browser, r); for (const includeAA of [false, true]) { const n = pixelmatch(a.data, b.data, null, a.width, a.height, { threshold: 0, includeAA }); console.log(`radius 6 vs ${r}, includeAA ${includeAA}: ${n} px differ`); } } await browser.close();

Output:

radius 6 vs 8, includeAA false: 0 px differ radius 6 vs 8, includeAA true: 72 px differ radius 6 vs 18, includeAA false: 170 px differ radius 6 vs 18, includeAA true: 324 px differ

Playwright's toHaveScreenshot() has no includeAA option in the assertion docs I read, so this cannot be switched on there. I did not look for a way to swap the comparator.

A smaller screenshot helps, but only partly

The second half of the matrix takes the screenshot of the .card element, 470 x 324 = 152 280 pixels, so 1% is about 1 523 pixels:

card screenshot | default | threshold 0 | threshold 0.05 | maxDiffPixels 100 | maxDiffPixelRatio 0.01 button blue to red | FAIL 3875 px | FAIL 3984 px | FAIL 3941 px | FAIL 3875 px | FAIL 3875 px card text shifted 1 px | FAIL 1954 px | FAIL 19770 px | FAIL 2368 px | FAIL 1954 px | FAIL 1954 px one digit: $1,240.00 to $1,248.00 | FAIL 16 px | FAIL 49 px | FAIL 33 px | pass | pass button radius 6 to 18 | FAIL 170 px | FAIL 170 px | FAIL 170 px | FAIL 170 px | pass h1 bold to regular | FAIL 907 px | FAIL 1224 px | FAIL 1127 px | FAIL 907 px | pass

(Rows left out gave the same result as in the page table.) The ratio now catches the red button and the shifted text. It still passes bold turning regular, the changed digit and the larger radius. Cropping to the component makes the ratio mean something, but it does not make it strict.

What I would use

For a component screenshot taken in a controlled environment, I would use zero tolerance:

await expect(page.locator('.card')).toHaveScreenshot('card.png', { threshold: 0, maxDiffPixels: 0 });

That caught every change in the table except the radius one. It also did not fail on its own: the noise test in the spec above reloads the same page 20 times and compares at threshold: 0 with maxDiffPixels: 0.

page: 0/20 failed; 800x500 px card: 0/20 failed; 470x324 px

Limits of that result: it is one machine, one browser build and one font. I did not run it on another host or in a container, and I would not expect zero tolerance to survive a baseline made on a laptop and compared on a CI runner with different fonts. Make the baselines in the same environment that compares them, and mask or style away what you know changes (clocks, carets, animations) instead of loosening the whole comparison.

For values a pixel diff cannot see, assert them. The last test in the spec checks computed style:

button blue #0b5fff to #0a58e0: FAIL, got rgb(10, 88, 224) button radius 6 to 8: FAIL, got 8px

toHaveCSS() caught both the blue that default toHaveScreenshot() passed and the radius that no screenshot option can see. It only checks the property you name, so it complements the screenshot instead of replacing it.

Sources: the Playwright toHaveScreenshot options for threshold, maxDiffPixels and maxDiffPixelRatio; the pixelmatch README for includeAA; the numbers above are from my own runs.

Written with AI assistance. Every code sample was run on 2026-11-13 with Playwright 1.63.0 (Chromium headless shell) and pixelmatch 8.0.0; the output above is from that run.