Golden-image regression tests for an image pipeline in CI

A golden-image test runs fixed input images through your pipeline and checks two numbers per file: the output size ratio stays under a ceiling, and the quality (PSNR) stays over a floor. Both thresholds come from a measured run, set just past it. In my test, a quality slip from 75 to 90 wrote 83.6% more bytes; a ceiling 5% over the measured ratio failed it, and a ceiling of 0.5 let it pass.

Why not compare output bytes or hashes?

Because encoders change. A libwebp or mozjpeg upgrade shifts output by a few bytes without anyone touching your code, and a hash test fails on every upgrade. People then stop trusting it. A ratio ceiling and a PSNR floor survive small shifts and still fail a real regression. You test the contract (smaller than this, at least this good), not the bit pattern.

How does a production engine do it?

The image engine I work on keeps its golden suite in packages/engine/src/process.test.ts. Three ideas carry it:

1. The corpus is reproducible. A fixtures script draws the synthetic images with a seeded PRNG (mulberry32, seed 42), so anyone can regenerate the exact same files. One real 9 MP photograph sits beside them, because synthetic files hid a codec cost twice.
2. Every ceiling carries its measured value in a comment. For example photoJpegToAvif: 0.32, // measured 0.2885. A reviewer sees the headroom at a glance.
3. A report script prints the numbers to tune against. npm run report runs every fixture into every format and prints ratio and PSNR. Today it printed, among others:

photo.jpg → avif ratio 0.291 psnr 29.5 kind photo photo.png → same (png) ratio 0.137 psnr 27.5 kind photo graphic.png → avif ratio 0.108 psnr 53.3 kind graphic 

The suite ran green on this machine: 28 tests in 17.23 s.

One comment in that file is the reason for this article. An old AVIF ceiling of 0.5 sat over a stale "measured 0.418". The file notes it had 42% headroom, enough to pass a size regression "several times larger than any real change". The fix was to re-measure and set 0.32 over 0.2885.

A golden suite you can copy

Here is the same idea in 60 lines, with node:test and sharp. The pipeline under test fits an image inside 1200 px and writes WebP:

// pipeline.mjs import sharp from 'sharp'; export const QUALITY = Number(process.env.QUALITY ?? 75); export function encode(input) { return sharp(input).rotate() .resize({ width: 1200, height: 1200, fit: 'inside', withoutEnlargement: true }) .webp({ quality: QUALITY }).toBuffer(); } 

PSNR compares decoded pixels. Flatten alpha onto a fixed grey first, so the colour under fully transparent pixels does not count:

// psnr.mjs import sharp from 'sharp'; export async function psnr(reference, candidate) { const decode = (b) => sharp(b).flatten({ background: '#808080' }).removeAlpha().raw() .toBuffer({ resolveWithObject: true }); const [a, b] = await Promise.all([decode(reference), decode(candidate)]); if (a.info.width !== b.info.width || a.info.height !== b.info.height) throw new Error('size differs'); let sum = 0; for (let i = 0; i < a.data.length; i++) { const d = a.data[i] - b.data[i]; sum += d * d; } const mse = sum / a.data.length; return mse === 0 ? Infinity : 10 * Math.log10((255 * 255) / mse); } 

The reference is the same resize saved losslessly, so PSNR measures the encode and not the resize. I ran a report first:

photo.jpg bytes 112234 ratio 0.1946 psnr 34.68 screenshot.png bytes 32562 ratio 0.0290 psnr 40.50 transparent.png bytes 1442 ratio 0.1818 psnr 55.89 

Then the thresholds go into the test, about 5% over each ratio and 0.5 dB under each PSNR:

// golden.test.mjs import { test } from 'node:test'; import assert from 'node:assert/strict'; import { readFile } from 'node:fs/promises'; import sharp from 'sharp'; import { encode } from './pipeline.mjs'; import { psnr } from './psnr.mjs'; const GOLDEN = { 'photo.jpg': { ceiling: 0.205, floor: 34.2 }, // measured 0.1946, 34.68 dB 'screenshot.png': { ceiling: 0.031, floor: 40.0 }, // measured 0.0290, 40.50 dB 'transparent.png': { ceiling: 0.19, floor: 55.4 }, // measured 0.1818, 55.89 dB }; for (const [name, { ceiling, floor }] of Object.entries(GOLDEN)) { test(`${name} stays under its size ceiling and over its PSNR floor`, async () => { const input = await readFile(`fixtures/${name}`); const out = await encode(input); const { format, width, height } = await sharp(out).metadata(); assert.equal(format, 'webp'); const ref = await sharp(input).rotate().resize(width, height).png().toBuffer(); const ratio = out.length / input.length; const score = await psnr(ref, out); assert.ok(ratio < ceiling, `ratio ${ratio.toFixed(4)} >= ceiling ${ceiling}`); assert.ok(score > floor, `PSNR ${score.toFixed(2)} dB <= floor ${floor}`); }); } 

The format check matters too. A pipeline that silently falls back to PNG passes a PSNR test with a perfect score.

Does it catch a real regression?

I broke the pipeline two ways through the QUALITY variable. At quality 90 the files get bigger:

✖ photo.jpg stays under its size ceiling and over its PSNR floor (178.751916ms) ✖ screenshot.png stays under its size ceiling and over its PSNR floor (140.856875ms) ✔ transparent.png stays under its size ceiling and over its PSNR floor (22.85425ms) AssertionError [ERR_ASSERTION]: ratio 0.3572 >= ceiling 0.205 AssertionError [ERR_ASSERTION]: ratio 0.0436 >= ceiling 0.031 

At quality 50 they get worse, and the floor catches it:

 AssertionError [ERR_ASSERTION]: PSNR 32.82 dB <= floor 34.2 AssertionError [ERR_ASSERTION]: PSNR 38.81 dB <= floor 40 AssertionError [ERR_ASSERTION]: ratio 0.1975 >= ceiling 0.19 

The last line surprised me. The 512 × 512 transparent logo grew from 1,442 to 1,566 bytes at the lower quality. Flat art does not follow the photo rule, which is why it needs its own fixture.

How loose can a ceiling be?

Not loose at all. I set the photo ceiling to 0.5 and ran quality 90 again:

✔ photo.jpg stays under its size ceiling and over its PSNR floor (180.679ms) 

The photo output grew from 112,234 to 206,038 bytes and the test stayed green.

Note: a threshold is only as good as its headroom. Write the measured value next to every ceiling and floor, and re-measure when you change an encoder on purpose. A comment that says "measured 0.418" next to a ceiling of 0.5 tells the next reviewer the test is asleep.

What should the fixture set hold?

One file per kind of content your users send, because each kind breaks in its own way: a real photograph, a screenshot with text, flat art with transparency. Commit them. Keep them small enough for CI; my three run in under half a second. How we measure shows how the site ties every published number to one real photo run through the engine.

How I measured

• Machine: Apple M1 Pro (8 cores), macOS, 2026-10-17.
• Node v24.22.0, sharp 0.35.4, libvips 8.18.6, libwebp 1.6.0, vitest 4.1.11 for the engine suite.
• Fixtures: photo.jpg (1920 × 1280), screenshot.png (2880 × 1800) and transparent.png (512 × 512) from the site's reference set.
• Every output block above is pasted from node --test, node report.mjs or npm run report -w @blinkimg/engine.

What I did not test

Linux CI runners: the same sharp version on another platform may write slightly different bytes, which is what the 5% headroom is for. I did not try SSIM or Butteraugli as the quality metric.

I build Blinkimg, an image compressor and converter with a developer API. I wrote this article with help from an AI assistant (Claude), which ran the code on my machine; every number in it comes from those runs.