So I had this problem. We deploy to production pretty frequently -- sometimes twice a day -- and every once in a while something visual breaks. A button moves, a font changes, some CSS gets overridden by a new component. The kind of stuff that doesnt show up in unit tests.

For a while I was just... manually checking pages after deploys. Open the site, click around, squint at things. You know how that goes. Sometimes I'd catch issues, most of the time I convinced myself everything looked fine.

Then one day a client sent a screenshot of our pricing page where the CTA button had disappeared. It had been gone for 4 days. Four days. Nobody on the team noticed.

The pixel diff approach

I wrote a quick script that takes screenshots of key pages and compares them pixel-by-pixel against a baseline. Nothing fancy -- headless Chrome, save PNGs, compare with pixelmatch. If the diff exceeds a threshold, flag it.

const pixelmatch = require("pixelmatch"); const { PNG } = require("pngjs"); const fs = require("fs"); const img1 = PNG.sync.read(fs.readFileSync("baseline.png")); const img2 = PNG.sync.read(fs.readFileSync("current.png")); const diff = new PNG({ width: img1.width, height: img1.height }); const numDiffPixels = pixelmatch( img1.data, img2.data, diff.data, img1.width, img1.height, { threshold: 0.1 } ); console.log("Changed pixels: " + numDiffPixels); 

Simple enough. But the first run was a disaster -- every page flagged because of dynamic content. Timestamps, user avatars, ad slots, cookie banners. The noise was unbearable.

What actually worked

Had to get smarter about it. Added masking for dynamic regions, ran comparisons only on specific viewport sizes, and set different thresholds per page. The checkout page gets a tight threshold (even small shifts matter there), blog pages get a looser one.

The biggest win was scheduling it to run after every deploy and dumping results into Slack. No more manual checking. If something breaks visually, I know within minutes.

Couple things I learned the hard way:

  • Fonts load differently between runs sometimes. Had to add a wait-for-fonts step or I'd get false positives from text rendering slightly differently.
  • Responsive breakpoints matter. A bug at 768px width might not show at 1440px. I capture both now.
  • Dark mode doubles your work. If you support it, you gotta screenshot both themes. I... didn't at first. Guess where the bugs were hiding.

The annoying parts

Maintaining baselines is tedious. Every intentional design change means updating screenshots. I tried automating that too -- if the PR is merged and approved, auto-update baselines -- but it still requires someone to eyeball the diff first.

Also storage. Full-page screenshots at 2x resolution add up fast. Had to set up rotation to keep only the last 30 days.

Its not perfect but its caught probably a dozen visual regressions that would have otherwise shipped to users. The setup took a weekend. Honestly should have done it way sooner.

Anyone else doing visual regression testing? Curious what tools or approaches work for you.