Visual regression testing with a screenshot API and pixel diffs in Python

Functional tests tell you the checkout button still works. They do not tell you it turned white-on-white after a CSS refactor, or that a footer slid under a banner on the pricing page. Visual regression testing catches that class of bug: take a screenshot of a page, compare it with an approved one, and fail when too much has changed.

You do not need a heavy framework for this. Many teams already have a staging or production URL they want to keep an eye on, and the two hard parts, rendering the page and comparing two images, are small. This post builds a complete check in about 70 lines of Python: capture with a screenshot API, diff with Pillow, ignore areas that change on purpose, and exit non-zero on a regression so any scheduler or deploy script can act on it.

We ran the code on 10 November 2026 against a real page, and we found a flaw in our first threshold. That is covered below because it is the most useful thing in the post.

A note on what was tested: the captures came from the public Pageframe Screenshots Actor, the engine behind our screenshot API. The /v1/screenshot call in the script uses the same options, but we did not re-call that endpoint with a key for this post. The difference is only how the image is downloaded (the API's fileUrl is a signed link, the Actor's store needs your token).

The approach

Visual regression has four steps:

1. Capture the page at a fixed viewport so runs are comparable.
2. Compare the new capture with a stored baseline, pixel by pixel.
3. Decide whether the difference is noise or a regression.
4. Show where it changed, so a human can approve or fix it quickly.

Steps 1 and 4 matter more than people expect. A capture that is not repeatable (different viewport, a cookie banner that appears on some runs) produces false failures, and a bare "FAIL" with no picture trains people to ignore the check.

Capture settings that keep runs repeatable

Use the same options on every run. The ones that matter:

curl https://siftwright.com/v1/screenshot \ -H "Authorization: Bearer $SIFTWRIGHT_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "urls": ["https://example.com/pricing/"], "format": "png", "fullPage": true, "viewportWidth": 1280, "viewportHeight": 800, "blockCookieBanners": true }' 

• PNG, not JPEG. JPEG compression adds noise that a pixel diff reads as change. Use lossless output for comparisons.
• A fixed viewport. Pick one width and keep it. If you care about mobile, run a second named check at 390 wide.
• Full page. Layout bugs often live below the fold, in the footer or a long table.
• Cookie banner handling. Dismissal is best effort. If a banner still appears, it shows up as a diff, which is the correct behaviour: you want to know.

A failed capture returns status: "error" and does not count against your plan, so a flaky target does not cost you money. The script treats that as an error, not a pass.

The script

Install Pillow (pip install pillow) and set SIFTWRIGHT_API_KEY:

import io, json, os, re, sys, urllib.request from pathlib import Path from PIL import Image, ImageChops API = "https://siftwright.com/v1/screenshot" BASE = Path("baselines") OUT = Path("vrt-out") TOLERANCE = 16 # per-channel difference (0-255) ignored as antialiasing noise MAX_CHANGED = 0.002 # fail if more than 0.2% of pixels changed TILE = 80 # region report granularity in px def capture(url, **opts): body = {"urls": [url], "format": "png", "fullPage": True, "viewportWidth": 1280, "viewportHeight": 800, **opts} req = urllib.request.Request(API, json.dumps(body).encode(), {"Authorization": f"Bearer {os.environ['SIFTWRIGHT_API_KEY']}", "Content-Type": "application/json"}) with urllib.request.urlopen(req, timeout=300) as r: res = json.load(r)["results"][0] if res["status"] != "ok": raise RuntimeError(res.get("error", res["status"])) with urllib.request.urlopen(res["fileUrl"], timeout=60) as r: # signed link, no key needed return Image.open(io.BytesIO(r.read())).convert("RGB") def slug(url): return re.sub(r"[^a-z0-9]+", "-", url.lower().split("://", 1)[-1]).strip("-") def compare(base, cur, ignore=()): """Return (changed_ratio, bbox, hot_tiles, diff_image, size_note).""" note = None if base.size != cur.size: note = f"size changed {base.size} -> {cur.size}" w, h = min(base.width, cur.width), min(base.height, cur.height) base, cur = base.crop((0, 0, w, h)), cur.crop((0, 0, w, h)) d = ImageChops.difference(base, cur) r, g, b = d.split() mask = ImageChops.lighter(ImageChops.lighter(r, g), b).point( lambda v: 255 if v > TOLERANCE else 0) for x0, y0, x1, y1 in ignore: # blank out known-dynamic areas mask.paste(0, (x0, y0, x1, y1)) changed = mask.histogram()[255] ratio = changed / (mask.width * mask.height) tiles = [] if changed: for ty in range(0, mask.height, TILE): for tx in range(0, mask.width, TILE): t = mask.crop((tx, ty, tx + TILE, ty + TILE)).histogram()[255] if t > TILE * TILE * 0.02: # tile is >2% changed tiles.append((tx, ty)) overlay = Image.new("RGB", cur.size, (255, 0, 0)) dim = Image.blend(cur, Image.new("RGB", cur.size, (255, 255, 255)), 0.6) return ratio, mask.getbbox(), tiles, Image.composite(overlay, dim, mask), note def check(url, update=False, ignore=(), **opts): BASE.mkdir(exist_ok=True); OUT.mkdir(exist_ok=True) name, path = slug(url), BASE / f"{slug(url)}.png" cur = capture(url, **opts) if update or not path.exists(): cur.save(path) return {"url": url, "status": "baseline-saved"} ratio, bbox, tiles, diff, note = compare(Image.open(path).convert("RGB"), cur, ignore) ok = ratio <= MAX_CHANGED and not tiles and note is None if not ok: cur.save(OUT / f"{name}-current.png"); diff.save(OUT / f"{name}-diff.png") return {"url": url, "status": "pass" if ok else "FAIL", "changed": round(ratio, 5), "bbox": bbox, "hot_tiles": len(tiles), "note": note} if __name__ == "__main__": update = "--update" in sys.argv urls = [a for a in sys.argv[1:] if not a.startswith("--")] results = [check(u, update=update) for u in urls] for r in results: print(json.dumps(r)) sys.exit(1 if any(r["status"] == "FAIL" for r in results) else 0) 

Run it once to save baselines, then again whenever you want to check:

python vrt.py https://example.com/pricing/ # first run: saves the baseline python vrt.py https://example.com/pricing/ # later runs: compare python vrt.py --update https://example.com/pricing/ # accept the new look as the baseline 

On a failure you get vrt-out/<name>-diff.png: the current page washed out, with changed pixels in solid red. Open it and you see at once whether the change is a moved button or a rewritten paragraph.

Design decisions, and why

A per-channel tolerance of 16. Antialiasing and font hinting produce tiny colour differences that are not bugs. Ignoring differences of 16 or less on each channel removes that noise and still catches any real colour change, which is far larger.

A tile check on top of the ratio. This is the mistake we made. Our first version failed only when more than 0.2% of all pixels changed. We tested it on the full-page capture of siftwright.com/pricing/, which is 1280 by 5785 pixels, then painted a button-sized red rectangle (200 by 48) onto a copy to simulate a regression. The result:

On a long page, a clearly visible broken button is a small share of the pixels. A global percentage dilutes real bugs the longer the page gets. The fix is the tile report: split the diff into 80-pixel tiles and fail if any tile is more than 2% changed. With that rule the same test produced six hot tiles and a FAIL, with a bounding box of (100, 300, 301, 349) pointing at the rectangle. Keep the global ratio too, because it catches a sitewide change (a new font, say) that no single tile flags.

Size changes fail on their own. If the page height changes, everything below the change shifts and the comparison turns into red noise. The script reports size changed (1280, 5785) -> (1280, 5825) and fails without pretending to diff the shifted content. That is usually what you want: someone added or removed content, so look.

Ignore regions for things that change on purpose. Dates, ad slots, "last updated" labels, live counters. Pass rectangles as (x0, y0, x1, y1) and the script blanks them out of the mask. Coordinates are exclusive on the right and bottom, so add one pixel if you measured from a drawing tool. In our test, ignoring the exact rectangle brought the changed ratio back to 0.0 and the check passed.

What we measured

We ran the code against https://siftwright.com/pricing/, captured by the Actor with the settings above:

Two captures of the same page seconds apart were pixel-identical, and the same held for two captures of the Hacker News front page. That is a favourable case: the pages are static. A page with a carousel, a video or a rotating testimonial will not be, and you should expect to add ignore regions or wait for a selector before capturing. Behaviour on your own pages depends on what they render.

The synthetic regression was painted onto a real capture rather than produced by changing the live page, because we cannot edit a page we do not control. The diff logic is what we tested there; the capture was real and repeatable.

Practical advice

• Store baselines in version control or object storage, keyed by URL and viewport. Review the diff image when you update them, like any other code review.
• Compare like with like. Staging versus a production baseline will differ in data. Capture a baseline from the same environment you test.
• Wait for the page to settle. If content loads late, pass a waitForSelector option (the Actor supports it) so you do not capture a loading state.
• Keep the page list short. Ten to twenty pages that matter (home, pricing, checkout, a long article, a form) give most of the benefit. Every extra page is a capture to pay for and a diff to review.
• Do not run it on every commit by default. Scheduled runs against staging, or one run per release, are usually enough and cost far less.

Limits

Pixel diffing is blunt. It cannot tell a deliberate redesign from a bug, so someone has to approve baselines. It is sensitive to fonts and rendering engine differences, which is why you capture with the same service every time instead of comparing a local browser capture with a cloud one. And it does not test behaviour: a page that looks right can still be broken.

For pure design-system work on components in isolation, a dedicated tool such as Storybook-based visual testing may suit you better. For watching real pages at real URLs, with nothing to install beyond Pillow, the approach above is enough to catch the regressions that embarrass you.

Try it

The screenshot API takes a URL and returns a PNG, JPEG or PDF, with full-page capture, a fixed viewport and cookie-banner handling. Plans start at $29 per month for 10,000 successful results, and failed captures are not counted. You can try a capture for free with the website screenshot tool, then read the screenshot API page for options and pricing. If you want to see how to batch captures and cache them, our post on website thumbnails and link previews covers it, and the screenshot API tutorial has the basics in Python and Node.