Characterization Tests First: Freeze a Messy Function, Then Make One Safe Edit

Freeze the outputs first. Make one edit second. Prove the diff is zero third.

That order matters in a messy repo. Most refactors fail at step one, not step three.

This post gives a small, reproducible harness for that order. It has four parts: harvest real inputs, pin golden outputs, kill three planted mutants, then allow exactly one edit.

Why "it has tests" is not a promise

A test that executes a line does not constrain that line. Coverage tells you where you may edit. It does not tell you whether a mistake would be noticed.

Messy code has a second problem: nobody knows the intended behavior. So you cannot write a spec. You can only record what the code does today and demand it stays that way.

That recording is a characterization test. Its value depends on one number: how many wrong edits it can catch. We will measure that number instead of assuming it.

Step 1: Harvest inputs from reality, not imagination

Do not invent inputs from reading the function. Pull them from fixtures, logs, and seed data that already exist.

Then add boundary cases. A seeded generator keeps them reproducible.

// tests/cases/prices.json [ {"id": "plain_two_items", "items": [{"sku": "A1", "qty": 2, "cents": 499}], "coupon": null}, {"id": "qty_zero", "items": [{"sku": "A1", "qty": 0, "cents": 499}], "coupon": null}, {"id": "coupon_over_total", "items": [{"sku": "B2", "qty": 1, "cents": 100}], "coupon": {"off_cents": 500}}, {"id": "negative_cents", "items": [{"sku": "C3", "qty": 1, "cents": -1}], "coupon": null} ] 

Four cases is a start, not a suite. Aim for one case per branch you can name, plus one per branch you cannot.

Step 2: Pin the outputs in files you commit

Golden files beat inline assertions here. Reviewing a diff of recorded behavior is faster than reading test source.

The sketch below is runnable, but I have not executed it against your repo. Treat it as a template.

# tests/test_char_prices.py import json, os, pathlib, pytest from billing.prices import total_cents CASES = json.loads(pathlib.Path("tests/cases/prices.json").read_text()) GOLDEN = pathlib.Path("tests/golden/prices") @pytest.mark.parametrize("case", CASES, ids=[c["id"] for c in CASES]) def test_characterized_total(case): got = total_cents(case["items"], case["coupon"]) path = GOLDEN / f"{case['id']}.json" if os.environ.get("CHAR_REGEN") == "1": path.write_text(json.dumps({"total": got}, sort_keys=True)) assert path.exists(), f"no golden for {case['id']}; regenerate and review" assert got == json.loads(path.read_text())["total"] 

Regenerate once, read every golden file by hand, then commit them. From that commit on, CHAR_REGEN is forbidden in CI.

Step 3: Kill three mutants or admit the oracle is thin

A green suite proves nothing until you know it can go red. Plant bugs on purpose and check that the suite fails.

# mutants/list.txt: name<TAB>patch file flip_coupon_cmp mutants/flip_coupon_cmp.patch round_up_cents mutants/round_up_cents.patch drop_qty_guard mutants/drop_qty_guard.patch 

while IFS=$'\t' read -r name patch; do git apply "$patch" if pytest -q tests/test_char_prices.py >/dev/null 2>&1; then echo "SURVIVED: $name -> oracle too thin"; git checkout -- .; exit 1 fi git checkout -- . done < mutants/list.txt echo "all mutants killed" 

Three is not a theory of testing. Three is a cheap smoke alarm. If any mutant survives, add a case before you touch the code.

Round out the picture with branch coverage on the single target module:

pytest -q tests/test_char_prices.py --cov=billing.prices --cov-branch --cov-report=term-missing 

Step 4: Allow exactly one edit

The gate for the edit is not "tests pass". It is "tests pass and no golden file changed".

git switch -c refactor/prices-total pytest -q tests/test_char_prices.py # 1. green before # make one change: rename, extract, or flatten one branch git diff --stat # 2. one file, one hunk pytest -q tests/test_char_prices.py # 3. green after git diff --exit-code -- tests/golden # 4. must be empty 

Step four is the whole method. If behavior is preserved, the recorded bytes cannot move. If they move, you changed semantics, even when the suite is green.

Commit that single edit. Then repeat from step four with the next change. Never batch two edits into one golden gate.

Step 5: Write the receipt into the pull request

One short table per PR keeps reviewers honest.

A reviewer can audit that table in ten seconds. That is the point.

When this workflow does not fit

Characterization gates are conservative by construction. They preserve quirks you may want to remove.

| Situation | Use this workflow? | Reason |
| --- | --- |
| Untested billing, pricing, parsing | Yes | Deterministic, high blast radius |
| Time, timezone, or random dependent code | Only after freezing the clock and seed | Otherwise goldens flake |
| Network or database bound modules | No | Record a fixture layer first |
| Latency or memory refactors | No | Goldens compare values, not cost |
| Code you intend to delete next sprint | No | The gate costs more than the rewrite |

Rule of thumb: if you cannot freeze the environment in one test fixture, stop and fix that first.

Where a model fits, and where it does not

Two steps in this loop are tedious: proposing edge cases, and writing three plausible mutants. Both are narrow, well-specified tasks with a machine-checkable answer.

That is a reasonable place to use generated code. The suite decides. A bad suggestion costs one red test, not one bad merge.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is a project the team describes as open source, and the entry offer it advertises at the time of writing includes free model access plus a free server option. The same announcement lists a free token allowance; check the project page for the current figure, since quotas change and this post cannot promise them. Pasted guidance about "10 million tokens" should be treated as the operator's wording, not a measurement.

A short-lived job like a three-mutant loop is a fair first workload for a free tier. Verify the current limits before pointing it at a real repo.

Keep the division of labor strict. The model drafts cases and patches. pytest and git diff --exit-code decide.

Who should skip this

If your repo already has a trusted oracle, this is overhead. If your target module is three lines long, just rewrite it. If you need a behavior change rather than a behavior-preserving edit, goldens will fight you.

And if nobody will review the golden files once, do not commit them. Unreviewed goldens are a rubber stamp.

Start with the alarm, not the edit

Most refactor advice starts with the change. Start with the thing that can fail.

Harvest inputs. Pin outputs. Kill three mutants. Then allow one edit and demand a zero-byte golden diff.

If you try MonkeyCode on this, run the mutant loop before anything else. A survived mutant is a more useful first report than a feature request.