Freeze the Output Before You Touch the Function: A Behavior Census for Untested Code

Most messy-repo refactors break at the first edit. You change one helper. Three callers change behavior quietly. Nobody notices until Friday.

So invert the order: record behavior first, edit second. Write golden files for the ugly function. Then make one small change and diff the output.

This is characterization testing, not specification testing. You are not asserting what the code should do. You assert what it does today, bugs included.

1. Why a census beats reading the function

Reading code tells you intent. It rarely tells you behavior at the edges. Empty strings, duplicate keys, and None hide there.

A behavior census is a folder of JSON files. Each file holds one call and one result. After your edit, the diff is the argument.

Cost is small: one script, one cases file, one afternoon. Payoff is permanent: every later edit becomes a two-command check.

2. Build the census driver

Save this as census.py. It imports your module, calls each case, and normalizes the result.

#!/usr/bin/env python3 """Record or check behavior of existing functions as golden JSON files.""" import importlib import json import pathlib import sys def norm(value): """Force a stable JSON shape; unknown types fall back to repr().""" return json.loads(json.dumps(value, default=repr, sort_keys=True)) def run_case(spec): module = importlib.import_module(spec["module"]) fn = getattr(module, spec["function"]) record = {"id": spec["id"], "target": f'{spec["module"]}.{spec["function"]}'} try: result = fn(*spec.get("args", []), **spec.get("kwargs", {})) record.update(status="ok", result=norm(result)) except Exception as exc: # a raise is behavior too  record.update(status="raise", error=type(exc).__name__, message=str(exc)) return record def main(argv): mode = argv[1] if len(argv) > 1 else "record" outdir = pathlib.Path(argv[2] if len(argv) > 2 else "golden") outdir.mkdir(parents=True, exist_ok=True) specs = json.loads(pathlib.Path("cases.json").read_text()) drift = 0 for spec in specs: record = run_case(spec) path = outdir / f'{spec["id"]}.json' text = json.dumps(record, indent=2, sort_keys=True) + "\n" if mode == "check": old = path.read_text() if path.exists() else "" if old != text: drift += 1 print(f"DRIFT {spec['id']}") else: path.write_text(text) if mode == "check": print(f"{drift} case(s) changed out of {len(specs)}") return 1 if drift else 0 return 0 if __name__ == "__main__": sys.exit(main(sys.argv)) 

Now pick a messy target. Here is a small one with a buried rule.

# pricing.py def line_total(qty, unit_cents, discount=None): if discount is None: return qty * unit_cents if qty > 10 and discount > 20: return qty * unit_cents return int(qty * unit_cents * (100 - discount) / 100) 

Discounts above twenty vanish past ten units. That looks like a bug. The census freezes it anyway, because you are refactoring, not fixing.

[ {"id": "plain", "module": "pricing", "function": "line_total", "args": [3, 500]}, {"id": "discount_small", "module": "pricing", "function": "line_total", "args": [3, 500, 10]}, {"id": "discount_bulk", "module": "pricing", "function": "line_total", "args": [12, 500, 25]}, {"id": "zero_qty", "module": "pricing", "function": "line_total", "args": [0, 500, 10]}, {"id": "bad_type", "module": "pricing", "function": "line_total", "args": ["3", 500]} ] 

Run two commands. The second one must report zero drift.

python census.py record golden python census.py check golden 

If check fails on an untouched tree, your code is nondeterministic. Fix that before editing anything.

3. Triage nondeterminism before trusting the folder

A flaky golden file destroys trust in every other file. Triage it now.

If a case cannot be frozen, delete it and write it down as a known gap. A quarantined case is honest. A pretend-stable case is not.

4. Put the census behind pytest

Wrap check mode in one test. Then every edit re-runs it for free.

# test_census.py import json import pathlib import census def test_behavior_census(): specs = json.loads(pathlib.Path("cases.json").read_text()) for spec in specs: golden = json.loads((pathlib.Path("golden") / f'{spec["id"]}.json').read_text()) assert census.run_case(spec) == golden, spec["id"] 

pytest -q test_census.py now answers one question. Did behavior move?

5. Make the smallest safe change

Order matters here. Follow these steps literally.

1. Record golden files from the untouched code.
2. Confirm check reports zero drift twice in a row.
3. Edit exactly one thing: one branch, one extraction, one rename.
4. Run check again and read the drift list.
5. Accept only drift you can explain in one sentence.
6. Commit code and golden files together when drift is intended.
7. Delete cases that only protected code you removed.

Step five is the method. If you cannot explain a changed golden file, you changed behavior by accident. Revert and take a smaller step.

Useful rule: for pure refactors the golden diff must be empty. Any drift means you refactored and changed behavior at once. Split that into two commits, always.

6. Where a free model and a free server help

Two parts of this loop are tedious. Inventing cases is one. Reading a long drift list is the other.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option, per the operator. I point the free model at one job: propose cases.json entries from a signature and its call sites. Then I delete the silly ones myself.

The free server helps when the repo demands a specific runtime. Clone it there, install dependencies, run census.py record. The golden files come back portable, with no local environment surgery.

What a model must never do is invent expected values. Every golden file comes from a real execution. A guessed output is a specification nobody agreed to.

Keep the model on the boring edges:

• propose edge inputs from type hints and call sites
• sort a long drift list into intended and suspicious
• draft norm helpers for odd return types

Verify each suggestion by running it. Suggestions are cheap. Executions are evidence.

7. Limitations and who should skip this

Characterization tests lock in bugs. That is the point, and also the risk. If current behavior overcharges a customer, freezing it makes the wrong thing permanent.

Skip this workflow when:

• the module's behavior must change, not be preserved
• correctness outranks parity (billing, auth, medical)
• the code touches production systems at import or call time
• the function is thin glue over an already well-tested library

One safety note deserves its own paragraph. A census executes your code with your inputs. Never point it at a live database or a real payment client. Use fixtures, sandboxes, and a throwaway machine.

Closing

Freeze behavior before you improve structure. The golden folder is small and boring. It is also why your refactor can stay a single reviewable commit.

Pick the function that scares you most this week. Record its golden files first, and change nothing else.