Retire the Golden Master: A Deletion Ledger for Refactors That Already Landed

A pinned test that outlives its refactor is dead weight. It fails on formatting, not behavior. It slows every future diff. And nobody remembers why it exists.

Most characterization-test advice stops at the merge. This post covers the part after: retiring the scaffold deliberately, with evidence that each deletion is safe.

The artifact is a two-part loop. First, a ledger that tags every pinned test with a seam, an owner, and a removal trigger. Second, a delete-one experiment that removes a single pin and proves the remaining suite still catches a deliberate break.

Why the scaffold survives too long

Characterization tests are cheap to write and expensive to delete. Deletion needs a decision. Decisions need an owner.

A typical repo after a refactor has three kinds of pins. Behavior pins encode rules the business still wants. Formatting pins encode a string layout nobody reads. Wiring pins encode the call order of a module that no longer exists.

Only the first kind deserves to live. The other two rot quietly, and they return false failures during unrelated work.

Example inventory shape (illustrative, not measured from your repo):

Step 1: Tag pins at write time

Add one machine-readable comment above each pinned test. Three fields: seam, owner, removal trigger.

# tests/test_rates.py  # PIN: seam=billing.rates owner=@dakota remove=after-tax-split-merged def test_legacy_rounding_matches_golden(): assert legacy_round(1.005) == "1.00" 

The seam is the code region the pin protects. The owner is a human who can say yes. The removal trigger is a condition, not a date.

Triggers age better than dates. after-tax-split-merged stays meaningful in eight months. 2026-12-01 does not.

Step 2: Build the ledger from the tests

Parse the markers into a CSV. No test framework plugin, no new dependency.

# ledger.py import csv import pathlib import re import sys MARKER = re.compile( r"#\s*PIN:\s*seam=(?P<seam>[\w./-]+)\s+" r"owner=(?P<owner>[\w@.-]+)\s+" r"remove=(?P<remove>[\w-]+)" ) TEST_DEF = re.compile(r"^def (?P<name>test_\w+)", re.MULTILINE) def rows(root: pathlib.Path): for path in sorted(root.rglob("test_*.py")): text = path.read_text(encoding="utf-8") markers = list(MARKER.finditer(text)) defs = list(TEST_DEF.finditer(text)) for marker in markers: after = [d for d in defs if d.start() > marker.start()] if not after: print(f"orphan marker in {path}", file=sys.stderr) continue name = after[0].group("name") yield { "test_id": f"{path.as_posix()}::{name}", "seam": marker.group("seam"), "owner": marker.group("owner"), "remove": marker.group("remove"), } def main(root: str) -> None: out = pathlib.Path("ledger.csv") found = list(rows(pathlib.Path(root))) with out.open("w", newline="", encoding="utf-8") as handle: writer = csv.DictWriter( handle, fieldnames=["test_id", "seam", "owner", "remove"] ) writer.writeheader() writer.writerows(found) print(f"wrote {len(found)} pins to {out}") if __name__ == "__main__": main(sys.argv[1] if len(sys.argv) > 1 else "tests") 

Run it and read the CSV grouped by seam.

python ledger.py tests cut -d, -f2 ledger.csv | tail -n +2 | sort | uniq -c | sort -rn 

Seams with many pins are the most valuable to inspect first. They usually mark the places where the refactor was least understood.

Step 3: Run the delete-one experiment

Deletion is a hypothesis. Test it like one.

The rule: remove exactly one pinned test from the suite. Run the suite with a deliberate source-level break applied. The remaining tests must still fail. If nothing fails, that pin was the only detector, or the break was too small.

Write two or three break probes by hand. Keep them trivial and reversible.

# probes.py — each probe edits one file, returns the restore callback def probe_rounding(path): original = path.read_text(encoding="utf-8") path.write_text(original.replace('>= 5', '> 5'), encoding="utf-8") return lambda: path.write_text(original, encoding="utf-8") 

Drive the whole sweep from the ledger.

#!/usr/bin/env bash set -euo pipefail while IFS=, read -r test_id seam owner remove; do [ "$test_id" = "test_id" ] && continue if pytest -q --deselect "$test_id" >/dev/null 2>&1; then echo "GREEN_WITHOUT_PIN,$test_id,$seam" >> sweep.csv else echo "SURVIVED,$test_id,$seam" >> sweep.csv fi done < ledger.csv 

Run it against a deliberately broken branch, not against clean HEAD. On clean code every row prints GREEN_WITHOUT_PIN, which tells you nothing.

Step 4: Classify with a decision table

Sort results into three buckets. Keep the table boring and mechanical.

Retirement needs a paper trail. Append the seam, owner, and date to a RETIRED.md. Future readers then know the pin was removed on purpose.

Where a free model and a free server fit

The tedious part is not the sweep. It is reading the assertion diffs. Dozens of pins produce near-identical strings, and deciding behavior versus formatting is judgment work.

MonkeyCode offers free model access and a free server option (operator-supplied availability). That maps onto the two chores here.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

First, use the free model access to draft classification proposals. Feed it one pin, its assertion, and the delete-one result. Ask for a label from the decision table plus a one-line reason. Treat the output as a proposal, never as a verdict. A human owner still signs the retirement.

Second, run the sweep on the free server option. The loop is CPU-bound, repetitive, and slow. Keeping it off a laptop is the whole benefit. Do not assume any quota, runtime, or hardware figure — check the current terms yourself before you plan a batch.

# sweep on a remote box, results pulled back bash sweep.sh && scp sweep.csv local:/tmp/sweep.csv 

Keep prompts chunked to one pin per request. Batch summaries hide the formatting cases, which are exactly the ones you need to see.

Limitations

• The ledger is manual metadata. Untagged pins are invisible, so adoption must be enforced in review.
• Delete-one proves coverage against your probes only. A weak probe set proves nothing.
• The sweep is serial and slow on large suites. Parallelize only after the results are stable.
• The classification step is a proposal pipeline. Model output can be confidently wrong.

Who should skip this

Skip it if your suite is not green. Fix the baseline first.

Skip it if your suite takes longer than twenty minutes. The sweep multiplies that cost per pin.

Skip it for regulated code where every assertion is audit evidence. Retiring pins there is a compliance decision, not an engineering one.

What to do Monday

Tag ten pins. Run ledger.py. Run one delete-one sweep on a broken branch. Then retire one formatting pin and write the RETIRED.md line.

If you try the ledger, tell me which removal trigger you picked first — that field is where most teams learn something about their own repo.