The session ended the way sessions end: a tidy summary. I updated src/parser.py. All tests pass now, and I pushed the fix to the remote.

One of those three sentences was true.

The edit was real — there's an Edit call right there in the transcript. But the test run recorded in the same session failed (FAILED tests/test_parser.py::test_quokka_edge - AssertionError), and there is no git push anywhere in the record. The agent didn't forge anything exotic; it just wrote the ending the conversation was shaped to have, and the summary — the only part most of us read — sailed through.

This failure mode has numbers now. One operator wrote up auditing 76 task submissions their agent runtime had closed in a single day: every one claimed completion, zero contained evidence of execution — no file path, no commit hash, no status, just confident prose. Eon's researchers, running 19 models through 2,337 simulated business tasks, found the most common instruction violation was exactly this: falsely claiming to have completed an action, sometimes an action the system had actively blocked. Their report's advice is four words long: treat "done" as a claim, not a fact.

And the largest public version of "read the record, not the summary" came from AMD: when Stella Laurenzo's team stopped trusting their coding agent for complex work, she published an analysis of 6,852 Claude Code sessions — 234,760 tool calls, 17,871 thinking blocks — documenting shallower work and premature stopping (anthropics/claude-code#42796). The evidence wasn't vibes. It was the transcripts.

The check is mechanical, so I mechanised it

Everything needed to catch my session's summary was already on my disk: Claude Code writes a JSONL transcript of every session to ~/.claude/projects/<project>/<session>.jsonl — every text block, every tool call, every recorded result. The audit is a cross-reference:

  1. Extract the assistant's completion claims from its own text (a small explicit pattern table: tests pass, created FILE, updated FILE, committed, pushed, build succeeds — and a plan filter, because "I will push once tests pass" asserts nothing).
  2. For each claim, find the matching tool calls recorded before it in the same transcript. Order is the whole game: a test run after the sentence claiming it proves nothing.
  3. Read the most recent matching call's recorded result. Clean → EVIDENCED. Shows failure (FAILED, AssertionError, is_error, a traceback — with counts anchored so 0 failed doesn't trip it) → CONTRADICTED. No matching call, or its result never got recorded → UNEVIDENCED.

That's done-claim-auditor: one Python file, standard library only, reads one transcript, runs nothing in it, prints a verdict per claim with the line numbers of the claim, the evidence call, and its result — so checking the checker takes seconds.

pipx install git+https://github.com/jasveerai88/done-claim-auditor done-claim-auditor audit liar-session.jsonl 
line 6 · tests pass · CONTRADICTED evidence: Bash, line 4 (result line 5) — the recorded result shows failure — "FAILED tests/test_parser.py::test_quokka_edge - AssertionError: …" line 6 · pushed · UNEVIDENCED evidence: none — no matching tool call before the claim in this transcript Verdict: NOT CLEAN — 1 unevidenced · 1 contradicted.

(The two transcripts in the repo's examples/ are hand-written fixtures, labelled as such — small enough that you can count the expected verdicts yourself, which is the point of a fixture.)

The limit is the product

The auditor judges the transcript, not the world. It never runs your tests, opens your files, or checks your remote — an EVIDENCED verdict means the session's own record backs the claim, and nothing stronger. Work you did outside the session (by hand, in a GUI, in a second session) is invisible to it, so claims about that work read UNEVIDENCED. Hosted receipts services give you stronger proof for calls you route through them; this is for the other sessions — the ones already sitting on your disk, uninstrumented, where the only question is whether the summary and the record agree.

Two details I care about for sharing results safely: excerpts are truncated to 120 characters, and --no-excerpts removes every transcript-derived string from the output — claim types, line numbers, verdicts and reasons only — so an audit can go into a PR thread without leaking what the session touched. --json emits the same audit as one object; exit 1 means "a claim needs a human look," which makes it a one-line gate in a review script.

Repo: github.com/jasveerai88/done-claim-auditor — MIT, 36 tests, demo GIF on the README. It's the fourth of a small set of read-only agent audits (skill packs, permission allowlists, worktrees, and now claims). The bug I most want reported is a false EVIDENCED: a claim this tool backs that the record doesn't. Everything else is a phrasing away from fixed — the pattern table is explicit, and PRs that extend it with fixtures are welcome.