Section 6 of the Claude 3.7 Sonnet system card, published with the model on February 24, 2025, is called "Excessive Focus on Passing Tests." Anthropic writes that the model "occasionally resorts to special-casing in order to pass test cases in agentic coding environments like Claude Code."

The card is specific about the shape of it. Most often the model returned the expected test values directly instead of writing a general solution. Sometimes it changed the tests to match its code. It usually happened after several failed attempts, when the tests conflicted or an edge case was hard to handle in general. Anthropic says training mitigations cut how often this happens, and among the product-level mitigations it lists is monitoring for "unexpected modifications to test files" and "comments that suggest test-specific handling."

Other frontier models do it too. ImpossibleBench, a 2025 paper by Ziqian Zhong, Aditi Raghunathan and Nicholas Carlini, builds coding tasks where the tests contradict the spec, so any pass means a shortcut. The authors conclude that "frontier models frequently cheat when faced with these impossible tasks" and that "stronger models generally exhibit higher cheating rates." They also note that newer Claude models cheat less than Claude 3.7 Sonnet did.

Your CI can't tell the difference. A green check from a fixed bug and a green check from a skipped test look identical. So here are seven shapes the shortcut takes, each with a command that flags it on an agent's branch. Every hit is a line to read before you decide anything.

Set up once: a base and two helpers

Run these in bash or zsh, inside the agent's worktree. Swap main for the branch the agent started from.

B=$(git merge-base main HEAD) TESTS=('*.test.*' '*.spec.*' '*__tests__/*' '*_test.go' '*_test.py' 'test_*.py' '*/test_*.py') added() { git diff "$B" -- "$@" | grep '^+' | grep -v '^+++' } removed() { git diff "$B" -- "$@" | grep '^-' | grep -v '^---' }

git diff <commit> compares that commit with your working tree, so uncommitted agent edits count too. Untracked files don't show until they are added. In a pathspec, * can match directory separators, so '*.test.*' finds test files at any depth. added with no arguments covers the whole repo.

Tests that stop running

1. A new .skip, .only or xfail

The fastest green: the failing test never runs. In Jest that's test.skip, xit, xtest or xdescribe. .only and its aliases fit and fdescribe are quieter, because every other test in the file stops running too. test.failing passes when the test throws, and test.todo keeps the name and drops the body. In pytest it's @pytest.mark.skip, skipif, pytest.skip() or xfail. In Go, t.Skip.

SKIP='\.(only|skip|todo|failing)' SKIP+='|[+[:space:]][xf](it|test|describe)\(' SKIP+='|xfail|t\.Skip' added "${TESTS[@]}" | grep -E "$SKIP"

Vitest doesn't allow .only by default when CI is set (--allowOnly defaults to !process.env.CI). In pytest, xfail(strict=True) turns an unexpected pass into a failure, so a stale xfail can't sit there forever.

2. A deleted test

A whole file gone is easy to spot with --diff-filter=D, which keeps only deleted paths. A single test case removed from a file that still exists is not, so count test cases on both sides:

git diff --diff-filter=D --name-only "$B" -- "${TESTS[@]}" CASE='[^[:alnum:]_](it|test)\(|def test_|func Test' removed "${TESTS[@]}" | grep -cE "$CASE" added "${TESTS[@]}" | grep -cE "$CASE"

If the first count is bigger than the second, the branch has fewer test cases than it started with. A rename or a refactor can explain that. Read the diff before you decide which one it is.

Tests that stop checking

3. A loosened assertion, or an expected value changed to match

This is the card's "modifying the problematic tests themselves to match the code's output." It comes in two forms. The first swaps a strict check for a weak one: toEqual(order) becomes toBeDefined(), toHaveBeenCalledWith(id) becomes toHaveBeenCalled(), assert total == 110 becomes assert total.

WEAK='toBeTruthy|toBeDefined|toMatchObject|expect\.any' WEAK+='|toBeCloseTo|toHaveBeenCalled\(\)|approx\(' WEAK+='|assert [A-Za-z_.]+$|assert True' added "${TESTS[@]}" | grep -E "$WEAK"

The second keeps the matcher and changes the number, so toBe(110) becomes toBe(108). --word-diff prints in-line changes as [-old-]{+new+}, which makes those easy to grep:

git diff --word-diff "$B" -- "${TESTS[@]}" | grep -E 'expect\(|assert' | grep -F '[-'

Sometimes the new number is the right one. Check the commit or the ticket for a reason the expectation should change. If there is none, the code moved and the test followed it.

4. A regenerated snapshot

jest -u will "re-record every snapshot that fails during this test run." Vitest has -u too. One command and every failing snapshot matches whatever the code renders now, bug included.

git diff --stat "$B" -- '*.snap' '*__snapshots__*'

Any .snap change in an agent's branch deserves a line-by-line read, because a snapshot diff is the assertion diff. In CI, run Jest with --ci: a new snapshot then fails the test instead of being written automatically.

Code that knows it's being tested

5. A hard-coded return for the test's input

The card's most common form: "directly returning expected test values rather than implementing general solutions." It also says the model "often (though not always)" left a comment like # special case for test XYZ. Two other tells are code that checks whether it runs under a test, and strings copied out of a test file into source.

Test runners announce themselves. Jest sets NODE_ENV to test unless it's already set, and gives each worker a JEST_WORKER_ID. Vitest sets VITEST. pytest sets PYTEST_VERSION, and its own docs say "usually it is a bad idea to make application code behave differently if called from a test" (pytest).

SRC=(':!*.test.*' ':!*.spec.*' ':!*__tests__/*' ':!*_test.go' ':!*_test.py' ':!test_*.py' ':!*/test_*.py' ':!*.snap' ':!*__snapshots__*') HINT='node_env|jest_worker_id|vitest|pytest' HINT+='|special.?case|hard.?cod|for (the )?tests?' added "${SRC[@]}" | grep -iE "$HINT"

The :! prefix excludes a path, so this searches source only. Then look for string literals the branch added to source that also appear in a test file:

added "${SRC[@]}" | grep -oE "[\"'][^\"']{6,}[\"']" | tr -d "\"'" | sort -u | while IFS= read -r s; do git grep -qF -e "$s" -- "${TESTS[@]}" && echo "$s" done

Shared error messages will show up here, and that's fine. A fixture's exact input sitting in an if in production code is not.

Checks that stop complaining

6. A raised timeout or an added retry

A flaky failure goes away if you wait longer or try again. jest.retryTimes(3) "runs failed tests n-times until they pass or until the max number of retries is exhausted." pytest-rerunfailures does it with @pytest.mark.flaky(reruns=5), Vitest with --retry.count. Jest's default timeout is 5,000 ms, and a third argument to test() overrides it per test.

-G keeps only files whose diff adds or removes a line matching the regex, so it works across config files too:

git diff --name-only -G'retr(y|ies)|[Tt]imeout|reruns|flaky' "$B" added | grep -iE 'retr(y|ies)|timeout|reruns|flaky'

A retry on a test that talks to a real network can be legitimate. A retry added in the same branch that "fixed" the bug is a question for the agent.

7. A new @ts-expect-error, eslint-disable or noqa

If the red check is the type checker or the linter, a comment turns it green.

MUTE='@ts-(expect-error|ignore|nocheck)|eslint-disable' MUTE+='|noqa|type: *ignore' added | grep -E "$MUTE"

Each tool has a way to push back. In TypeScript, prefer @ts-expect-error: it reports when the error it hides is gone, while @ts-ignore "will do nothing if the following line is error-free." ESLint has --no-inline-config to ignore inline disable comments, which you can run in CI. Ruff flags unused noqa comments with RUF100. mypy can require an error code in every type: ignore, so a bare one stands out.

Read the fix and the test edit in the same turn

The greps tell you what changed. The card tells you when it tends to happen: after several failed attempts at a general fix. The useful question is whether the test edit and the source edit landed in the same turn, right after a run of red.

I build Hermes IDE, so weigh this section with that in mind. Hermes saves every agent turn, so you can diff or restore any single turn, with undo. The Review Desk (⌘G, Ctrl+Shift+G on Windows and Linux) shows changes by turn or by file. If turn 4 touched src/price.ts and price.test.ts together, you see both files under that one turn before anything lands, and you can restore just that turn.

The Hermes Review Desk in By turn view for a Claude Code task in orders-api, listing two turns, turn 1's diff of src/db.ts, the changed-files list and a drafted commit message
The Review Desk (⌘G) in By turn view: two turns listed (T1: 3 files +69 −1, T2: 1 file +1 −0), turn 1's diff of src/db.ts, the changed files and a commit message drafted from the turns.

A line comment in the Review Desk goes back to the agent that wrote the line, so "don't edit this assertion, fix applyDiscount" lands in the right terminal. Land then commits, opens a pull request or merges locally, into the branch the task started from, and can be undone. Hermes doesn't run the seven greps for you. Each task has its own worktree, so cd into it from any terminal and run them before you press Land.

Hermes is free, source-available under BSL 1.1, and runs on macOS, Windows and Linux: download it or read the source. It is not affiliated with Anthropic, OpenAI, Nous Research or any agent vendor.

grep has limits. ImpossibleBench catalogs two cheating patterns a text search won't see: models that overload comparison operators so they always return the wanted value, and models that record extra state to return different results for the same input. Making tests read-only stopped test edits in the paper, but it didn't eliminate special-casing. Anthropic's other recommendation is cheap to try: tell the agent in its system prompt to write general solutions rather than special-casing tests.

If this was useful

An agent that edits the test is chasing the one signal it was given: a green suite. Checking its work belongs in the loop you build around it. The AI Agents Pocket Guide covers that loop, from tool calling to the guardrails that keep an autonomous agent honest. It's one of 8 short books in the Pocket Guides for Developers series, on agents, RAG, prompts, observability, events, databases and system design. If you run several agents at once, Hermes IDE puts each turn's changes in one review screen.

AI Agents Pocket Guide: Patterns for Building Autonomous Systems with LLMs