I Tested Three Claude Reasoning Effort Policies on 20 of My Own Bug Fixes

Originally published on hexisteme notes.

I pulled 30 days of my own Claude Code usage and checked which model and which reasoning effort level each message ran at. The model choices matched Anthropic's guidance. The effort levels did not. My main loop on Opus 5.5 ran at xhigh for 77% of messages and max for 23%, with medium at 0%. Subagent messages on Opus 5.5 were no better: xhigh 77%, max 14%, medium 8%.

The cause was one block in my settings. A modelSettings entry pinned effortLevel: xhigh for every model entry alike, Opus and Sonnet included. That meant routing a subagent down to Sonnet changed the model but not the effort, so it still ran at xhigh. Anthropic's guidance points subagents toward low.

Deviation is not the same as waste

The tempting conclusion was that I had been burning money for a month. I did not have evidence for that. Anthropic's published numbers come from a single-shot benchmark. My sessions are long and interactive, with a lot of back-and-forth, so the deviation from the guide was a fact and the waste was only a guess. I wrote the verdict down that way: model selection matches the guide, effort deviates heavily, and nothing yet shows the deviation costs me anything.

Here are Anthropic's published numbers I was comparing against, from their page on optimizing for cost and intelligence, measured on SWE-bench Pro:

• low effort, with failures rerun at high: about 97%, $0.17 per task
• medium, with failures rerun at high: about 97%, $0.24 per task
• high on everything: 95.3%, $0.29 per task

The same page attaches a precondition: these policies make sense when you have a way to check the output, such as tests or a verifier. The effort documentation also says the API default for Opus 5.5 is medium. So my setting was not just above the subagent guidance. It was above the default.

The first count was wrong by 2.5x

Before any of this, my first pass at the audit overcounted tokens by 2.5x. The transcript logs write streaming output as several lines that share one message ID, and I had summed every line. Codex pointed it out. The fixed version deduplicates by message ID, and that is the version the percentages above come from.

Designing a test I could afford

I asked Codex and Gemini (through agy) for independent opinions on how to check whether lower effort would hurt me, then compared all three positions. We agreed on several constraints before anything ran.

The first was statistical power. Codex worked out that 20 tasks cannot resolve a 5-point threshold: even if the cheaper policy failed zero tasks, the one-sided 95% upper bound on its failure rate would still be 13.9%. At 60 tasks that bound drops to 4.87%. So 20 tasks would be exploratory, and 60 would be the size for an actual decision.

The second was where tasks would come from. I wanted real fixes from my own work, each with a test. Counting fix commits that came with tests, one repository had 61, another had 3, and the rest had 0 to 1. So any result would only speak for that one repository.

The third was leakage. If the model can see the future git history, it can find the answer instead of writing it. Each task ran on a copy of the repository without the future history. Grading happened outside the run with a separate grader that checked the fixed tests, required zero skipped tests, compared the number of collected tests, and checked which paths were changed.

There was one disagreement. agy recommended switching my main loop to medium immediately. Codex said a global switch should wait until after the test. I went with Codex: a reversible pilot, then a measurement, then a decision.

Running Claude headless is something my own setup treats as an explicit opt-in, so the design waited for my approval before any of it ran.

The design

• Task selection: group the fix commits by cause, and keep only tasks where the reference fix passes and reverting it makes the regression test fail.
• Three arms on Opus 5.5: A at xhigh (my current setting), B at medium, and C at low with one retry at high if the first attempt failed.
• Each run was claude -p --effort X --output-format json with a fixed turn limit, in randomized order.
• Metrics: paired success difference per task, and total cost including failed attempts (total_cost_usd), time, and turns.
• Falsifier, written before the runs: if B or C loses at most 1 task in 20 against A in paired comparison, and its total cost is at most 70% of A's, lower subagent effort first as a pilot. Otherwise keep xhigh.

Results

I ran it on November 11: 61 runs on 20 fix commits from that one repository. Twenty per arm, plus the one high-effort retry C needed.

The paired comparison is the part that matters, because the pass totals hide which tasks moved.

• A to C: 0 losses, 0 gains. The 95% upper bound on the loss rate is 13.9%.
• A to B: 1 loss, 0 gains, with an upper bound of 21.6%. On the lost task, B's change made the target test pass but broke 2 other regression tests. A grader that checked only the target test would have scored that as a pass.

One task failed under all three policies. Its fixed signature test, test_v2_known_vector_cross_repo, required a byte convention that the commit title alone did not determine, so no policy had enough information to produce it. I counted that as a defect in the task, not in any arm. Since all three failed it, it does not affect the paired comparison.

The integrity checks came back clean: 0 violations (protected files, conftest edits, skips, collected-test count), 0 changes to the original repository, and 0 leaks under the strict rule. During the runs a LEAK flag did fire, but it was a false positive on a path string inside the file being fixed.

What I changed, and what I did not

Both B and C met the falsifier rule. Following what we had agreed, I lowered subagents first and left the main loop at xhigh. The change was one environment variable: CLAUDE_CODE_SUBAGENT_EFFORT=medium, with a backup of the previous settings.

I chose medium, not low, even though C was the cheaper arm with no paired losses. C's result depends on a verifier: low effort is only safe there because a failed test triggers the retry at high. My subagents' outputs have no automatic verifier that would trigger a retry. The test also covered only bug fixes that tests can check, and a lot of my subagent work is research and exploration. Extrapolating from one to the other is weak.

I then checked that the change took effect, comparing before and after in the same session with the same general-purpose agent on Sonnet 5.5. At 08:02 it ran at xhigh. At 08:41, after the change, it ran at medium.

Checking that also showed me the precedence order, which turned out to matter. From highest to lowest: an effort field in the agent's frontmatter, then the effort passed on an individual call, then CLAUDE_CODE_SUBAGENT_EFFORT, then whatever the session inherits. A forked agent ignores the environment variable and takes the session's effort. My custom agents that declare effort in their frontmatter, some at high and some at medium, kept those values. So the environment variable only reaches subagents that set nothing themselves.

The limits of this result

This is an exploratory result and I am treating it as one. It covers one repository, the tasks were easy (xhigh passed 19/20), and n is 20. Codex's power calculation applies directly: with zero paired losses for C, the loss rate could still be as high as 13.9%, which is well above the 5-point threshold I would want for a real decision. The test says lower effort did not visibly hurt on these tasks. It does not say lower effort is safe in general.

The falsifier for the rollout is a 2-week check on November 25: if the rate at which my main Claude session has to redo subagent output goes up compared with before, I delete the environment variable line and revert. That check has not happened yet.

Lowering the main loop is a separate test. When I run it, it will start with work that has a verifier, meaning code changes with tests, under the low-then-high retry policy, which is also the precondition Anthropic's page attaches to its own numbers.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.