You've probably written this into a Claude Code prompt:
"Make sure you handle all the edge cases."
"Write the tests first."
Or you bumped effort to high. So did I. So I measured whether it does anything. 504 runs later: it mostly makes the bill bigger.
- Haiku 5.5: "handle edge cases" + "read the existing tests first" raised cost per run by 134%
- Opus 5.5: the edge-case line alone raised cost per run by 60%
- effort high: +60% for Haiku, +22% for Opus
- Share of the cases the bug report left out that the fix actually handled: no condition moved it by more than 4 points. Effort high made it slightly worse for both models.
The setup
21 real bugs fixed upstream between August and October 2026 in 12 open-source projects: cachetools, zod, marked, js-yaml, anyio, undici, fastify, p-queue, python-dateutil, Python-Markdown, qs, tomlkit. Claude Code 2.1.295, headless, Haiku 5.5 and Opus 5.5, four conditions, 3 runs each. 21 × 2 × 4 × 3 = 504.
| Condition | What changed |
|---|---|
| Baseline | Bug report cut to symptom, repro, expected result |
| Effort high | Same report, --effort high |
| Edge-case prompt | + "Before changing code, list the inputs and variants this report does not mention but the fix must also handle. Write tests for them first, then fix." |
| Edge-case + read tests | + "Read the existing tests for the code you change before you start." |
Scoring used the maintainers' own tests for each fix. Before the first run I labelled every test case as "shown in the report" or "left out" and pre-registered it on GitHub with sha256 hashes, so I couldn't move the goalposts later. The bar for calling something an effect: 20 points or more.
Results
Share of the left-out cases passed (mean per bug):
| Condition | Haiku 5.5 | Opus 5.5 |
|---|---|---|
| Baseline | 84% | 88% |
| Effort high | 82% | 86% |
| Edge-case prompt | 84% | 91% |
| Edge-case + read tests | 86% | 84% |
API cost per run:
| Condition | Haiku | Opus |
|---|---|---|
| Baseline | $0.050 | $0.51 |
| Effort high | $0.081 (+60%) | $0.62 (+22%) |
| Edge-case prompt | $0.10 (+108%) | $0.82 (+60%) |
| Edge-case + read tests | $0.12 (+134%) | $0.81 (+58%) |
Median time per run went from 121s to 299s for Haiku and 120s to 271s for Opus. The models did exactly what they were told: listed variants, wrote tests. It just didn't turn into better fixes.
Why it didn't work: the bug decided, not the prompt
- Easy bugs were handled anyway. On 12 of 21 bugs both models passed every left-out case in every condition. Nothing to gain.
- Hard bugs stayed missed. On a js-yaml bug (a BOM before each document in a stream, not just the first), 0 of 504 runs handled it. Asking "list the variants the report doesn't mention" didn't make the model think of that variant.
- "Read the tests" sometimes hurt. On a Python-Markdown backtick bug, Opus passed the left-out case in 2 of 3 runs at baseline and 0 of 3 with the read-tests line.
What to do instead
- Name the edge cases yourself. Both models passed over 94% of the cases the report did show. They don't fail at following instructions; they fail at imagining what you didn't say.
- Leave effort at the default for bug fixing. Here, high only added cost.
- Default to Haiku. Across 252 runs each: Haiku $0.088 per run, Opus $0.69, about 87% less. Opus was a few points ahead and broke other tests less often, so it still earns its price on hard, multi-place bugs in big codebases.
Limits
21 bugs, 3 runs per cell. A 20-point effect is ruled out; a few points is not. One prompt wording per condition; a different wording or agent might do better.
Data
All 504 runs (scores, cost, time, tokens) are public under CC BY 4.0, with the pre-registration and the scoring code:
- Dataset (DOI): https://doi.org/10.5281/zenodo.23796306
- Pre-registration: https://github.com/johnisking/token-cost-by-language/blob/main/claude-report-edge-cases/PREREG.md
What do you add to your Claude prompts that actually works? Drop it in the comments and I'll test it in the next round.
