The diff was one line. claude-sonnet-4-5-20250929 out, claude-sonnet-5-5 in. It was correct, it was tidy, and every check we had said yes.
It would also have failed on the first request.
I build Since, which tracks the packages, models, APIs and runtimes code depends on and opens the fix pull request when one of them changes. So this was not a side project bug. This was the product doing its one job and getting it wrong in a way that looked exactly like getting it right.
Here is how we found it, what was actually wrong, and the part that still bugs me.
1. The setup
Anthropic retires claude-sonnet-4-5-20250929 on November 30, 2026. Their deprecations page names claude-sonnet-5-5 as the replacement (re-read today, Nov 15). A retirement like that is the cleanest case there is: a known date, a named replacement, a string in your code. If any fix should be mechanical, it is this one.
So the fix was mechanical. Find the string, swap it, open the pull request.
2. The question every review asked
We had specs for the fix. We had reviews of the specs. We had reviews of the code. When we went back over them on Oct 26, every one of them had asked the same question:
Does this edit fix the old change?
And the answer was yes. The old model ID was gone. The new one was in its place. Nothing in the diff was wrong.
What nobody asked was the opposite question: what does the new thing remove, rename or newly require?
3. The clue was not in the diff
The replacement model does not accept the same request the old one did. Anthropic's deprecations page has a short table under "API parameter deprecations". temperature, top_p and top_k are removed in API version 2026-10-21 and return a 400 error. On earlier API versions they return the error when set to a non-default value on Claude 4.7 and later models.
claude-sonnet-5-5 is a later model. The code we were fixing sent temperature: 0.2.
The worst part is where it sent it from. Not the line we changed. A shared options object in another file, spread into the request. Here is the shape, cut down to the bone:
// src/defaults.js: shared request options, written a year ago export const defaults = { temperature: 0.2, max_tokens: 1024 }; // src/summarize.js import { defaults } from "./defaults.js"; export function buildRequest(text) { return { ...defaults, model: "claude-sonnet-5-5", // the one-line fix messages: [{ role: "user", content: `Summarize: ${text}` }], }; } I ran it in an empty folder with a two-line script that prints the request:
{ "temperature": 0.2, "max_tokens": 1024, "model": "claude-sonnet-5-5", "messages": [ { "role": "user", "content": "Summarize: release notes" } ] } And grep -n temperature src/summarize.js finds nothing. The file you fixed is clean. The request it builds is not.
A reviewer looking at the diff sees one correct line. A reviewer looking at the file sees one correct line and a spread. You only see the problem if you go and ask what the target model refuses, and then go looking for that across the whole repo.
4. The fix that looks right
The obvious patch is a rule: when you swap to a model that rejects sampling parameters, delete temperature next to the swap.
That handles the case where temperature sits on the next line. It does nothing for the spread from another file, the wrapper function with a default argument, the config object built in a barrel file, or the alias someone set up so they could switch models from an environment variable. Those are not exotic. That is just what a codebase looks like after a couple of years and a few contributors.
It also has a nasty false positive. One of the cases in our own test set has a field called temperature that has nothing to do with a model at all. It is a mood score. No request ever sends it. A rule that deletes temperature near a model swap would happily delete it.
So the rule was wrong in both directions: it missed the real ones and it would have broken an innocent one.
5. The real fix: treat a fix as a change
The thing that changed was the question, not the code.
A fix is itself a change to what your code depends on. You are moving from one model, one package version, one runtime, to another. That target has its own list of things it removed, renamed and now requires. So every fix now gets checked against both sides:
- Does the edit fix the old change? (The question we always asked.)
- What does the target remove, rename or newly require?
- Where else in the whole repository does the code touch any of that? Wrappers, spread options built in other files, re-exports, aliases, config.
Then either fix those places in the same pull request, or don't open it and say exactly which places are in the way. A fix that leaves you one deploy away from a 400 is not a fix. It is a different outage with a nicer commit message.
The commit that proved it landed the same day, Oct 26, with three cases in its message: a client hidden behind a module's default export, a swap that would have left a duplicate key, and the mood score called temperature that no request sends, which is now left alone, signature and all.
6. The test set was too clean
The deeper problem was not the reviews. It was what we tested on.
Our early examples were one file, one call, one string. Every fix looks perfect on a one-file example, because the only place the problem can be is the line you changed. Real code spreads options. Real code wraps clients. Real code has a variable called temperature that means something else.
So the test set changed. Our demo set now has 70 messy cases across models and packages, plus 18 API cases, each with the exact lines it should touch and the result it should get, and with traps for false positives built in on purpose. A test case that cannot fail the way production fails is not testing much.
7. It happened again, differently
I would love to say that was the end of it. On Nov 12 a pull request on an open-source project showed the same family of bug from another angle. A Google preview model was retiring, and the replacement we named handled function calls with a different default. Every place Since showed the change said "new ID". None said "and it behaves differently".
Same lesson, other end of the pipe. A replacement is not a synonym. So now every replacement Since names comes with what works differently about it, wherever it is shown, read from the provider's own documentation.
What I'd tell myself before
The bug that gets past every review is the one in the question everybody is asking. We had plenty of review. All of it was pointed at the old thing.
If I had to compress it into one line: the target of a migration is a dependency too, and it has its own changelog.
When this is overkill
Not every swap needs a whole-repo search. A one-file script that calls one model, that you will run once, is fine with a grep and a look. The check earns its keep when options are shared, when the same client is used from many places, and when nobody remembers who set temperature: 0.2 or why.
What's the worst bug you shipped in a change that only touched one line?
I'm Nabil. I build Since, which tracks what your code depends on and tells you before it breaks.
