We just put a page on the web for every package we have read breaking changes for: 2,495 of them this morning, on npm, PyPI, crates.io and eleven other registries. Each one lists what a release removed, renamed or changed, with its source.

The package at the very top, with more breaking changes than anything else on the list, is happy-dom. 13,444 of them in 30 versions.

It did not break 13,444 times. Almost none of that is real. This is the story of the fake number, the cleanup we wrote to remove it, and the dry run that showed the cleanup wanted to rewrite 31,368 rows that were perfectly fine.

1. The package that broke on every patch

On Nov 17, while we were getting the package pages ready, happy-dom stood out by a mile. Every patch release "removed" hundreds of members, 429 of them with names like this:

AbortController.__@window@3465511 

That is TypeScript's internal name for a member keyed by a symbol, and the number at the end is the symbol's id in one particular compile. Build the same code again and the number changes. So every release looked like it had deleted a pile of members and added a pile of identical ones with new numbers. Nothing had changed. The names were just not stable from one build to the next.

The fix going forward was easy: name such a member the way a person writes it, [window] or [Symbol.iterator], so two builds of the same code agree.

That left the rows already stored. Production held 19,784 of them across 52 packages on Nov 17. They had leaked into answers and fix inputs, and pushed packages up the rankings. If you open happy-dom's page this morning (Nov 18), it still says 13,444. That number is the bug. They had to go.

2. The cleanup

We already had a backfill job, a script that walks stored releases and brings their rows up to date with today's rules. Adding one rule to it was the obvious move: a row keyed by a name with a compile-time symbol id is withdrawn as "no longer found".

We ran it on production with writes turned off first. Not out of wisdom. It is just how that script runs unless you tell it otherwise.

The dry run said:

What it would do Rows
Withdraw (the symbol-id rows) 19,829
Rewrite a title 31,368

The first number made sense. It had grown a little since the day before, to 19,829 across 53 packages. The second number made no sense at all. The rule we had added does not touch titles.

3. The obvious explanation

The obvious explanation was that the titles really were stale. The backfill rebuilds a row's title with today's titling code, and if titles had been written by older code, a rebuild would update them. Thirty thousand old titles getting a refresh sounded almost plausible.

So we checked when the rows were written.

Every live row of that kind in production, 559,035 of them, was written after Oct 29. The code that writes titles had not changed since Oct 28. Every one of those titles was written by the exact code the backfill was about to use to "fix" them.

There was no such thing as an old title in that table. Whatever the backfill was doing, it was not catching up. It was drifting.

4. The real cause, part one: the row did not hold everything the title came from

When the reading of a release finds two changes that are really one (say, two members deprecated together), it folds them into one row with a title that names both:

napi::Env::create_arraybuffer and ...create_arraybuffer_with_data are deprecated in napi 3.0.0 

The row keeps the list of members it covers, but in a separate field from the one the backfill read when it rebuilt titles. So the rebuild saw one member, wrote a title naming one member, and called that an update. Rebuilt from the stored row, that title lost its second half.

The title was right. The rebuild was the thing that was wrong, because it was rebuilding from less than the original writer had.

5. The real cause, part two: the same name, spelled two ways

The other kind was quieter. When a release is first read, the package is named the way its registry spells it: NumSharp. The backfill named it by an internal key that is lowercased for matching: numsharp.

Same package, a different string. Every title that contained the name differed by case, so every one of them looked "changed" and went on the rewrite list.

Neither of these is a dramatic bug. Both are the same mistake: a correction job recomputed a value from what it had, instead of from what the original writer had, and treated every difference as a fix.

6. The fix

The rule we landed on: a title stands as the reading wrote it, unless this run changes what the row names. If the run folds a copy into a row, or drops a claim so the row now says something different, the title is rebuilt. Otherwise the title is not the backfill's business.

The package name now comes from the registry's own spelling, the same place the original reading got it.

While we were in there, a second read-through found one more disagreement. The backfill marked a rule as stale for a copy of a member under another entry point of the same package, because its name split ignored a single :. The code that stores new changes already split it correctly. Two places answering the same question in two ways is a bug by itself, so both now call one function.

Then we ran the read-only pass again over all 14,631 production release steps:

What it would do Before the fix After
Withdraw (the symbol-id rows) 19,829 19,829
Rewrite a title 31,368 0
Fold real duplicates 20
Fix a row's member list 19
Mark a rule stale 0

The withdrawals did not move. The 31,368 went to zero. The 20 folds are real duplicates, like a rename stored once from the release notes and once from a diff of the code, which should always have been one row.

That is the run that gets to write.

7. What made the difference

Honestly, nothing clever. The script defaults to a dry run, and the dry run reports each kind of change as its own count, not one total. If it had printed "51,197 rows updated", it would have been easy to shrug. Two separate lines, one of them for a kind of change nobody asked for, are hard to shrug at.

And the question that cracked it was not "is the new rule right?". The new rule was fine. It was "when were these rows written, and by what code?". Once the answer was "by today's code, all of them", there was nothing left for a title rewrite to correct, so every rewrite had to be a bug.

What I'd tell myself before

A correction job carries two kinds of risk. There is the rule you meant to add, which you test. And there is everything else the job does on every run, which nobody looks at any more because it has always been a no-op. The second kind is the one that rewrites 31,000 good rows.

Before a stored-data correction writes anything, ask when the rows were written. A rule meant for rows older than the fix may match nothing, and then it rewrites what is right.

And if you want to see what the packages you use really broke, the pages are at since.dev/packages. Take happy-dom's number with a grain of salt until the cleanup lands.

What's the closest a migration or backfill of yours came to rewriting data that was already correct?

I'm Nabil. I build Since, which tracks what your code depends on and tells you before it breaks.