I build a small tool called opendrs. It gives a pull request a risk score from 0 to 1, using only the repo's git history. I wrote about the tool itself here. This post isn't about the tool. It's about what I saw when I pointed it at other people's projects.
Over the last month I ran it on 33 open-source repos: 27 replays of up to 4,000 commits each (express, django, cargo, rails, tokio, prettier and more), and 1,010 merged PRs from six active repos (black, cli/cli, actix-web, vite, fastlane, mockito) that I checked against what happened after merge.
What "needed a fix" means here
None of these projects publish an incident list, so I can't say which PR caused an outage. What I can see is:
- a revert ("This reverts commit ...")
- a later fix commit whose changed lines, blamed back with
git blame, land on lines that PR wrote (this is called SZZ)
For the 1,010 PRs I only counted fixes within 90 days of the merge. So "fixed later" below means "a later fix touched lines this PR wrote". It doesn't mean the PR was bad. SZZ is noisy: a commit called "fix typo" counts as a fix, and a refactor can get blamed for a bug that was older. Read everything here with that in mind.
1. Size still matters most
Of the 1,010 PRs, 99 were later fixed or reverted (9.8%). Split by how many files the PR changed:
| Files changed | PRs | Later fixed or reverted |
|---|---|---|
| 1-2 | 550 | 25 (4.5%) |
| 3-5 | 275 | 25 (9.1%) |
| 6-15 | 149 | 33 (22.1%) |
| 16 or more | 36 | 16 (44.4%) |
Nothing new, every study of this says the same. But it's still striking to see it on six projects with careful review. A PR that touches 16+ files was about ten times as likely to get a later fix as one that touches one or two. Part of that is just more lines to blame. Still, if you only take one thing from this post, it's "split the big PR".
2. Dependency manifests get fixed later more than anything else
opendrs flags a PR that adds a new dependency to package.json, Cargo.toml, go.mod, pyproject.toml and so on. On the six repos, the manifest it named got a later fix 23.3% of the time. The other files in the same PRs: 5.0%. That's 4.6 times as often, the strongest signal of anything I measured. Reviewers did look at these files a bit more than at the rest of the PR (1.32 times as many comments), and they still got fixed later 4.6 times as often.
In vite, the single file blamed by the most fixes was packages/vite/package.json (6 PRs). Two honest caveats: Conventional Commits like fix(deps): ... count as fixes in my labels, so routine dependency bumps can show up here too. And it's only 43 cases, 10 of them from one cli/cli PR.
Environment variables showed a different pattern. When a PR added a new env var read, reviewers commented on that file less than on the PR's other files (0.23 times as often), but it was fixed later more often (1.44 times). That's only 26 cases, and 10 of them come from one generated file in cli/cli, so treat it as a hint. My guess is that an env var read just looks boring in a diff.
3. Some files are hotspots by design
When I looked at which files the later fixes landed on, the lists read like a map of each project's hard parts:
- black:
src/black/comments.py(4 PRs),src/black/lines.py(3),src/black/linegen.py(2). Comment handling in a formatter is famously fiddly. - vite:
src/node/plugins/css.ts(5 PRs),src/node/config.ts(4),src/node/plugins/importAnalysisBuild.ts(4). - fastlane:
sigh/lib/assets/resign.sh(2 PRs), the code-signing script.
These files aren't badly written. They are where the hard problem lives, so everyone has to touch them. That has a side effect on scoring. In black, linegen.py and nodes.py have been touched by 200-350 different people. An untuned score counts that history, so in an early field test 7 of 8 non-docs PRs on black and 6 of 8 PRs on vite scored red, whether the PR was 22 lines or 759. A raw count of past fixes tells you the file is hard. It doesn't tell you this PR is risky.
The fastlane one is the inverse. A real security fix in resign.sh (blocking a symlink escape) scored green, because the default "critical paths" are folders like auth/ and security/. Code signing lives in a shell script under sigh/. No generic default would catch that, each project has to name its own critical files.
4. Projects label themselves very differently
The share of commits that ended up labelled "later fixed" ranged a lot:
- spring-boot: 152 of 4,000 (3.8%)
- quarkus: 497 of 6,000 (8.3%)
- rails: 351 of 4,000 (8.8%)
- django: 1,102 of 4,000 (27.6%)
- cargo: 1,163 of 4,000 (29.1%)
Same for the six PR repos: vite 17.6%, cli/cli 10.7%, black 10.0%, mockito 6.0%, fastlane 5.5%, actix-web 5.1%.
I don't read this as "cargo has 8 times more bugs than spring-boot". It's mostly how people write commit messages. If a project says "Fix ..." for everything, more commits look like fixes and more lines get blamed. So comparing raw rates across projects tells you more about their habits than their code.
5. Fitting to each repo helped some a lot and hurt others
opendrs can fit its weights to a repo's own history (calibrate). I compared that with the untuned weights on the newest commits, which the fit never saw:
| Repo | Untuned | Fitted to the repo |
|---|---|---|
| prettier | 0.71 | 0.85 |
| cargo | 0.61 | 0.75 |
| etcd | 0.63 | 0.76 |
| express | 0.62 | 0.73 |
| spring-boot | 0.61 | 0.62 |
| requests | 0.79 | 0.71 |
| eslint | 0.48 | 0.40 |
(AUC: 0.5 is a coin flip, 1.0 is perfect ranking.)
In the first nine repos, fitting helped every time. In the next 18, it helped in 11, was flat in 1 and hurt in 6. The ones it hurt mostly had small windows, fewer than 15 fixed commits in some, so the numbers jump around. Since then calibrate keeps the default weights when the fit doesn't beat them.
The two Java repos, spring-boot and quarkus, barely moved (+0.01 each). On the Apache Java projects in the public ApacheJIT dataset, fitting helped in all 14 with labels, but groovy is the odd one out: 0.609 for opendrs, 0.760 for a classic regression on the dataset's own metrics. Whatever predicts bugs in groovy, plain git history doesn't see it well.
6. Where it simply failed
- serde was close to a coin flip untuned (0.51). serde is a small, very stable codebase written mostly by a few people, so "how many people touched this file" and "how much has it changed" carry almost no information there.
- tokio, terraform, eslint looked near random (0.45-0.50) on their newest commits. Not because the score was wrong, but because those commits were weeks old and their bugs hadn't been found yet. Every "clean" recent commit was just "not fixed yet". That's why opendrs now leaves out the newest 60 days when it evaluates.
I re-ran tokio this week with the current version to check. With the newest 60 days left out, the untuned score ranked its last 1,314 commits at AUC 0.72 (131 of them later fixed), and the riskiest 10% held 27.5% of the fixed ones. The fitted model's own check only had 12 fixed commits to test on, so I can't tell you anything about fitting tokio yet.
- Languages. In the field test, Go import resolution in cli/cli picked a test-fixture
go.modthat declares the same module path as the real one, so heavily imported files looked like they had zero dependents. Ruby had no public-API detection at all, and Swift had no import parsing. The score didn't crash, it just went quiet. The Go and Ruby ones are fixed now, Swift only partly. But "silent" is the failure mode to look out for in any tool like this.
What I take from it
- Big PRs still get fixed later far more often. Splitting them is the cheapest risk reduction there is.
- New dependencies and env var reads deserve more care than they look like they need.
- A file being a hotspot is a fact about the project, not about the PR. Don't let it drown out the PR itself.
- Don't compare fix rates across projects. Compare a project with itself.
If you maintain one of these projects and something here looks wrong to you, I'd like to hear it. The raw data (PR list, labels, per-repo tables) is in the repo under bench/outcomes/ and docs/results.md: https://github.com/error-path/opendrs
