2:07 a.m.
The phone buzzes. The nightly transactions job has failed again. You open the laptop, squint at the error, find the batch that came in twice, write a quick dedupe, rerun, watch it go green, and crawl back to bed.
Three days later finance asks why one region’s revenue is off.
If you’ve been in data engineering long enough, you’ve lived some version of that night. I’ve lived it for 15 years. So when teams started asking me whether they could hand that 2 a.m. page to an AI agent, I didn’t have a confident answer. And the question they always asked next was even harder: which model?
The small one is cheap and fast. The big one is supposedly smarter. Nobody wants to pay for the big one if the small one would do. I wanted a real answer, not a vibe. So I built a test and put three models on call: Claude Haiku, Sonnet and Opus.
How the test worked
The setup is a small card transaction pipeline: raw files land, three silver tables clean them, four gold tables aggregate them. Think of a typical medallion pipeline, shrunk down to run on a laptop with DuckDB.
I broke it 64 different ways, all things I’ve seen in production:
- a column renamed upstream
- a batch delivered twice
- corrected records resent later, where the newest copy must win
- one batch that quietly switched to UTC
- a dimension table that picked up history rows
Twelve of the 64 have no safe fix from the evidence. The right answer there is to page a human and touch nothing.
Each model got what an on-call engineer would get: the error or the alerts, the pipeline SQL, a profile of every column, a few sample rows per batch, and the change notes from the source team. It had to answer with a decision (fix or escalate), a confidence score, and the SQL.
Then came the part most evaluations skip. I actually ran every proposed fix and compared every output table, row by row, with a clean build. A fix only counted if the data came out exactly right. “The job runs again” didn’t count.
The scoreboard
Sonnet and Opus tied exactly, and Opus cost about four times as much to get there.
| Model | Fixed correctly | Wrote wrong data | Paged a human when it didn’t need to | Paged a human correctly | Avg. output tokens | Relative cost |
|---|---|---|---|---|---|---|
| Haiku | 38 | 2 | 12 | 12 | 110 | 0.16 |
| Sonnet | 50 | 0 | 2 | 12 | 258 | 0.63 |
| Opus | 50 | 0 | 2 | 12 | 1,198 | 2.56 |
The relative cost is an index built from token counts and list-price ratios, so it’s only useful for comparing the three. It’s not a dollar figure.
A few things jumped out.
All three knew when to stop. Every model escalated all 12 incidents that had no safe fix. None of them tried to be a hero on those, which honestly surprised me.
Opus wasn’t smarter here, just wordier. It wrote about 1,200 tokens per answer against Sonnet’s 260. Most of the extra cost is extra explanation, not extra correct fixes.
Sonnet and Opus missed the same two. Both escalated one seed of a merchant table with duplicate rows, because the evidence didn’t show whether the duplicates disagreed. Both also escalated one case of a lateness filter whose intent was unclear. I scored those as unnecessary, but I’d have been fine with a junior engineer making the same call.
Where Haiku lost
Haiku failed in two different ways, and only one of them is cheap.
The cheap failure: it escalated 12 incidents it could have fixed. Mostly duplicate handling, account numbers that lost their leading zeros, and merchant IDs that gained a prefix. That costs a human some time, but no data gets hurt.
The expensive failure: two fixes that ran fine and produced wrong data.
- On a case where corrected transactions were resent later, it kept the oldest copy instead of the newest. Its own diagnosis said the copies “might differ.” It went ahead anyway, at confidence 0.6.
- On a customer table that picked up history rows, it fixed the gold totals but left the history rows in the silver table, where any other consumer would still read them. It reported confidence 0.85.
That second one is the kind of fix that passes review. The dashboard looks right. The table underneath isn’t.
Then I took away the change notes
In real life, the source team doesn’t always tell you what changed. So I reran the 18 incidents that depended on a change note, with those notes removed. This is where the tiers finally separated.
| Model | Fixable cases solved, with notes | Without notes |
|---|---|---|
| Haiku | 12 of 14 | 6 of 14 |
| Sonnet | 13 of 14 | 11 of 14 |
| Opus | 13 of 14 | 13 of 14 |
Haiku halved its fix rate and escalated whenever the data didn’t explain itself. Sonnet dropped two, both on a partial time zone rollout.
Opus kept all 13. On the time zone case, it worked out from a handful of interleaved sample rows that one collector’s timestamps ran exactly six hours ahead of the other. No note told it that. It read the data.
So the honest answer to “is the expensive model worth it?” is: it depends on how well your organization documents its changes. With good change notes, Sonnet was enough. Without them, Opus was the only one that kept going. Writing better change notes might be cheaper than paying for the bigger model.
The part that actually worried me
Remember the 2 a.m. dedupe from the start? One incident in the benchmark was exactly that night, and it fooled all three models the same way. Size didn’t help at all.
A batch arrived with duplicated transactions, and some copies carried different amounts. Nobody knew which amount was final. The only right move was to escalate.
I generated this incident twice with different random samples.
- Run 1: one of the conflicting pairs happened to land in the sample rows the models saw. All three escalated. Correct.
- Run 2: the samples showed no conflict. All three deduplicated and kept one copy, silently picking one of two amounts. All three reported confidence 0.6, and two of them wrote in their own diagnosis that the copies weren’t known to be identical.
Same incident. Same models. Same prompt. The only thing that changed was which three rows happened to be in the sample. That was the whole difference between a safe page and quietly corrupted revenue. Not the model.
And here’s what really bothered me. I also ran 10 production-style data quality monitors after every fix: duplicate keys, volume against the 14-day median, nulls, reconciliation between tables, value domains and so on. Across 275 proposed fixes, those monitors never fired once. Not on the 5 fixes that corrupted data, not on anything.
That’s not a bug in my monitors. Monitors look for symptoms. A plausible fix removes the symptom. Keeping the wrong copy of a duplicate clears the duplicate-key alert just as well as keeping the right one. Monitors can tell you the data has a problem. They can’t tell you which of two clean-looking repairs is correct.
What I’d actually do
If I were putting an agent on data on-call tomorrow, here’s the setup the numbers point to.
- Start with the mid-tier model. Sonnet matched Opus whenever the evidence was complete, at about a quarter of the cost.
- Add a confidence floor. The models were badly calibrated overall, but the harmful fixes clustered at the bottom: four of the five came in at exactly 0.6. A simple rule of “page a human below 0.65” gave Sonnet zero harmful fixes in both conditions, while it still solved 47 to 49 of the 52 fixable incidents.
- Skip the clever cascade. I also tried “Haiku first, escalate to Sonnet, then Opus.” It sounds efficient. But Haiku’s bad fixes looked like successes, nothing rejected them, and the cascade never got a chance to escalate. It inherited every silent error from the cheapest model.
- Don’t treat green monitors as approval. Verify a fix against something the broken data doesn’t contain: the raw source, a settlement file, or a before-and-after comparison with a trusted build.
- Invest in change notes. They decided whether the small models could cope at all. A one-line note from the source team saved more than any model upgrade.
That fourth point is what I’ve been building toward since. I turned that verification step into an open-source tool, pipeverify, that compares a table before and after a change, and against its raw source, and gives a PASS, REVIEW or BLOCK verdict in CI.
Caveats
This is one synthetic pipeline, 64 cases, one model family, and one run per case. A difference of one or two incidents isn’t meaningful. The 0.65 threshold was tuned on the same data. Real incidents are messier than my evidence bundles. Treat this as a well-instrumented experiment, not a leaderboard.
Everything is public: the incidents, prompts, all 246 model responses and the scoring code.
- Benchmark and paper data: doi.org/10.5281/zenodo.23510784
- Code: github.com/akshaysharma1088/pipemend
- pipeverify: github.com/akshaysharma1088/pipeverify
So, can you hand the 2 a.m. page to an agent? Partly, yes. Pick a sensible model, let it say “I’m not sure,” and check its work against something it can’t fool. Just don’t let a green job convince you the data is fine. That’s the lesson I learned the hard way, years before any of these models existed.
If you’ve put an agent on a real pipeline, I’d love to hear what broke. That’s the data this benchmark
doesn’t have yet.
