Some of the worst bugs in my CLI came from my own fixes

A bug report lands: one file with a very long expression crashes the whole scan, with no file name and no summary. You add a guard, write a test with a file that crashes, watch it pass, close the ticket.

How do you find out that the guard has a hole? Not from the test. The test is the exact file from the report, and the guard was shaped around it.

I ran into this with a CLI that reads a codebase and lists the analytics events it can send, and the small SDK next to it that sends them. Both are on npm, both had a few hundred tests, and I was fairly sure they were in decent shape. So I asked for a different kind of check: three AI reviewers who had never seen my code or my reasoning. Each started from the README and the published packages, installed them into empty folders the way a stranger would, and tried to make them misbehave. I told them to distrust my tests and my changelog, and to run every finding at least twice before reporting it. Three of them for a whole afternoon was the only version of this I could afford.

I did it three times. After each round I fixed what they found and sent a fresh set at the new build.

Round one: the obvious holes

The first round found what you'd expect from a tool that grew by examples. A script on Node 18 threw on every track() because that version has no global crypto. An SDK flush() returned instantly when another flush was already running, so a serverless function could end before anything was delivered. An object I'd passed in, then changed later, could make a background send crash the process. A broken value in the browser's saved queue made the constructor throw on every page load. On the CLI side, a template-heavy Astro page killed the whole scan, and with no plugin installed a run quietly overwrote a catalog of 68 events with 2.

Boring, real, fixed in an evening. The count of findings the reviewers rated high was 6.

Round two: my fix was the bug

The second round is the reason for this post. Two of the three high findings were things I had added in round one.

The exit that wouldn't happen. A Node script that ends with events still queued should send them and go away. Node fires beforeExit every time the event loop drains, and a script can track more events after the first drain, so I made the SDK listen again each time. It felt careful. Then a reviewer pointed an SDK at a server that always answers 500 and let the script end normally. With a one-second shutdown budget, the process stayed alive for 44 seconds. With the default ten seconds, 67. Every time the loop drained, the SDK tried again, failed again, and drained again. My careful change turned "give up quietly at exit" into "retry until the circuit breaker opens".

The fix was one condition: listen again only after a flush that delivered everything. The next round found a second route into the same problem, a timer that started its own flush behind the exit flush and ran without the exit limit. I had fixed the symptom I was shown and not the property I cared about, which is that exit is bounded.

The stack overflow I "fixed". The original crash was a stack overflow in my own code walking a file with a very long "a" + "a" + ... chain. I rewrote the walk to use an explicit stack, added a guard that skips files nested too deeply, and exempted left-leaning binary chains from the depth count, because the TypeScript compiler handles + chains in a loop and I'd tested 12,000 terms fine. A reviewer wrote the same shape with a different operator:

const x = a && a && a && a /* ... 800 operands */; 

Whole scan dies, no file named. TypeScript's binder recurses on logical operators. My exemption said "binary chains are flat", and that was true for the one operator I had tried.

The honest version of that lesson is small. I generalized from one case, and the test I wrote for the guard checked only the case that had already crashed.

High-severity findings in round two: 3.

Round three: I had assumed something about a tool I don't own

By the third round the reviewers rated nothing high (the top rating was one "medium-high", the && chain above), and several of the interesting findings were again mine. One is worth telling because it's about a dependency.

The Astro plugin hands your page to Astro's own compiler, which is a Go program compiled to WebAssembly. On big pages it runs out of memory and dies hard, taking whatever else was running with it. After round one I'd added a check that looks at a page's markup first and refuses to feed it the pages likely to kill it. In that check I counted a missing end tag the way HTML does. In HTML, <li>a<li>b is two sibling items, because the end tag of li is optional.

A reviewer built a page of nested lists written that way and it ate 4.2 GB and four and a half seconds before the compiler gave up. My check had waved it through.

I stopped guessing and ran the compiler on small versions of the same markup, printing the depth of the tree it returned. 600 unclosed <li> in a row came back as a tree 602 levels deep. The compiler doesn't close them. It nests each item inside the previous one. 150 of the nested-list blocks came back 753 levels deep. My assumption about the spec was correct about HTML and wrong about the tool I was calling, and I had even written it as a comment.

Fixing it meant counting an unclosed tag as nesting, which makes the page look deeper than a browser would see it, which is what the compiler actually does. The reviewer's page is now refused up front, and the file is read without its template and reported as read only in part. The shape of that page is a test now.

The numbers

I counted the headings in the three reports, each of which was written by the reviewers and ranked by them.

The count fell slowly and the severity fell fast. The share of findings that were my own fixes going wrong is the number I find more interesting: in round 2 more than a quarter of what they reported was code I had written a day earlier. The CLI now has 1,359 unit tests and the SDK 245, roughly 200 and 75 more than before the first round. That is not why they got better. What changed things was that each round added a reader who didn't share my picture of what was normal.

Two cautions about those numbers. The ratings are the reviewers' and a model-written rating is a rough signal at best. And "labelled as a bug in a fix" is the reviewer's guess from reading my changelog and the behavior; some of those labels may be wrong.

What the reviewers got wrong, and what they missed

I fixed nearly everything on the three lists, including the low ones. A few I handled differently. One finding was a real behavior I kept on purpose: in the browser, with several tabs open, each tab sends its own events, and I left that as it is for now. A handful of limits I wrote into the README instead of fixing, because the fix wasn't worth the risk, such as a browser's total cap on keepalive requests. Two third-round SDK items were marked informational by the reviewer and were not defects (a Node warning when more than ten instances are alive, a lone surrogate in a string property), and one CLI item the reviewer couldn't confirm without a real server, so I treated it as a guess.

What they missed matters more. Each round found things the earlier rounds had walked past: the second labelled 11 of its findings as old bugs the first round missed, and the third labelled 6. Two examples from the second round. A scan reported itself as complete to the server even when it had skipped an oversized file or a file with a syntax error, which could make events in that file look removed. And two symbolic links pointing at their own folder made a scan run forever. None of the first-round reviewers touched either. So three rounds is not "everything found". It is three samples from a process that kept turning things up, and the fall in severity is a better sign than the fall in count.

What I'd do again, and what I'd skip

I'd keep the rule of pointing the next review at the fix, not at the original bug. A fix is new code written under pressure, with one example in mind, and nobody has read it yet. Mine produced bugs at a rate no worse than my first drafts.

I'd write a guard's test from the other side. A limit like "refuse anything nested more than N" gets tested with a file below N that works and a file above N that is refused, and nobody tests the file that crashes you at 0.8 N because nobody has met it yet. Now when I add a guard I try a few different shapes at about 60 percent of the limit before I trust it.

And I'd measure any third-party behavior I rely on, even when I'm sure. Two minutes with a probe script beat a day of reasoning from a spec that the tool didn't follow.

What I'd skip is trusting my own count of tests as a quality signal. Every one of these bugs lived in a codebase with a green suite, because every test came from the same head as the code.

The CLI is on npm and the changelog lists what changed in this release. If it misses something in your code, or crashes on a file shaped in a way I haven't thought of, I would like to see that file.