The worst thing a QA tool can do is tell you the app is fine

There's a whole category of AI dev tools whose output is a claim about reality: this test passes, this patch fixes it, this app is healthy. And there's one specific way that category dies.

It doesn't die by being wrong sometimes. Everything is wrong sometimes. It dies when being wrong looks the same as being right: when the green check that means "verified" and the green check that means "couldn't tell" render identically. The moment a team learns that, they stop reading the output. A tool nobody reads is worse than no tool, because you're paying for it and you stopped doing the thing it replaced.

I've been building ProdCritic, a QA agent for backends. You point it at a Node.js, TypeScript or Python app and it:

1. boots the app on a port it checks is free,
2. finds the routes, both from the links the app serves and from the routes its source declares,
3. probes them and reproduces the crash,
4. writes a fix that has to parse before it touches the file,
5. restarts the app and re-probes the route to prove the fix works,
6. restores your original code and hands you a .patch.

That loop isn't the interesting part; plenty of tools generate patches now. The interesting part is what the tool does when the loop doesn't work. Here are five places that shows up, all with real output from the current build (trimmed to the summary block it prints at the end of a run).

1. "Probed 1 of 1 routes" is not coverage

Early on, ProdCritic found routes the way most crawlers do: from href and action attributes in the HTML the app serves. Then I pointed it at a plain JSON API. The root returns JSON, which links nothing, so it found exactly one route, /, probed it, got a 200, and said:

healthy, probed 1 of 1 routes

Meanwhile GET /api/orders/summary threw Cannot read properties of undefined (reading 'map') on every request.

The ratio was honest. The denominator wasn't.

Now it also reads the routes the source declares: Express routers and the prefixes they're mounted under, Flask and FastAPI decorators, Next.js file routes, and req.url === '/x' comparisons in hand-written node:http servers. It fires concrete GETs only: a POST would need an invented body, a :id would need a guessed value, and a route named like /cache/reset is never fired at your data. Every route it didn't fire is named, with the reason. Here's the same API today:

 VERDICT FIXED cause A value was undefined/null when '.map' was accessed. crashed /api/orders/summary scope probed 6 of 10 discovered route(s) found 0 served link(s) · 10 source declaration(s) in routes/legacy.js, routes/reports.js, server.js unprobed POST /api/orders — the probe sends GET only /api/orders/:id — needs a parameter no link or declaration supplied /status — declared in source but answered 404/405, so no handler ran /api/cache/reset — its name says it changes data, so it was never fired fix verified against the running app assumes a missing 'o.lines' is treated as an empty list — check that this is the right behaviour answers status only — no .prodcritic/expectations.json, so a wrong answer at HTTP 200 would pass 

And the patch it handed back:

 app.get('/api/orders/summary', (req, res) => - res.json({ skus: orders.flatMap((o) => o.lines.map((l) => l.sku)) }) + res.json({ skus: orders.flatMap((o) => (o.lines || []).map((l) => l.sku)) }) ); 

Notice the assumes line. The restart proves the crash is gone. It can't prove that an empty list is the behaviour you wanted, so the report says so.

2. A reviewer liking a patch is not evidence the patch works

The pipeline has an adversarial review step: a model reads each candidate patch and tries to tear it apart. It would be easy, and it's the natural shape of the code, to let "the reviewer approved it" mean "fixed". It reads well.

It's also worthless as evidence. So there are two fields, and merging them counts as a bug:

Only the second is a statement about the world. The GitHub Action can open a draft PR with a fix, the only thing ProdCritic ever writes to your repository, and that is gated on patchVerified, explicitly not on skepticPassed.

3. Fixing one crash out of six is not a pass

Real apps don't have one crash. This is a deliberately broken test app with six:

 VERDICT PARTIAL cause A value was undefined/null when '.trim' was accessed. crashed /calculate, /items, /report, /config, /import, /legacy STILL DOWN /items, /report, /config, /import, /legacy scope probed 8 of 8 discovered route(s) fix verified against the running app assumes a request without 'input' should be answered 400 "Missing required parameter: input" rather than processed — check that this is the right behaviour 

It diagnosed a real crash, wrote a patch, restarted the app, and proved /calculate came back. That's a genuine, verified fix, and the app is still down on five routes. A tool optimising for the demo reports that as a success. So the verdict checks the partial case before the clean one:

if (run.patchVerified) { if (parseRouteList(run.remainingCrashes).length > 0) return 'partial'; return 'fixed'; } 

(With --fix-all it keeps going: each verified fix is applied to a throwaway copy and the next crash is worked, until nothing crashes or a fix doesn't hold. You get one patch with every verified fix in it.)

4. Exit 0 never means "could not tell"

Point it at a folder that isn't a runnable app:

 VERDICT NO-RUN cause No package.json start/dev script and no entry server file (server.js/app.js/index.js) found — ProdCritic cannot tell how to start this app. fix not attempted — the app never booted 

That run exits 1. Nothing crashed, but nothing was probed either, and "I found no crashes" is a very different sentence when you never looked. In CI the exit code is the whole interface:

A tool that exits 0 when it couldn't do its job teaches a pipeline to ignore it, which is the same failure as lying, just slower.

5. The blind spot I keep on purpose

There's a test app in the repo called silent-defect. Every route answers 200, and /tax?amount=100 returns {"amount":100,"rate":0.08,"total":8}. The total should be 108.

 VERDICT HEALTHY scope probed 5 of 5 discovered route(s) fix not needed — nothing crashed answers status only — no .prodcritic/expectations.json, so a wrong answer at HTTP 200 would pass 

ProdCritic reports it healthy, because a status-code probe can't tell a right answer from a wrong one. I kept the fixture so the blind spot is measured instead of hidden, and every clean report says "status only" out loud.

The way past it never involves the tool inventing what "right" means, because a check nobody wrote, failing, is the tool making up a bug. Instead the owner declares it, in .prodcritic/expectations.json:

{ "route": "/tax", "query": { "amount": 100 }, "status": 200, "json": [{ "path": "total", "equals": 108 }] } 

The same app, with that declaration:

 VERDICT EXPECTATION-FAILED cause No crash, but a declared answer did not hold — /tax?amount=100: total equals: expected 108, got 8. fix none proposed — ProdCritic does not patch a wrong answer (no crash to reproduce) answers 4 declared: 3 held, 1 broken · status only on 1 probed route(s): / 

You can also declare a relation instead of a value ("equalsExpr": "amount * (1 + rate)"), which holds for every input, or let your OpenAPI document act as the contract. And if you'd rather not write any by hand, it can draft them from what your app actually answered, for you to correct.

Why this is more than a README

Rules written in a README are decoration. These are enforced by 211 spec files across two test suites, and breaking one fails the build. Among them:

• a test whose whole job is to fail if the published report ever merges skepticPassed and patchVerified into one sentence;
• tests that pin the draft-PR refusals (off by default, never the default branch, verified fixes only) before the happy path;
• tests that pin no-run and partial to non-zero exits;
• a migration test that applies a schema change to a populated database and checks the incident history survived.

What it doesn't do

A launch post that only lists strengths would be doing the thing this project is against, so:

• It never rewrites your repo. It patches to verify, restores your source, and hands you a .patch.
• It doesn't check routes it can't reach on its own: ones that need a request body, a login it wasn't given, or an ID it couldn't find. Each one is named in the report.
• It doesn't know whether an answer is right unless you declare it or have an OpenAPI document.
• It doesn't patch mobile apps. On Android and iOS it drives a real device or simulator and captures real crashes, but there's no source to patch, and the report says so.
• It's proven on my test apps, not yet on many real ones. That's the honest risk, and it's why I'm asking for help below.

I'd like to run it on your API

ProdCritic isn't public yet. If you maintain a public Express, Fastify or Flask API, drop the repo link in the comments and I'll reply with what it finds: which routes it reached, which it couldn't and why, and any crash with a verified fix. I'm especially interested in the ones where it's wrong.