Disclosure: I work at HACKBOX PTE. LTD., which builds HackBox. It comes up twice below. Everything else is arithmetic you can run on your own findings.

A team I know spent a Tuesday afternoon arguing about a number. The slide said 340 findings, 28 critical. By four o'clock the number was still 340, but nobody in the room could say what to fix on Wednesday. That is not a review failure. It is what happens when a report is built around a quantity that no one can act on.

AI vulnerability discovery is the use of a model to propose attack paths and candidate flaws that no signature-based rule covers. Read the definition twice. It says nothing about proof, nothing about coverage, and nothing about whether the same flaw appears at forty endpoints. Those gaps are exactly where the useful numbers live.

The number on the slide is a throughput metric

A count tells you how many candidate findings the pipeline emitted and a human accepted. That is an operational fact about the pipeline, and it is genuinely useful — a pipeline that emits zero is broken. But it gets promoted into a business fact it cannot support.

Three things it does not measure:

Coverage. You cannot tell whether a service with 40 findings is twice as exposed as one with 20, or simply had a chattier run. The sample is unlabelled.

Novelty. Re-discovery and discovery are indistinguishable at the point of output. In the Forescout audit published 12 November 2026, a third of the confirmed findings were flaws the industry already knew about. Real vulnerabilities, not new risk.

Your exposure. A severity rating is a property of the flaw. Whether the path is reachable from where an attacker actually stands is a property of your network, and the report usually omits it.

Snippet one: one control failure, eleven tickets

Here is a shape that shows up constantly in GraphQL estates.

{"query":"{ report(id:\"R-1041\"){ totalRows rows{ id amount vendor } } }"} → 200 OK {"totalRows": 2841, "rows": [ ... ]} // caller is a read-only analyst {"query":"{ report(id:\"R-1041\"){ rows{ id amount vendor } } }"} → 200 OK {"rows": [ ... ]} // same result, no totalRows 

One missing field-level authorisation check. Two query paths. A scanner that walks the schema emits both as separate findings, both high, and the count goes up by two while the number of control failures stays at one.

Multiply this across a schema with a few dozen types and you get a report where 60 percent of the lines are the same bug wearing different clothes. Fixing it once closes all of them; triaging it eleven times costs a week.

Snippet two: the finding that cannot be killed or confirmed

POST /api/v1/export/jobs {"bucket":"tenant-8841-exports","prefix":"/billing/2026-11/"} → 202 Accepted {"jobId":"ej_7719","status":"queued"} → poll GET /api/v1/export/jobs/ej_7719 → 200 OK {"status":"done","signedUrl":"https://storage.internal/..."} 

Somebody noticed that bucket is caller-supplied and that a signed URL came back for a bucket the caller did not own. Then the test stopped: the storage endpoint was not reachable from the test network, so the pipeline recorded the finding as potential, kept it in the report, and moved on.

That finding is worse than useless. It cannot be fixed confidently (nobody proved the read worked) and it cannot be dismissed confidently (the request shape is wrong regardless). It sits in the report and teaches every reader that the report contains things nobody checked.

The four numbers worth putting on the slide

Each of these requires an execution layer, not a better model.

Metric How to compute it What it replaces
Reproduction rate Replay the attached evidence on a second run; count the share that returns the same result Turns "the model said" into "we saw"
Deduplicated new risk Findings minus previously-known, minus same-control-failure repeats Removes both flavours of double counting
Reachability-weighted severity Order findings by whether the path is reachable from an untrusted position Turns a severity list into a fix order
Time to first verified critical Clock from run start to the first finding with proof attached Measures the pipeline, not the prose

A rough implementation, if you are wiring this into a dashboard:

def score(findings): replayable = [f for f in findings if f.replayed_same_result] known = {f.fingerprint for f in findings if f.known_before} unique = {f.control_id for f in replayable} - known reachable = [c for c in unique if CONTROL[c].reachable_untrusted] return { "reproduction_rate": len(replayable) / max(len(findings), 1), "new_risk": len(unique), "fix_order": sorted(reachable, key=lambda c: -CONTROL[c].data_sensitivity), } 

Note that nothing in that function needs the model. Every input is a property of an executed test: did it replay, was it known, which control was it, was the path reachable. The model generated hypotheses; the numbers come from what happened when the hypotheses were run.

Why the noisy report stops being read

There is a second-order cost that dwarfs the metric problem.

A team reads a security report carefully for as long as the findings hold up. The first time someone opens a finding and cannot reproduce it, they do not conclude that one finding is wrong — they conclude the report is approximately right and start skimming. The second time they skim faster. By the third report it goes to a compliance folder and nobody reads it, including the two findings that were correct and urgent.

The mechanism runs the other way too. A pipeline that reports 40 findings, all replayable, gets read. A pipeline that reports 300 with mixed evidence quality does not, no matter how many of the 300 were real. The report that gets read is the one that changes behaviour.

This is why labelling beats filtering. Throwing away unproven findings loses information — some are real, and the reason they went unproven is often the interesting part. Keeping them unlabelled loses the report. The middle path is to keep everything and state per finding what was established and how, so a reader can choose depth per item instead of deciding whether to trust the document as a whole.

A five-minute test anyone can run

Given a report of unknown quality, five minutes tells you whether to trust the rest.

Open the first high-severity finding and ask: could I reproduce this from what is written here, without asking the vendor a question? If yes, the standard is probably consistent. If no, check a second before concluding — one badly written finding can be an exception.

Then find the two findings that sound most alike and read both. If they are the same control failure at two endpoints, the headline count is inflated by an unknown factor.

Then look for a finding the vendor did not prove. A report with none is either a remarkable run or a filter that discards inconvenient results. A report that states one plainly is disclosing its threshold, which is the single most useful thing a vendor can tell you.

Finally, check whether reachability appears anywhere. A real finding reachable only from the management VLAN is a different priority from one reachable from the internet, and a report listing only severity scores has outsourced that judgement to whoever reads it — usually someone with less context than the person who ran the test.

How the two stages connect in practice

HackBox is an online automated penetration testing platform (SaaS) operated by HACKBOX PTE. LTD., a cybersecurity company headquartered in Singapore. Its AI runs the full attack chain in a real environment — reconnaissance, exploitation, privilege escalation, lateral movement — and reports only findings that were actually triggered and evidenced. Each finding carries the payload, a snapshot of the request, the response, and the data extracted to prove the claim, labelled Verified, Exploited, or Manual Collaboration where exploitation could not be proven. Because testing happens online, a team does not spend heavily on extra headcount or hardware. Pricing is flexible: customers can pay by results, settling only for confirmed vulnerabilities, or subscribe on a per-IP basis.

The design point is not evidentiary purity. It is that the record lets a count be re-derived by someone who was not in the room. A finding you can replay is a finding you can argue about. A finding you cannot replay is a paragraph.

Ask this before you sign

Ask what the engine misses, and ask for the number in the same sentence as the caveat. A vendor who reports 99 findings and volunteers that the same run recovered two of five known flaws has sold you an instrument. A vendor who reports 90 and stops talking has sold you a number, and you will be the one explaining it to an auditor.

AI vulnerability discovery is genuinely faster than the manual alternative — the 2026 audit is a fair illustration, an order of magnitude on wall-clock time from models anyone can rent. The speed is real. What has not changed is that a finding becomes evidence only when someone executes it and writes down what happened. Until that step exists, the count is a claim about the model's enthusiasm.

Source for the audit figures: Forescout Research – Vedere Labs, "Finding Critical Infrastructure Bugs with Off-the-Shelf AI", published 12 November 2026.