Postman approved its own hook. Newman approves its own collection.

On October 25 a developer opened Postman on a Mac and found their coding agents had been changed. A Postman plugin had been installed into Codex and Claude Code; a session-start hook had been registered; and inside their own ~/.codex/config.toml a trust record had been written for that hook, a trusted_hash under a hooks.state key naming the hook file the installer had just put there.

That report is postmanlabs/postman-app-support#14370. When I read it, it was still open, with 315 thumbs-up and 51 comments, on desktop build 12.33.4.

I cannot measure that. I have no Postman desktop install here, and I am not going to build a claim about someone's app out of a screenshot. What held my attention was one sentence the reporter wrote about his own evidence: the source shows the approval attempt, but the trust record alone does not prove who created it, so he does not present it as proof of a particular interaction.

That sentence is the whole post in miniature. A record that says "trusted" is not a measurement of "I chose this." The record is real. The claim it appears to support is one step larger than the record.

So I went looking for the same missing measurement in the tool that runs API tests, where I can run everything myself.

What it prints when it runs one request out of three

A collection with three requests. The first sends and asserts. The other two assert 200 against an endpoint that returns 500, so they would fail loudly if they ran. The first request's prerequest script calls postman.setNextRequest(null).

newman 6.2.3, exit code 0:

│ iterations │ 1 executed │ 0 failed │ │ requests │ 1 executed │ 0 failed │ │ assertions │ 2 executed │ 0 failed │ 

Every number there is true. One request was executed, two assertions ran, nothing failed. The collection has three requests.

Nothing in that table has a denominator. It says 1 next to the word requests, and you have to already know the collection had three to notice that this run did two-thirds less work than the run that passes in the good week. The table cannot tell the two apart, and neither can the exit code, and neither can your CI badge.

The other ways it stays green

Everything below is newman 6.2.3 on one Windows machine, against a local Python server that returns 200 on one path and 500 on another.

The control matters as much as the others. Point a failing assertion at a real 500 and newman does exit 1 and does print the failure. The instrument is not broken. It answers the question it was asked.

The empty collection is the one that should worry you. Exit code 0, total run duration 3 ms, 0 B received, and a summary table full of zeros that looks exactly like a clean sweep. An empty collection is what you get when a config file is missing, when a path resolves to nothing, when the loop that builds the items takes a branch you did not expect. The runner reports that as success, and it is not lying: zero failures were found among zero assertions.

Three green checks that are not green over your suite

I expected more of them to be red than were. Two arms I built to be quiet stayed red, which is worth saying before the other way round:

• A pm.test wrapped in try/catch still fails. I assumed swallowing the exception would swallow the assertion. It does not: Assertions 1 executed, 1 failed, exit 1. Whatever newman does internally, it records the assertion before the throw leaves your script, so a try/catch around pm.test is not a way to lose a failure. I was wrong, and the arm cost me ten minutes.
• An assertion reading pm.iterationData.get('id') with no data file fails. Exit 1, expected undefined to be a string. Also wrong in my favour.

So the holes are not "assertions can be lost". They are all about the size of the set:

1. Requests that were skipped by a control-flow call are not distinguishable from requests that passed.
2. A collection with no assertions passes, correctly, having asserted nothing.
3. A collection with no requests passes, in 3 milliseconds.

One line that turns two of those red

newman's JSON reporter writes everything it did to a file, so the comparison I want is available:

$ npx newman run collection.json --reporters json --reporter-json-export report.json $ node -e "const r=require('./report.json'),c=require('./collection.json');\ const n=i=>i.reduce((a,x)=>a+(x.item?n(x.item):1),0);\ const ran=r.run.executions.length,declared=n(c.item);\ console.log(ran+' of '+declared+' requests executed');\ if(ran!==declared)process.exit(1)" 1 of 3 requests executed $ echo $? 1 

On the passing-looking run above, that prints 1 of 3 requests executed and exits 1. Your CI goes red on the collection that skipped two requests, while newman itself stays green.

Then I ran the same line against the other three collections, and it has a hole of exactly the kind this post is about:

The last row is fine and expected: that line compares counts, it does not look at failures, and newman already answers the failure question with its exit code. Two instruments, two questions.

The third row is not fine. 0 of 0 is the same shape as "the log file was missing, so zero calls were recorded, so the gate passed" — a zero that cannot distinguish nothing happened from nothing was recorded. My check compared a number to a number and was satisfied by both being zero. So the version I would actually put in a pipeline adds one clause:

if (declared === 0 || ran !== declared) process.exit(1); 

With that clause the empty collection exits 1 and the skipped-request collection still exits 1. Two lines, and only the two arms that should pass do.

The three things worth adding to your pipeline

1. Run with --reporters json and assert executions.length against the declared item count. The runner already writes both. Nobody compares them.
2. Refuse a collection with zero assertions. Exit 0 over zero assertions is correct arithmetic and worthless assurance; if a collection is meant to be a regression gate, its assertion count belongs in the gate.
3. Print the denominator next to the green. The one number that made all of this visible was 3, taken from the collection file, not from the run. A summary that cannot print it is a summary of a different question.

What this is not

newman is an open-source CLI runner; the desktop app in that GitHub issue is a different artifact, and none of the above is evidence about it. I have not installed Postman desktop, and I have not tested whether the plugin install, the hook registration or the trust-record write still happen on any build. Those are the reporter's measurements, not mine, and his own caveat about what the trust record proves is the reason I'm borrowing the shape and not the accusation.

I also did not test other runners. Bruno, hurl, and the rest may skip requests in ways that surface differently; on this machine I measured newman 6.2.3 and nothing else, so treat the table as one runner on one day, not a survey.

What I think does generalise is smaller than any of this. Both artifacts — a test summary and a trust record — are truthful about what they contain, and both get read as an answer to a question one size larger. In one case the question is "did my tests pass", and the answer came from whichever requests happened to run. In the other the question is "did I agree to this", and the answer came from a row that says the hook is trusted.

Neither artifact was lying. Both were counted over a set that nobody named out loud. If you take one thing from this, take the habit: when something reports success, find the number it counted, and then find the number it should have counted. They are almost never the same one, and the second one is usually sitting in a file the first one already wrote.