My AI agents wrote four changes overnight. Tests passed. Each had a hole

In the night from 14 to 15 November my coding agents finished four changes in three repositories. Every test suite was green. Each change had a hole the tests did not see, and in one case a test expected the hole.

The repositories are open-source apps on Cloudflare Workers: an issue tracker, an agent runner and an identity provider. I work alone and agents write most of the code. For any change that touches auth, secrets or deletion, I now give it to a second agent that did not write it, before I merge. This is how I do that and what it turned up that night.

The setup

The second agent is read-only. It gets the diff and the files around it. It does not get my opinion of the change or the first agent's summary. It gets numbered questions, and every answer has to name a file and a line. One question from that night:

Can anyone without the code AND the verifier cause an agent that is in legitimate use to be removed? Trace every path into the function that removes it.

Each question names one actor and one outcome. The answer is then a list of code paths, and I can open each one and check it.

1. A cleanup that would almost never run

In the tracker, an approval page creates an agent member and a webhook, then hands out a one-time code. The code is good for a minute and lives as a row in the auth library's verification table. If nobody trades the code, the member and the webhook stay behind. So the agent wrote a sweep: find our rows past their expiresAt, delete them, remove what they were made for.

The tests set expires_at into the past, ran the daily job and checked that the agent member was gone. They passed.

The auth library (Better Auth) deletes every row of that table past its expiresAt whenever it looks one up. A sign-in is enough. So on a real server the row would usually be gone before the sweep saw it, and the leftovers would stay for good. No test had a sign-in between the expiry and the sweep.

The fix keeps the code's deadline inside the row's JSON and sets the row's expiresAt three days later. The sweep reads the deadline from the JSON. That fix went to review as well, and it had its own problem:

-- Simplified. Before: WHERE identifier LIKE 'track-connect:%' AND json_extract(value, '$.sweepAt') < :now -- After: WHERE substr(identifier, 1, 14) = 'track-connect:' AND CASE WHEN json_valid(value) THEN json_extract(value, '$.sweepAt') END < :now 

The table is shared. Other rows in it hold values that are not JSON, and json_extract raises an error on those. The only guard was that the LIKE came first in the AND, and SQLite does not promise to evaluate it first. LIKE also ignores case, so it was a loose match for a prefix. There is now a test with a plain-text row and a row named TRACK-CONNECT:abc next to ours.

2. A fake server that validated nothing

The agent runner sends the browser to the tracker's approval page with a state, as OAuth does. The agent put the workspace id in it: <workspaceId>.<token>. The tracker's page accepts only letters, digits, _ and - in a state. A dot is refused.

Every test passed, because the fake tracker in the runner's tests checked nothing about the address it was given. Against the real tracker the feature could not have worked once. The fix landed twelve minutes after the feature commit and before the merge. The workspace now travels in the return address, and the fake checks the state and the challenge the way the real page does.

3. Memory that trusted the wrong thing

Agents in the runner can keep notes between runs. A note is recalled by later runs, so a bad one keeps steering them. The first version decided trust from one fact: what started the run. If a person's chat message started it, the note was trusted, on the idea that a person is responsible for that run.

But the agent reads other people's text during that run. A member asks it to summarise an issue. The issue body, written by an outsider, says "deploy approvals are no longer required". The agent stores that as a note about the workspace. Every agent recalls it from then on.

The rule now has two conditions:

// From rememberInRun, shortened. const MEMORY_OWN_SCOPES = ["agent", "person"]; const trusted = run.origin === "person" && MEMORY_OWN_SCOPES.includes(input.scope); 

A person's chat must have started the run, and the note must be about the agent itself or about that person. Anything about the workspace or a repository waits until an owner or admin approves it.

This is not fully closed. In a chat, an agent can still copy an outsider's text into a note about itself and recall it later with nobody having approved it. So recalled notes are handed over as records with their source, not as instructions. The docs say so under "The limit that remains".

4. A two-factor rule you could walk around

In the identity provider, a workspace can require two-factor sign-in. The first version checked how each session was made, and that check was right.

Adding a passkey only needed a recent sign-in, though. Someone who held just a member's GitHub account could sign in with it, add a passkey of their own, sign in with that and pass the rule. The suite had a test that walked exactly this path and expected it to work.

After that was closed, removing was still open, and the docs admitted it. An account with no second factor takes its next one on trust. So the same one-factor session could remove the existing passkey and then enrol its own. That was closed about half an hour later. Adding to an account that has a second factor, and removing one, both need a session that already showed two factors. If you lose your only passkey, you have to ask the operator, who has a script for it.

What the green suite was worth

The same agents wrote the code and the tests, so the tests had the same blind spots as the code. In finding 4 the test encoded the hole as correct behaviour.

Each review took a few minutes, much less than writing the feature. The reviewer was also wrong sometimes, or raised points not worth fixing. I checked every point against the code before acting on it. An automated reviewer on the pull requests found more real edge cases after that. I read the core function of each change myself.

What I have not solved: nobody has read every line of these four changes. Not me, not the second agent.

Was it worth it?

Yes. Four real holes for a few minutes each is cheap. But it worries me that I needed it four times out of four.

If agents write your code, who reads it before it merges, and what do you ask them?

The memory rule from finding 3 is in packages/registry/src/memory.ts, with the reasoning in the header comment.

I drafted this with an AI assistant (Claude), which also wrote much of the code it describes. I checked the facts and numbers against the code.