Who Checks the Tests When an Agent Rewrites the Code?

When code and tests change together, a green build is not enough.

Imagine a coding agent optimizing a billing endpoint. The tests pass, yet a missing tenant check could let one customer read another company's invoice. What if the agent can also edit the tests used to approve its work? In Design for Churn, Ben Halpern argues that cheap rewrites make durable interfaces, schemas, and behavioral tests more important. I agree. Agents should improve development tests, but final security checks must remain outside that agent's control.

Green tests can mislead in two ways. The first is omission: the suite never checks an important requirement. UTBoost (ACL 2025) identified 36 SWE-bench tasks with insufficient tests, allowing incorrect patches to pass. The second is manipulation: an agent exploits the tests themselves. ImpossibleBench (ICLR 2026) deliberately made tests contradict specifications. Some agents modified tests despite instructions not to; others used shortcuts without editing them. Read-only tests stopped direct edits but not every shortcut. Stronger prompts helped in some settings, but instructions cannot enforce permissions. These are benchmark findings, not production failure rates.

Three practices make a green build more meaningful.

Keep acceptance independent. Agents can update development tests. Critical security checks should come from a protected baseline and run in separately controlled CI with limited privileges. Changes to those checks or the CI definition require independent review. An AGENTS.md file can guide agents, but it cannot enforce permissions. Keeping some security tests hidden from the agent can help, although it also reduces useful feedback.

Test the tests. In a throwaway copy, remove one authorization check at a time and rerun the security suite. If nothing fails, investigate missing coverage. Security-focused mutation testing exposes gaps; it cannot prove a system is secure.

Enforce critical boundaries separately. Database row-level security or a separately enforced authorization policy can limit damage if an application check disappears. That layer also needs restricted privileges and independent tests.

A small, deterministic Python demo shows both the missed check and how mutation testing exposes it. No LLM is involved. The original invoice lookup checks the caller's tenant. A deliberately flawed rewrite replaces a scan with a dictionary lookup but drops the tenant condition. The three existing tests cover only one tenant, so they still pass. Add a negative test that Alice at Acme cannot read a Globex invoice, and the rewrite passes only three of four. A separate mutation removes only the tenant condition from the original function: the basic suite misses it, while the security suite catches it. Try it yourself: github.com/xiaoleiren/green-tests-broken-isolation.

Frequent rewrites are not the problem. Before merging, ask who controls the final checks and whether they would turn red if a security boundary disappeared.

Xiaolei Ren works on software verification and the security of AI-driven systems.