"The tests pass" is how most of us accept a coding agent's work. For security-sensitive code, a new benchmark says that's a weak signal.

On November 4, 2026, eight researchers (Yu et al.) posted A.S.E 2.0 to arXiv. It tests how the repository context an LLM sees changes the security of the code it writes, using 145 tasks built from real CVEs in 51 open-source projects. Each task runs in Docker with functional tests and a validated proof-of-concept exploit, and counts as secure only if the tests pass and the exploit fails. It's a preprint, not yet peer reviewed.

Passing is not safe

Claude Code running Sonnet 4.5 passed the functional tests on 74.0% of tasks, but only 18.4% of its output was secure. 55.6% passed every test while a working exploit still succeeded.

That pattern held across every setup. As a share of test-passing output, 73.2% to 77.3% was exploitable, whether it was Claude Code, SWE-agent on Sonnet 4.5 or GPT-5.4, or plain retrieval with no agent loop.

Agents search for the wrong things

The agents read far more of the repository than any retrieval pipeline, yet saw the relevant safeguard (an existing validator or access policy) in only 27% to 36% of their runs. 71% of their search queries named functional things, such as a file format; only 9% named security concepts.

When the safeguard was in view, results changed:

  • Runs that saw and used the safeguard reached a 34.5% secure rate, against 12.7% for runs that never saw it.
  • Seeing it wasn't enough, though: 41% of runs that observed a safeguard didn't use it.
  • With a reference context of annotated safeguards, secure output was 24.8%, against 12.6% with the target file alone.
  • A generic "write secure code" reminder only moved it to 13.9%.

More context, more copying

Feeding the model more of the repository didn't help. A truncated 128K-token repository raised the functional pass rate from 56.9% to 60.9% and cut the secure rate from 12.6% to 12.1%. Inserting one insecure example raised correct-but-exploitable output by 6.3 points, even with the safeguard present.

The agent loop also bought working code, not safe code. Claude Code cost $1.41 per task against $0.09 for its retrieval setup without the loop, and read 204K tokens against 7.2K, for a secure rate of 18.4% against 17.9%.

What to do with this

If you let an agent touch security-sensitive code:

  • Name the safeguard in the prompt ("use validate_path() from utils/security and check can_edit() before writing"). The agent won't reliably go looking for it.
  • List your validators, auth checks and safe APIs, with paths, in your agent instruction file.
  • Remove or label insecure examples in the codebase the agent reads.

If you review agent pull requests:

  • Treat "tests pass" as a functional result, not a security result.
  • Check that the safeguard is called, and called in the right place. A permission check that runs too late is a real failure mode in the paper.
  • Don't rely on static analysis alone: it disagreed with the exploit oracle on 36.8% of 28,984 functionally correct generations.

Limits: the tasks come from known CVEs, so they're security-sensitive by construction; the paper tested Sonnet 4.5 and GPT-5.4, not newer models; and an exploit can demonstrate a vulnerability but not exclude one.

The full write-up has the per-agent cost table, the cost per secure solution and the context experiments: AI-Generated Code Security Study 2026 on StackChief.

This post was written with the help of AI, as a summary of the StackChief article above.