My AI agent's 34 tests passed. Then I broke the code 31 times.

"All tests passing" is the most common line in an AI agent's final summary. I wanted to know what it is worth.

Disclosure first: this week I'm working with the team behind Lemon Code, a free coding agent. That's exactly why I didn't want to stop at "it passed".

The build

I asked it for a weather and travel planner: search a city, show the 7-day forecast, pick trip dates, rank the best days to go, and suggest a packing list. Plain Vite and vanilla JS, live data from Open-Meteo.

About 3 minutes later I had a working preview. On the way, 3 of its tests failed on the first run. It read the errors, fixed them, and reran. The final summary said 34 tests passing.

Check 1: run the tests outside the agent

I downloaded the zip and ran it on my own laptop.

 ✓ test/storage.test.js (8 tests) ✓ test/weather.test.js (11 tests) ✓ test/planner.test.js (15 tests) Tests 34 passed (34)

Good, but that only proves the tests pass. It says nothing about whether they would fail.

Check 2: delete each function and see who notices

There's a good post here on dev.to, The Test That Survives Its Own Implementation, about agent tests that keep passing after you delete the code they cover. So I tried it.

For each of the 21 exported functions in the logic files, I added return undefined; as the first line and reran the suite.

for name, body_start in exported_functions(file): write(file, original[:body_start] + "\n return undefined;" + original[body_start:]) failed = run("npx vitest run --reporter=json")["numFailedTests"] write(file, original)

Result: 21 out of 21 caught. Every gutted function broke at least one test, from 1 failure (geocodeUrl) up to 8 (parseISODate). No dead weight.

Check 3: small, realistic bugs

Deleting a function is a loud bug. Real bugs are quiet. So I made 10 one-line changes to the trip planner:

Change	Tests that failed
Off-by-one in the day count (drop `+ 1`)	3
Date filter drops the last day (`<=` to `<`)	2
Best days sorted in reverse	2
"End date before start date" check removed	1
Rain penalty halved	0
Wind threshold 20 to 30 km/h	0
Sunny-day bonus removed	0
Snow penalty removed	0
"Too hot" threshold 30 to 35°C	0
Tie-break between equal days reversed	0

4 caught, 6 survived.

The pattern is clear. Dates, ranges and sort order are well tested. The day-scoring rules are not. scoreDay has one test, and it only checks that a clear, calm day scores higher than a storm. Any weight you tweak still passes that.

Also worth saying: the UI code (ui.js and main.js, about 480 lines) has no unit tests at all. The agent checked the UI by driving the app in a browser instead, searching two cities and saving a trip. That's a reasonable choice for a small app, but it's not something npm test will protect.

What I'd take from this

"34 passing" was true, and the tests weren't fake. They caught every deleted function and the date bugs I'd worry about most.
The gaps are where the logic is fuzzy. Scoring rules with magic numbers got one sanity test. That's the kind of code where I'd ask the agent for more: "write a test that fails if the rain penalty changes."
The check is cheap. Breaking 31 things and rerunning took a couple of minutes with a small script. I'd do this on any agent-written suite before trusting it.

If you want to try the same thing, Lemon Code is free and doesn't need an API key: lemoncode.ai. Build something, download it, and break it. I'd like to hear what your tests let through.