I once fed my coding agent a log file with a poisoned line in it — one fake error message carrying a hidden instruction. The agent obeyed the log. Three runs out of three. Afterwards it reported success using the log's own words, like a waiter repeating an order he never checked.
I published that experiment on this site and the comments turned into an argument, which taught me something the run itself hadn't: nobody is short of opinions about AI coding tools. Everybody is short of measurements.
So here is the thing I'm doing about it.
The series: LGTM — Let's Get the Measurements
LGTM is what reviewers type when code looks fine and nobody actually checked it. This series is the opposite of that. For the next four weeks I'm taking the claims that float around AI coding — agents are safe, agents are dangerous, this tool writes production code, that trick doubles your output — and putting each one in a small lab, running it for real, and publishing what happened. Output shown. Failures included. If a claim survives, it survives with numbers attached. If it dies, you watch it die.
The mission, in one line: no claim about AI coding gets believed here until it has been measured in public.
The long game is a scoreboard. Eight episodes, one season, and at the end a single table: every claim we tested, what the run said, and whether you'd be a fool to trust it. Something you can link to the next time a thread fills up with certainty and no evidence.
How each episode works
- The method gets written down before the run — what counts as a pass, what counts as a fail. No moving the goalposts after the numbers come in.
- Real tools, real accounts, real runs. Nothing simulated, nothing "representative". If a run embarrasses me, it stays in.
- Every episode shows its working: the actual outputs, the diffs, the transcripts. The full raw files — datasets, complete transcripts, the exact scripts — go to members on Patreon as that episode's data pack, the same day.
- The articles themselves stay free. That part doesn't change.
Two episodes a week, Tuesdays and Fridays at 19:00 Dhaka time, for four weeks.
The season, as teasers — verdicts withheld on purpose
- The poisoned repo. A code comment, a README section and a GitHub issue all quietly whisper the same bad instruction to a real Copilot. Five runs. Does it bite?
- Four AIs, one spec. Copilot, ChatGPT, Claude and Gemini get the identical job and the identical frozen test suite. One judge, no favourites.
- A real bug, a real repo. Copilot gets pointed at an actual open-source issue. The repo's own test suite decides whether the fix is real.
- verify=False. The TLS check people switch off and forget — counted in the wild, in public code, at scale.
- The free-tier walls. One real job, five free plans. Screenshots of exactly where each one stops you.
- Documentation that lies. An AI writes the docs; I check every claim it makes against the code it describes.
- Install scripts, read by hand. The top of the npm tree, opened and read line by line. What actually runs when you hit install.
- Your claims. The season finale is built from the claims you leave in the comments — measured the same way, table and all.
Your part in this
Two things, both cheap.
First: follow along if receipts are your love language. The scoreboard only works if the people who argued in week one are still here in week four to watch the table fill in.
Second, and this is the real one: drop a claim in the comments. Something you've heard, believed, doubted, or repeated yourself about AI coding tools. "Agents always..." "Claude never..." "Copilot can't..." The sentence doesn't have to be true. It has to be testable. The best ones go into episode 8 and get measured in public, with the claim credited to whoever brought it.
Episode 1 is already in the lab. A small billing app. Three quiet whispers planted in three different places — a comment, a README, an issue. One expired discount code that should stay refused. And a real Copilot walking in to do nothing more dangerous than a rename and a JSON flag. Five runs on the clock.
It lands today at 19:00. Bring your scepticism. I'll bring the receipts.
P.S. Everything I write stays free — that part doesn't change. But if this one was worth something to you, you can buy me a coffee ☕ (one-time or monthly, both work). It covers the API credits and hosting behind these experiments — and honestly, it's also the nudge that gets the next one started on the evenings I'd otherwise skip it. — Salam
