In part 1 I described the bug that started Lanes: Ana's agent renames priceWithTax to grossPrice, Bea's agent writes a new file that calls priceWithTax, git merges both cleanly, and the build breaks.
Saying "Lanes fixes that" is easy. Showing it is harder, because agents aren't deterministic, the bug depends on timing, and the person running the benchmark (me) wants a particular answer. This post is about how I tried not to fool myself, where that worked, and where it didn't.
Rule one: write the method before running anything
On 1 November I published method.md before the first run. It fixes:
- The arms. C is Lanes leases alone (the beta that already shipped). D is C plus team context. Same model, prompts, budget and starting commit in every arm.
- The metrics, with definitions: task success (the scenario's checks pass on the final tree), lost lines, and "disruption", which counts agent messages that ask the user to decide something, matched by a fixed pattern in the runner.
- What gets reported: every run, including failed and excluded ones, with the reason.
- What would change my plan: if team context is no better than leases alone, it's cost without value, and I shrink it.
Then, before each new round, I added the scenarios and the success bar to that same file, dated, before running them. That turned out to matter more than anything else in the method, as you'll see.
The runner drives real Claude Code sessions (Haiku, about $0.007 per two-agent run at API prices) against a small template repo, then runs the checks. Everything's in the repo, so you can rerun it: node scripts/runner/run.ts I1 I2 I3 --runs 10, then again with --team.
The pilot: everything passed, which proved nothing
The first round used two scenarios from my earlier spikes: four agents on two hot files, and four agents where one renames a function another starts calling. Three runs each, arms C and D.
Both arms passed every task in every run. No lost work, no syntax errors, and the one agent that stopped to ask its user was in C.
It's tempting to call that a win. But if the control passes everything too, the scenario can't tell the arms apart. Here the reason was specific: the new caller was written in the same file the renamer edited, so the lease handed Bea's agent the renamed code before it wrote anything. Leases alone already solved it.
The pilot write-up says exactly that: the only claim it supports is "team context added no disruption and no lost work". Nothing about whether it helps.
Harder scenarios: the caller goes in a new file
On 3 November, before running them, I added three "intent" scenarios where the new caller lives in a file the other agent never touches. Leases have nothing to act on there. The only thing that could help is knowing what a teammate is doing.
| ID | Ana | Bea | Fails when |
|---|---|---|---|
| I1 | renames priceWithTax to grossPrice | 8 s later, adds POST /invoices in a new file calling priceWithTax | Bea's file still imports the old name |
| I2 | renames, 8 s after Bea | adds the endpoint first | the rename misses Bea's file |
| I3 | changes addOrder(total) to addOrder({ net, total }) | 8 s later, adds POST /refunds calling addOrder | Bea calls the old signature |
I checked each scenario's checks by hand against a correct tree and a broken one before any agent ran.
First round, 3 runs per arm: leases alone failed I1 in 3 of 3. Good, the bug was real. But team context didn't fix it: 1 of 3 on I1, 0 of 3 on I3.
The logs showed why. Bea's agent got a brief naming Ana's task, and wrote against the code it had read anyway. The heads-up never fired, because it only fired when an agent was about to edit a teammate's file. Bea never edits billing.ts. She reads it and imports it.
Fixing team context, with the bar set first
Before writing any code for the next version (arm D2), I wrote its success bar into the method on 9 November:
- D2 passes I1 and I3 in at least 9 of 10 runs each.
- C, the control, passes I1 in 2 of 10 or fewer, so the conflict is real in this round.
- D2 asks the user nothing.
- D2 needs at most 2 extra turns (median) compared with C.
D2 adds a heads-up when an agent reads a file a teammate is changing, and a one-time check at the end of a turn: if a file it read or imported changed underneath it, the agent is asked to look again before it stops.
The result, 10 runs per scenario and arm, 90 runs in all:
| Scenario | C (leases only) | D (first wording) | D2 (shipped in 0.3.0) |
|---|---|---|---|
| I1: rename, caller 8 s later | 7 / 10 | 8 / 10 | 10 / 10 |
| I2: caller first, rename 8 s later | 10 / 10 | 9 / 10 | 9 / 10 |
| I3: signature change, caller 8 s later | 10 / 10 | 10 / 10 | 10 / 10 |
Against the bar: D2 met three of the four conditions. It passed 20 of 20, asked the user nothing, and used fewer turns than C in every scenario. The read heads-up fired in every D2 run of I1 and in no D run.
The condition it missed was the one about the control. C passed I1 7 times out of 10, where the bar said 2 or fewer. In this round Ana's commit often landed before Bea's agent read the file, so Bea saw the new name and there was no conflict to catch.
So the honest summary is 20 of 20 against 17 of 20. That fits the idea that team context fixes the stale-caller case, but the control didn't fail often enough to prove it. If I hadn't written the control condition down in advance, I could have reported "100% with Lanes" and stopped there. The bar is what made me see the result was weak.
(The benchmark page headline, 46% to 0%, pools the I1 control runs from both rounds: 6 broken builds in 13 runs, against 0 of 10 with D2. Same data, and the page says how small the samples are.)
The timing sweep, and the bug it found
The control passes when Ana's commit arrives before Bea reads the file, and fails when it arrives after. Timing decides the result, so the method's next step was a sweep: start Bea's agent 0, 4, 16 or 30 seconds after Ana's, C against D2, 5 runs each, 80 runs.
The sweep stopped at 46 of 80 runs when my machine ran out of memory. But it had already found something. At a 4-second delay, D2 passed I3 only 1 time in 5.
Here's what happened in the failing runs:
- Bea's agent read
orders.tsand decided what to write, against the oldaddOrder(total). - Ana's agent published the signature change.
- Bea's agent went to write its file. Lanes' check before the edit saw Bea's copy was behind and synced it, bringing in Ana's change.
- That sync didn't record that
orders.tshad changed under Bea. So at the end of Bea's turn, the stale check had nothing to compare, and it didn't fire. - Ana's agent had already finished. Nobody fixed the caller.
The end-of-turn check looked at files that arrived during the turn, and the sync before an edit was the one path that brought files in without recording them. The fix counts those files as arrived, plus a test that fails without it.
Two things I take from this:
- A sweep finds what a fixed delay hides. At 8 seconds the commit and the read were ordered in a way that never hit this path, and 30 runs of D2 passed.
- I'd never have found it by looking at passing runs. A fixed 8 s delay gave a clean 20 of 20 and a temptation to stop.
Where it stands
The fix changed D2's code, so the sweep runs again from scratch under a new label (D3) on the fixed build. It isn't done yet. When it is, every run goes in the repo, including the partial D2 sweep that found the bug.
The plumbing numbers don't depend on agent behaviour and are steadier: the check before an edit adds 39 ms at p95 on Linux and 68 ms on Windows, no edit was lost with 64 agents writing at once, and all 107 commits in the attribution test went to the right person.
If you're benchmarking agents
- Write the success bar before the run, including what the control has to do. A bar that only describes your arm can't tell you the scenario was too easy.
- Report every run. The failed rounds taught me more than the good ones, and they cost almost nothing to publish.
- Vary the timing. Concurrency bugs live in the gaps a fixed delay never hits.
- Use a cheap model for volume. At under a cent a run, 90 runs is less than a dollar, so there's no excuse for n = 3.
The method, scenarios, runner and every write-up are in the repo, and the summary is at trylanes.app/benchmark.
Part 3 is about the other half of the benchmark's headline: why Lanes never refuses an agent's edit, and what agents did when it did.
![When is the memory location of `Box<[u8]>` guaranteed to remain stable?](https://emoji.discourse-cdn.com/twitter/wink.png?v=16)
