The week before the good numbers
A week before we had anything worth publishing, Octomind used up to 3.1x the tokens Codex CLI used on its worst tasks, and took up to 2.3x as long. Six of them did this. I had the spreadsheet open and the column was red all the way down.
We had already told people internally that the run was going well. It was going well on average. Averages hide the tail, and the tail is where the bugs live.
So I stopped looking at aggregates and started reading traces. One hundred fifty paired runs across 65 tasks is a lot of text, but the six bad tasks narrowed it fast. Once I read those, the mechanisms were obvious enough that I could go looking for the same shapes everywhere else.
This is the notebook version. The company post has the clean table. This has the four things that were actually wrong.
What we measured
We ran Octomind 0.70.1 against Codex CLI 0.167.0. Same model on both sides, gpt-6.1-sol. Same benchmark, GOLD, 65 tasks. One hundred fifty paired runs, meaning the same task went to both clients inside the same window rather than a week apart.
The final ratios, per task: 0.75x tokens, 0.87x cost, 1.08x time. Across a full round we burned 76.6M tokens against Codex's 122.4M. Reasoning tokens came in at 96K per round for us and 196K for Codex.
Thirty-eight of the 65 tasks were solved by both clients. That intersection is the only place I trust a token comparison. A run that gives up early looks cheap for the wrong reason, and a run that thrashes for ten minutes and then fails looks expensive for a reason that has nothing to do with efficiency. If you compare clients on tasks only one of them finished, you are comparing outcomes and calling it overhead.
Two of those three ratios are wins. The third one is not, and I will get to it.
An extra model call is never one call
The first mechanism is arithmetic, and I missed it for days because it does not look like a bug in a trace. It looks like normal agent behavior.
Every call in an agent loop re-reads the whole conversation. There is no cheap way around that with a stateless completions API. So when something adds one call in the middle of a long task, you do not pay for that call's output. You pay the full accumulated prompt again, and then you pay it again on every call after it, because the extra turn is now part of the history.
One spurious call early in a task is not 1x overhead. It is a multiplier on everything downstream.
That reframed the six bad tasks for me. I stopped asking "why is this call expensive" and started asking "why is this call here at all." Three of the four mechanisms below are answers to that second question.
The replay bug fed the model a history it never wrote
We had a replay path that reconstructed a conversation and handed it back to the model. The reconstruction was wrong. It included turns the model had never emitted.
Read that from the model's side. It gets a transcript with assistant messages it did not write, in a style it did not use, committing to things it did not commit to. It has to reconcile that before it can do the task. Reasoning tokens go up, and they go up on every subsequent call because the bad history stays in context.
I found it by diffing what we sent against what the trace said we received. That diff should have been part of the harness from day one. It is now.
This one was purely ours. Codex was never going to have this bug, because Codex does not replay our reconstruction of anything.
The status instruction asked for something the model could not end
Our harness included an instruction telling the model to emit a work-in-progress status message with no tool call attached.
In a tool-calling loop, an assistant message with no tool call is how a turn ends. We were asking the model to produce a message that reads as terminal while expecting the loop to keep going. The model could not end the turn cleanly, so it kept working at a thing that had no clean exit.
I grepped Codex's messages for the same pattern: 0 hits in 241 messages. Codex never receives that instruction, so it never lands in that state. That zero is the whole finding. It is not that Codex handles the instruction better. It is that the instruction is ours, and it was malformed.
If you write harness instructions for a tool-calling agent, be careful about any instruction that describes a message shape rather than a task outcome. Shape instructions collide with the loop's termination rules, and the collision is silent. You just get more tokens.
The confidence gate never fired
We had a gate at 0.8 confidence. It was supposed to short-circuit work the model was already sure about.
It never fired in 40 runs. Not once.
So the code path was dead, and whatever we thought it was contributing to the numbers was contributing nothing. Every run I had been reasoning about as "gated" was ungated.
I still do not know which of two things is true here. Either the model never reported confidence at or above 0.8 on this benchmark, or the value never reached the check. Those have different fixes and I have not distinguished them yet. What I do know is that a threshold picked by intuition, on a signal whose distribution nobody measured, is a comment pretending to be code.
If you have a gate in your agent, log the values that pass near it. If the histogram is empty on one side, delete the gate or move it. Do not leave it there making your architecture diagram look considered.
Fourteen thousand tokens of whitespace
When a call hit our 300-second timeout, the stream had already emitted tokens. Across the runs, roughly 14,000 of them were whitespace. Fourteen thousand tokens billed for producing nothing, because a stream that is stalling still counts what it already sent.
That number is small next to 76.6M. I include it because it is the kind of line item that tells you your timeout handling is decorative. We counted the tokens and moved on. We have not made the timeout cheap.
There is also a hang I have not closed. One run sat with a silent stream for 336 seconds. That is longer than the 300-second timeout, so whatever that timeout is supposed to guard, it did not guard this. I do not have a diagnosis. It is still open, and if you run long agent sessions on a flaky network you should assume you have the same bug and go looking before it shows up in a cost column.
Where we still lose
Time is 1.08x. We are slower than Codex CLI per task, and after all the token work that did not move to a win. Fewer tokens does not automatically mean fewer seconds, because wall clock is dominated by round trips and provider latency rather than by volume. We cut volume. We did not cut round trips.
Pydantic went the wrong way. After the fixes, that task sits at 1.57x Codex's tokens. Everything else improved and this one regressed, which means one of the fixes has a bad interaction I have not isolated. My guess is that removing the extra calls changed the shape of the history enough to push that task down a longer path, but that is a guess and I am labeling it as one.
I would rather publish a regression with a name on it than average it into a ratio. The 0.75x is real. It is also real that one task got worse and we do not fully know why.
The comparison that measured the wrong thing
Before the paired runs, we compared the two clients a week apart. Octomind averaged 11.9 seconds per model call. Codex averaged 7.2.
That gap was not us. Running a week apart on a hosted model measures the provider, not the clients. Load shifts, routing shifts, and the model endpoint you hit on Tuesday is not the endpoint you hit on the following Tuesday. We nearly spent a week optimizing a latency difference that belonged to somebody else's infrastructure.
Pairing fixed it. Same task, same window, alternating clients. If your benchmark has a "we ran the baseline last month" step in it, that step is measuring time rather than code.
Running the same measurement on your own harness
You do not need our setup for this. You need a task list, two clients, and a results file.
Write one function that runs a task with a client and prints a single JSON object. Everything else below is generic.
# you implement this: print one JSON object per run # {"solved":bool,"tokens":int,"reasoning":int,"cost":float,"time":float} run_one() { client="$1"; task="$2" # your invocation here } export CLIENT_A=octomind CLIENT_B=codex for task in tasks/*; do for client in "$CLIENT_A" "$CLIENT_B"; do run_one "$client" "$task" \ | jq -c --arg c "$client" --arg t "$task" '. + {client:$c, task:$t}' \ >> results.jsonl done done Interleave the clients inside the loop. Do not run all of A and then all of B.
Then take the ratios on the tasks both solved:
jq -s ' group_by(.task) | map({task: .[0].task, a: (map(select(.client == env.CLIENT_A)) | .[0]), b: (map(select(.client == env.CLIENT_B)) | .[0])}) | map(select(.a.solved and .b.solved)) | map({task, tokens: (.a.tokens / .b.tokens), cost: (.a.cost / .b.cost), time: (.a.time / .b.time)}) ' results.jsonl > ratios.json Now sort for the tail, because the tail is where the bugs are:
jq -s 'sort_by(-.tokens) | .[0:6]' ratios.json Those six are the ones to read as traces. Record reasoning tokens separately from total tokens, or you will not be able to tell a long conversation from a model that is thinking hard inside a short one. They have completely different fixes.
What I still do not know
The 336-second silent-stream hang is open. The Pydantic regression at 1.57x is unexplained past a guess. The confidence gate is unresolved between a bad threshold and a broken signal path. Time is 1.08x and I have no plan for it yet beyond counting round trips, which I have not done.
What I take from the week is narrower than the headline ratio. Every mechanism that cost us tokens was something we put in the model's way: a malformed instruction, a bad replay, a dead gate, an extra call that compounded. None of them were the model being slow. When the traces got bad, the cause was upstream of the model every single time I looked.
Read your own worst six before you read your average.
