AI & ML
What Does a Harness Buy? Tokens, Mostly
arXiv cs.SE0 views
arXiv:2610.04433v2 Announce Type: replace-cross Abstract: A coding agent is a language model inside a harness: the system prompt, the tools, and the context management that let a chat model work in a repository. Most of the engineering on these agents goes into the harness, and vendors and leaderboards credit it with score gains. Earlier studies have fixed the model and swapped the harness, but have not measured whether the swap changes the pass rate more than running the same harness again. We run seven models through three production harnesses, Claude Code, mini-SWE-agent, and OpenCode, on SWE-bench Verified and two benchmarks of other kinds, and rerun them unchanged. No harness leads consistently across models. Where the data can bound the difference between two harnesses, the bound is 2 to 5 points, and a harness swap changes about as many task outcomes as a rerun. The exceptions are harness effects in particular pairings of model and harness, whose direction changes with the model: OpenCode trails by 8 to 9 points on one model and leads by 14 on another. Cost differences between harnesses also depend on the model, from under 10% to 3 to 4 times. Which harness is better therefore has no answer apart from the model, and a harness difference should be reported next to a rerun.
Source:https://arxiv.org/abs/2610.04433PreviousPrior Knowledge or Search? A Study of LLM Agents in Hardware-Aware Code OptimizationNextPreventing Duplicate Orders in PHP and MySQL: Transactions, Idempotency and Safe Retry Handling
Related posts

How to Interpret Logistic Regression and Odds Ratios
0dev.to
Kubernetes Service Account Tokens Do Not Need to Ride Along
0dev.to Kubernetes
Export a calendar day to ICS for your system calendar
0dev.to OpenSource
Comparing Axleta Repository Packer with Gitingest
0dev.to Programming

Animating and rigging by hand is sometimes hard....
0dev.to Programming
Anthropic is bullshitting you about subagents and loops.
0dev.to Programming