Making Agent Harnesses Readable Lets Teams Edit Agent Behavior Without Breaking It

Harness Handbook: Making Evolving Agent Harnesses Readable, Navigable, and Editable

Authors: Ruhan Wang, Yucheng Shi, Zongxia Li, Zhongzhi Li, Yue Yu, Junyao Yang, Kishan Panaganti, Haitao Mi, Dongruo Zhou, Leoweiliang (Tencent Hunyuan)

---

1]Tencent HY LLM Frontier 2]Indiana University 3]University of Maryland, College Park 4]University of Georgia 5]National University of Singapore \contribution⋆Work done at an internship at Tencent Seattle †Lead Project Collaborator. \headercontentProject link: https://ruhan-wang.github.io/Harness-Handbook/

Abstract

The capability of a modern AI agent depends not only on its foundation model but also on its harness, which constructs prompts, manages state, invokes tools, and coordinates execution. As models, APIs, execution environments, and application requirements change, the harness must be continually modified to add capabilities or adapt existing behaviors. Before a human developer or coding agent can make such a change, they must identify all code locations that implement the target behavior. This is difficult because production harnesses are often large, tightly coupled, and behaviorally distributed across files, functions, execution stages, and state transitions, whereas modification requests describe what the system should do and repositories are organized by files, functions, and modules. Existing approaches to code search, repository indexing, and long-context processing make code easier to inspect, but they still leave developers and coding agents to recover this mapping themselves. Behavior localization is therefore a central bottleneck in harness evolution. We introduce the Harness Handbook, a behavior-centric representation synthesized automatically from a harness codebase through static program analysis and LLM-assisted behavioral structuring, which organizes implementation knowledge around system behaviors and links each behavior to the corresponding source code. We also introduce Behavior-Guided Progressive Disclosure (BGPD), which guides coding agents from high-level behavior descriptions to relevant implementation details and verifies candidate locations against the current source. We evaluate Harness Handbook on diverse modification requests from two open-source agent harnesses. Handbook-Assisted planning improves behavior localization and edit-plan quality while using fewer planner tokens. The largest gains appear for changes involving scattered implementation sites, rarely executed code paths, and cross-module interactions. These findings indicate that evolving complex agentic systems depends not only on generating edits, but also on determining where those edits should be made.

1 Introduction

Recent advances in large language models (LLMs) have accelerated the development of agentic systems [1, 2, 3]. These systems extend model capabilities by interacting with tools, APIs, and complex execution environments [4, 5, 6, 7]. Such interactions are not governed by the foundation model alone. They are coordinated by the harness, which constructs prompts, manages state, invokes tools, and controls execution across system components. The harness therefore determines how model capabilities are translated into system behavior. As models, APIs, execution environments, and application requirements evolve, the harness must be adapted accordingly. Harness evolution has consequently become a recurring engineering challenge in modern agent development [8, 9, 10].

In practice, harness evolution requires changes that add capabilities, adapt existing behaviors, or refine execution workflows [11, 12]. Traditionally, human developers make these changes by reading the repository, tracing the relevant behavior, and editing its implementation. Coding agents are increasingly expected to perform the same work from natural-language requests [7, 13, 10, 14]. However, a modification request describes what behavior should change, but it does not identify where that behavior is implemented. Whether the modification is performed by a human developer or a coding agent, the first step is therefore to locate all relevant implementation sites.

Finding every relevant implementation site is difficult in production-scale harnesses [15]. A large harness may span hundreds of functions across many files, with execution logic distributed across stages and connected through shared state. As a result, a single behavior may depend on several nonadjacent implementation sites.

We refer to identifying all implementation sites associated with a behavioral request as behavior localization. In a large and structurally complex harness repository, this task is costly for both human developers and coding agents. Human developers must invest substantial time and effort in building a mental model of the system and tracing behavior across files and execution stages. Coding agents face an additional constraint from limited input context, which prevents them from examining all relevant code at once. They must therefore explore the repository iteratively and may still overlook scattered or rarely executed paths [16, 17].

Existing approaches to repository understanding make codebases easier to explore. Repository maps, code search, code summarization, repository memory, and long-context editing help coding agents find, inspect, and retain relevant code [16, 18, 19, 17, 20]. However, these approaches organize information around files, functions, and modules, whereas a harness modification request describes a desired behavior. They may identify individual pieces of relevant code, but they do not show how those pieces work together to produce the behavior or whether every affected location has been found. The coding agent must still connect these pieces and infer how the requested behavior maps to the underlying implementation. This missing connection is the gap we address.

We address this gap with Harness Handbook, an operational behavior representation that connects system behavior to source code. Instead of organizing implementation knowledge by files and functions, the Handbook organizes it by what the harness does and links each behavior to the code that implements it. Developers and coding agents can therefore identify the relevant behavior first and then navigate directly to the corresponding code. We construct Harness Handbook instances automatically using static program analysis and LLM-assisted behavioral structuring. To evaluate whether the Handbook improves harness modification, we use realistic requests from two open-source agent harnesses and compare Baseline and Handbook-Assisted planning on the same tasks. Handbook-Assisted planning improves behavior localization and edit-plan quality while using fewer planner tokens. These results show that making the connection between behavior and implementation explicit can make harness evolution more accurate and efficient. Our contributions are as follows:

• We define behavior localization as finding all code locations that implement the behavior described in a modification request. This step is necessary before a complete edit can be planned.
• We introduce Harness Handbook, which organizes implementation knowledge by what the harness does and links each behavior directly to its source code. We also develop an automated framework that builds Handbook instances from existing codebases using static program analysis and LLM-assisted behavioral structuring.
• We introduce Behavior-Guided Progressive Disclosure (BGPD), a workflow that guides coding agents from high-level behavior descriptions to relevant implementation details in stages.
• We evaluate Harness Handbook on diverse modification requests from two open-source harnesses. We compare Handbook-Assisted BGPD planning with planning without Handbook access on the same tasks. Handbook assistance improves behavior localization and edit-plan quality while using fewer planner tokens.

2 Related Work

Our work is closely related to three research directions: agent harnesses, harness evolution, and repository representations for coding agents. We briefly review each direction and discuss how Harness Handbook differs from existing approaches by introducing an explicit operational behavior representation for behavior localization.

2.1 Agent Harness

Recent work has increasingly recognized the agent harness as a first-class software abstraction that extends foundation models into deployable agentic systems. Rather than treating prompting, tool use, memory, and execution logic as implementation details, Code as Agent Harness formalizes the harness as an executable and stateful software layer supporting reasoning, acting, feedback, and coordination [15]. Complementary surveys and engineering efforts further characterize the harness as the runtime infrastructure responsible for prompt construction, state management, tool invocation, control flow, and interactions with external environments [21, 22]. This abstraction has been widely adopted by modern agent frameworks and production systems. Frameworks such as AutoGen and OpenHands expose modular runtime components for agent orchestration, tool execution, environment interaction, and multi-agent collaboration [23, 13]. Industrial systems such as Claude Code and Codex similarly emphasize harness design as a key factor in building reliable long-running coding agents [24, 25]. While these works establish the importance of agent harnesses, they primarily focus on harness construction and system design rather than understanding and evolving existing production-scale harnesses.

2.2 Harness Evolution

As coding agents become increasingly capable, software engineering efforts are gradually shifting from manually implementing functionality to automatically evolving the underlying harness [7, 13]. Recent work has begun to investigate harness engineering, adaptive harness optimization, and automated harness repair [10, 14, 8, 26, 27, 28]. These approaches improve the capability and reliability of agent systems by constructing, adapting, or repairing harness implementations. Our work addresses a complementary problem. Rather than directly modifying harness implementations, we study the prerequisite challenge of harness evolution: behavior localization. Harness Handbook enables developers and coding agents to identify where behavioral changes should be made before reasoning about how to modify the underlying implementation.

2.3 Behavior Representations

Recent work has proposed various representations to improve repository understanding for coding agents, including structural repository representations, repository indexing, repository memory, and natural-language artifacts [29, 18, 19, 30]. These representations improve repository navigation, code retrieval, and editing by organizing implementation knowledge into more accessible forms. However, existing representations remain fundamentally implementation-centric, organizing information around source code, repository structure, or execution infrastructure. They do not explicitly capture how system behaviors emerge across distributed execution stages, functional modules, and shared states. In contrast, Harness Handbook introduces an operational behavior representation that explicitly bridges behavioral requirements and their underlying implementation, enabling behavior localization before repository exploration.

3 Harness Handbook

Harness Handbook has three parts for understanding and modifying an agent harness. The representation organizes source around runtime behavior (Section 3.1). The construction pipeline builds this representation from a repository (Section 3.2). The modification workflow uses the handbook to guide code changes and automatically resynchronizes it after every non-empty repository diff (Section 3.3).

3.1 Harness Handbook Representation

Source repositories show where code is stored, but they do not directly show how a runtime behavior unfolds. One behavior may cross several files, execution stages, and shared states. Harness Handbook reorganizes this information around behavior while preserving links to the source. As Figure 1 shows, it contains an L1–L3 document tree \mathcal{D} and a complementary state-register view \mathcal{Z}.

Figure 1: Overview of the Harness Handbook representation. Its three-level hierarchy progresses from a system-level overview to stage-level component overviews and source-backed unit details. The navigation pane provides component and state indexes for direct access and cross-stage tracing.

A reader normally starts at L1 (system overview), which summarizes the architecture, execution model, major stages, and global data flow. The reader then moves to L2 (component overview) for the responsibilities, inputs, outputs, dependencies, and local state of a selected stage. Finally, L3 (unit deep dive) links that stage to source-grounded implementation entries. The complementary view \mathcal{Z} records state relationships that cross stage boundaries.

Two rules keep the representation useful. Progressive disclosure means that readers move from L1 to L3 only when a task requires more detail. Behavior–implementation alignment means that every active L3 locator must still resolve to the current repository. If a locator cannot be revalidated, its entry is frozen and excluded from localization until it is refreshed. The repository therefore remains the authority for implementation details.

3.2 Harness Handbook Construction

Construction builds a handbook from a repository \mathcal{R}. The leaf mode g∈{function,file}, which remains fixed for the lifetime of the handbook, determines the granularity of L3 entries: in function-as-leaf mode each L3 entry covers a whole function or one or more contiguous regions, whereas in file-as-leaf mode each L3 entry represents a file.

The two modes differ in how the stage skeleton is obtained. function-as-leaf starts from a seed skeleton \mathcal{S}_{0} containing the stage and state-register definitions; we use it when a trustworthy seed skeleton that faithfully reflects the harness’s execution stages is available and function-level L3 entries fit the available budget. file-as-leaf instead infers the stage skeleton rather than starting from a seed; we use it when no such seed is available or when function-level organization would exceed the available budget. Both modes produce the representation described in Section 3.1, and Algorithm 2 defines the two branches.

Once g is chosen, construction proceeds in three phases, illustrated for the function-as-leaf branch in Figure 2.

Figure 2: Construction pipeline for Harness Handbook. Static analysis extracts source-linked facts, behavioral organization maps source units to execution stages, and hierarchical synthesis builds the L1–L3 handbook.

1. Phase I. Static Fact Extraction. Language-specific adapters parse the repository and extract functions, named external boundaries, source locations, signatures, and call edges. The program graph \mathcal{G} keeps only calls that resolve to an internal function or a named boundary. Unresolved calls are logged rather than assigned guessed targets. This phase is deterministic and makes no LLM calls.
2. Phase II. Behavioral Organization. The pipeline organizes source units into the execution-stage skeleton \mathcal{S} according to the selected leaf mode. • Function-as-leaf. The pipeline uses source and call-graph context to propose function-to-stage assignments, which are refined through iterative review. A function can be assigned to one or more execution stages, either as a whole or as contiguous regions when it serves multiple behavioral roles. Accepted assignments and revisions update the stage skeleton \mathcal{S}. • File-as-leaf. The pipeline summarizes the scanned files as cards and combines them with the program graph to infer the stage skeleton \mathcal{S}. It then organizes the files by execution stage. An optional proposal-and-review process iteratively refines this structure, while uncovered files and unresolved organizational issues remain explicitly recorded in \mathcal{Y}. Appendix A provides the full organization procedures and validation checks for both modes.
3. Phase III. Hierarchical Synthesis and Packaging. This phase converts the stage skeleton and source organization from Phase II into the L1–L3 document tree and cross-stage state-register view. Each L3 entry is linked to a statically identified source location and validated against the current repository. This keeps the handbook traceable to the source. Finally, the pipeline renders \mathcal{V} and packages the structured data needed for source localization and future resynchronization.

3.3 Handbook-Guided Modification and Resynchronization

After construction, the handbook serves as a behavior-oriented guide for modifying the repository. Given a request q, the workflow uses the handbook \mathcal{H} together with its corresponding repository \mathcal{R}. The leaf mode g remains unchanged throughout the process.

The workflow consists of four steps. First, Behavior-Guided Progressive Disclosure (BGPD) localizes the requested behavior through coarse-to-fine handbook navigation. It progressively reveals relevant information, follows it to candidate source locations, and verifies those locations against the current repository. This process produces a set of source-grounded evidence for planning. Second, the planner converts the evidence into an edit plan \mathcal{P} and action declarations Γ. Third, a separate executor applies \mathcal{P} to the repository. Finally, any non-empty diff triggers handbook resynchronization. Algorithm 1 summarizes the complete workflow.

Algorithm 1 Handbook-Guided Modification with Automatic Resynchronization

3.3.1 Behavior Localization with BGPD

Rather than treating all source locations equally, BGPD uses the handbook to identify candidate sites from coarse to fine. It then verifies these sites against the current repository.

Localization begins at the level of execution stages. BGPD first uses L1 and L2 of \mathcal{D} to identify the stages directly relevant to the request. It then follows \mathcal{Z} to include stages coupled through shared state. This step captures mutually dependent stages even when they are structurally distant. Within these stages, BGPD selects the most relevant L3 entries and retrieves their source locators.

BGPD then expands the candidate set along call relations. In function-as-leaf mode, it uses the function-call graph; in file-as-leaf mode, it uses the induced file-call graph. External boundary nodes provide context but are never returned as edit sites.

Up to this point, BGPD operates on the handbook. It then opens the current repository \mathcal{R}, resolves the candidate locators, and retains only the sites that remain relevant to q. This yields the verified evidence \widehat{\mathcal{E}}_{q}. Each record contains a file path, an optional function or region anchor, and a current source excerpt. The handbook thus guides the search, while the repository remains the basis for the edit plan.

3.3.2 Edit Planning and Execution

The planner converts \widehat{\mathcal{E}}_{q} into an edit plan \mathcal{P}. Each edit block specifies where the change should be made, which source evidence supports it, and what should change. The block records a target file, an optional function or region anchor, a current source excerpt, and the intended change.

For each edit, the planner also records an action declaration containing the target file, optional anchor, and one of three action types: modify, add, or remove. The resulting declarations are collected as Γ=(Γmodify,Γadd,Γremove). A rename is represented as one removal and one addition.

The executor applies \mathcal{P} to \mathcal{R} and produces the updated repository \mathcal{R}^{\prime}. The resulting diff is \Delta=\mathrm{Diff}(\mathcal{R},\mathcal{R}^{\prime}). Because Δ is the factual record of what changed, it drives handbook invalidation. By contrast, Γ is used only to check whether execution followed the plan.

3.3.3 Handbook Resynchronization

Every non-empty diff Δ triggers Resyncg. Whenever possible, the procedure updates only the affected parts rather than rebuilding the entire handbook.

First, it reparses the changed source, refreshes the program graph, and aligns the old and new versions to identify added, removed, and modified units: in function-as-leaf mode, it matches functions using fingerprints that do not depend on line numbers, and in file-as-leaf mode, it matches files using file-set differences and content hashes.

Second, it determines the update scope. If the stage skeleton \mathcal{S} remains valid, only the affected entries and their enclosing structure are refreshed; otherwise, Algorithm 2 is rerun on \mathcal{R}^{\prime} with the same leaf mode g and stored configuration Θ.

Finally, content that cannot be parsed or classified is handled conservatively: it is frozen or recorded in the coverage record rather than guessed.

Once the update completes, the new pair (\mathcal{R}^{\prime},\mathcal{H}^{\prime}) becomes the starting state for the next request. Within resynchronization, model calls are limited to four semantic steps: classification, file assignment, within-stage organization, and description revision. All other operations are deterministic.

4 Experiment

We evaluate Harness Handbook on two open-source agent harnesses around three research questions, describing the experiment setup in Section 4.1 and reporting results in Section 4.2.

• RQ1: Does handbook-guided localization improve plan quality while reducing planning cost?
• RQ2: With the handbook, can a weaker planner match the implementation-site localization of substantially more capable models?
• RQ3: Do these gains persist across request types and localization difficulty levels?

Figure 3: Plan quality and planner token usage on Codex and Terminus-2. (a) Overall win rates aggregated across the three judges. (b) Win rates reported separately by GPT-5.5, Opus 4.8, and DeepSeek-V4-Pro. (c) Average number of planner tokens per request; lower values indicate greater efficiency.

Figure 4: Per-judge win rates for three evaluation dimensions. Rows show results for Codex and Terminus-2, while columns show Localization, Scope Control, and Reasoning.

4.1 Experiment Setup

Evaluation Setting.

This experiment evaluates localization and edit planning in the modification workflow. Given a natural-language request, a read-only planner built with NexAU [31] and powered by DeepSeek-V4-Pro produces an edit plan \mathcal{P}, and we score the quality of \mathcal{P} directly.

We compare two arms: the Baseline explores the repository directly, whereas the Handbook-Assisted arm locates relevant source code under handbook guidance using a BGPD-consistent navigation policy. Apart from handbook access and navigation, the two arms are identical in requests, model, repository, tool permissions, and decoding settings. The handbook used in the Handbook-Assisted arm is built from the same source.

Plan quality is judged independently by three models: GPT-5.5, Opus 4.8, and DeepSeek-V4-Pro.

Figure 5: Win rates by (a) modification request type and (b) localization difficulty. Q, CF, and SH denote Query, Cross-file, and Search-Hostile requests.

Benchmarks and Modification Requests.

We evaluate Terminus-2 [32] and Codex [25], two open-source agent harnesses that adopt the two leaf modes from Section 3.2: Terminus-2 uses function-as-leaf because it has a reliable seed skeleton, and Codex uses file-as-leaf to infer the hierarchy at repository scale.

Each harness contributes 30 behavior-driven modification requests. They are grouped by type into Query (Q), which modifies existing behavior without revealing target locations; Cross-file (CF), which adds an end-to-end capability spanning files or modules; and Search-Hostile (SH), which places relevant implementations in locations that are hard to recover by keyword search. Requests are also labeled Easy, Medium, or Hard by localization difficulty.

Evaluation Metrics.

We evaluate plan quality, localization accuracy, and planning cost.

Plan quality.

Three independent judges score each plan on three dimensions: Localization (whether the edit sites are accurate), Scope Control (whether the plan stays focused, with less scope bloat), and Reasoning (whether the rationale and supporting evidence are adequate). The weighted score is

S=0.5S_{\mathrm{Loc}}+0.25S_{\mathrm{Scope}}+0.25S_{\mathrm{Reason}}, 

on a 0–100 scale, with Localization receiving the largest weight. Handbook-Assisted and Baseline are declared the winner when their scores differ by at least δ=3 points; otherwise the comparison is a tie. The win rate for any reported slice is the number of wins divided by the number of valid judge–request comparisons in that slice.

Localization accuracy.

We compare each predicted edit plan with independent reference plans from Opus 4.8 and GPT-5.5, reporting Recall, Precision, F1, and Wrong at file and symbol granularity. Wrong is the share of valid requests with zero overlap against the reference; lower is better.

Planning cost.

We report the average number of planner tokens per request, including the handbook and source context consumed during localization and plan construction.

4.2 Experiment Results

The results show three consistent patterns. Handbook guidance improves plan quality without increasing planner token use. It also helps a weaker planner produce file- and symbol-level predictions that more closely align with reference plans from stronger models. These gains persist across request types and localization difficulty levels. The following subsections examine each finding in turn.

4.2.1 Better Plans at Lower Token Cost

Figure 3 summarizes plan-quality win rates and average planner token use. The Handbook-Assisted arm achieves a higher overall win rate on both harnesses: 38.3% versus 28.3% on Codex and 45.6% versus 26.7% on Terminus-2. The improvement is consistent across all three judges: the gap is 10.0 percentage points for every judge on Codex and ranges from 13.3 to 26.7 points on Terminus-2. Thus, the direction of the result does not depend on any single judge.

Figure 4 decomposes these judgments by dimension. Averaged across the three judges, the Handbook-Assisted arm has higher win rates for Localization, Scope Control, and Reasoning on both harnesses. The respective gains are 12.2, 6.7, and 4.5 points on Terminus-2, and 2.2, 1.1, and 3.3 points on Codex.

The quality gains are accompanied by lower planner token use. Average use falls from 0.102M to 0.089M tokens per Codex request (12.7%) and from 0.058M to 0.053M tokens per Terminus-2 request (8.6%). The simultaneous increase in win rate and decrease in token use shows that the improvement is not explained by a larger planning-token budget.

4.2.2 A Weaker Planner Matches Stronger Models

To assess whether handbook guidance helps a weaker planner approach stronger models in implementation-site localization, we compare its predicted edit sites with independent reference plans from Opus 4.8 and GPT-5.5 (Table 1).

			Opus 4.8	GPT-5.5
Harness	Level	Metric	Baseline	Handbook	Gap	Baseline	Handbook	Gap
Codex	File	Recall	55.4	69.7	+14.3	49.4	49.8	+0.4
Precision	46.1	60.2	+14.1	53.0	62.2	+9.2
F1	46.6	61.8	+15.2	47.3	52.3	+5.0
Wrong ↓	37.0	14.8	-22.2	21.4	21.4	0.0
Symbol	Recall	47.1	65.4	+18.3	46.4	49.1	+2.7
Precision	38.0	55.3	+17.3	48.4	60.4	+12.0
F1	38.3	57.1	+18.8	43.8	51.2	+7.4
Wrong ↓	44.4	18.5	-25.9	28.6	21.4	-7.2
Terminus-2	File	Recall	74.7	83.9	+9.2	76.1	87.5	+11.4
Precision	74.1	86.2	+12.1	78.3	93.3	+15.0
F1	74.1	84.7	+10.6	76.5	89.3	+12.8
Wrong ↓	24.1	13.8	-10.3	20.0	6.7	-13.3
Symbol	Recall	64.7	76.3	+11.6	73.0	87.5	+14.5
Precision	65.1	78.4	+13.3	73.9	93.3	+19.4
F1	64.8	77.1	+12.3	73.0	89.3	+16.3
Wrong ↓	24.1	13.8	-10.3	20.0	6.7	-13.3

Table 1: Reference-plan localization metrics (%). Predicted implementation sites are compared against independent reference plans generated by Opus 4.8 and GPT-5.5 at file and symbol granularity. Metrics are macro-averaged across valid requests; missing outputs and planner errors are excluded. Wrong indicates zero overlap with the reference plan at the reported granularity, so lower values are better.

Across both harnesses, both reference models, and both granularities, all 24 Recall, Precision, and F1 comparisons are higher for the Handbook-Assisted arm, with F1 gains ranging from 5.0 to 18.8 points. Recall and precision increase together, so the improvement is not explained by returning more candidate sites at the expense of focus.

The absolute agreement is highest on Terminus-2. With handbook guidance, the weaker planner reaches file-level F1 scores of 84.7% and 89.3%, and symbol-level scores of 77.1% and 89.3%, against the Opus 4.8 and GPT-5.5 references, respectively. Precision reaches 93.3% against GPT-5.5 at both granularities. The consistent direction across two independent references shows that the improvement is not specific to one reference model.

Handbook guidance also reduces complete localization failures. Wrong, the share of requests with zero overlap against the reference, never increases and falls by as much as 25.9 points. Thus, the gains include fewer complete misses, not only better overlap among already plausible plans.

4.2.3 Consistent Gains Across Requests and Difficulty

Figure 5 stratifies the win-rate results by request type and localization difficulty.

All six harness-by-type comparisons favor the Handbook-Assisted arm, with gains ranging from 16.3 to 33.3 percentage points. Codex improves most on Query requests (26.7 points), whereas Terminus-2 improves most on Search-Hostile requests (33.3 points). Cross-file requests improve by 16.3 and 20.0 points, respectively. The gains therefore span existing-behavior changes, cross-file capabilities, and requests that are difficult to localize through keyword search.

All six harness-by-difficulty comparisons are also positive, ranging from 3.7 to 33.3 points. Codex improves most on Easy requests, whereas Terminus-2 improves most on Medium requests. Because the gains are not monotonic in labeled difficulty, the difficulty label alone does not explain their variation.

5 Conclusion

This work identifies behavior localization as a central challenge in evolving production-scale agent harnesses and introduces Harness Handbook, a behavior-centric representation that links behavioral descriptions to their distributed source implementations. Harness Handbook reorganizes repository knowledge around runtime behavior, constructs this representation directly from source, and supports modification through behavior-guided progressive disclosure and automatic resynchronization.

Our experiments demonstrate the value of this representation along three dimensions. First, handbook guidance improves plan quality while lowering planning cost: on Codex and Terminus-2, overall win rates rise by 10.0 and 18.9 percentage points, while planner token use falls by 12.7% and 8.6%, respectively. Second, with the handbook a weaker planner matches the implementation-site localization of substantially stronger models, improving all 24 file- and symbol-level Recall, Precision, and F1 comparisons against two independent reference plans. Third, these gains persist across request types and localization difficulty levels. Together, these results indicate that making the relationship between behavior and implementation explicit can improve both modification planning and localization efficiency without requiring a larger planning-token budget.

The utility of Harness Handbook extends beyond modification planning. As a behavior-centric repository representation that stays synchronized with the code, it can also support broader uses such as behavior auditing and regression-impact analysis. Our next step is to apply it to harness self-evolving: using the handbook as a shared behavioral memory, an agent can autonomously close the loop of localization, planning, execution, and resynchronization as the repository evolves, moving the harness toward self-improvement.

References

[1] L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, et al. (2024) A survey on large language model based autonomous agents. Frontiers of Computer Science 18 (6), pp. 186345. Cited by: §1.

[2] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao (2022) React: synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629. Cited by: §1.

[3] C. Huang, J. Wu, Y. Xia, Z. Yu, R. Wang, T. Yu, R. Zhang, R. A. Rossi, B. Kveton, D. Zhou, et al. (2025) Towards agentic recommender systems in the era of multimodal large language models. arXiv preprint arXiv:2503.16734. Cited by: §1.

[4] T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom (2023) Toolformer: language models can teach themselves to use tools. Advances in neural information processing systems 36, pp. 68539–68551. Cited by: §1.

[5] S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, et al. (2024) Webarena: a realistic web environment for building autonomous agents. In International Conference on Learning Representations, Vol. 2024, pp. 15585–15606. Cited by: §1.

[6] T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, et al. (2024) Osworld: benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems 37, pp. 52040–52094. Cited by: §1.

[7] J. Yang, C. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press (2024) Swe-agent: agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems 37, pp. 50528–50652. Cited by: §1, §1, §2.2.

[8] J. Lin, S. Liu, C. Pan, L. Lin, S. Dou, Z. Xi, X. Huang, H. Yan, Z. Han, T. Gui, et al. (2026) Agentic harness engineering: observability-driven automatic evolution of coding-agent harnesses. arXiv preprint arXiv:2604.25850. Cited by: §1, §2.2.

[9] R. Lopopolo (2026) Harness engineering: leveraging codex in an agent-first world. OpenAI. Note: https://openai.com/index/harness-engineering/OpenAI Engineering Blog Cited by: §1.

[10] H. Zhong and S. Zhu (2026) AI harness engineering: a runtime substrate for foundation-model software agents. arXiv preprint arXiv:2605.13357. Cited by: §1, §1, §2.2.

[11] R. Wang, Z. Wang, C. Huang, R. Wang, T. Yu, L. Yao, J. Lui, and D. Zhou (2025) Federated in-context learning: iterative refinement for improved answer quality. arXiv preprint arXiv:2506.07440. Cited by: §1.

[12] R. Wang, C. Huang, Z. Wang, J. Wu, R. Wang, T. Yu, J. McAuley, L. Yao, and D. Zhou (2026) FERA: uncertainty-aware federated reasoning for large language models. arXiv preprint arXiv:2605.10082. Cited by: §1.

[13] X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, et al. (2025) Openhands: an open platform for ai software developers as generalist agents. In International Conference on Learning Representations, Vol. 2025, pp. 65882–65919. Cited by: §1, §2.1, §2.2.

[14] N. Zhu, H. Wang, J. Zhou, F. Chen, S. Zhang, G. Chen, C. Liu, J. Wu, W. Chen, X. Mou, et al. (2026) SemaClaw: a step towards general-purpose personal ai agents through harness engineering. arXiv preprint arXiv:2604.11548. Cited by: §1, §2.2.

[15] X. Ning, K. Tieu, D. Fu, T. Wei, Z. Li, Y. Bei, J. Zou, M. Ai, Z. Liu, T. Li, et al. (2026) Code as agent harness. arXiv preprint arXiv:2605.18747. Cited by: §1, §2.1.

[16] S. Zhang, Y. Wang, J. Liang, Y. Shi, W. Zeng, M. Wang, S. He, N. Xu, S. Ye, K. Cai, et al. (2026) SWE-explore: benchmarking how coding agents explore repositories. arXiv preprint arXiv:2606.07297. Cited by: §1, §1.

[17] H. Tamoyan, S. Narenthiran, E. Arakelyan, M. Mezini, and B. Ginsburg (2026) SHERLOC: structured diagnostic localization for code repair agents. arXiv preprint arXiv:2606.24820. Cited by: §1, §1.

[18] I. Bhola, A. Krishnan, S. Kurmala, and M. NS (2026) Code isn’t memory: a structural codebase index inside a coding agent. arXiv preprint arXiv:2606.22417. Cited by: §1, §2.3.

[19] B. Wang, W. Xu, Y. Li, M. Gao, Y. Xie, H. Sun, and D. Chen (2025) Improving code localization with repository memory. arXiv preprint arXiv:2510.01003. Cited by: §1, §2.3.

[20] W. Cao, X. Yin, B. Dhingra, and S. Zhou (2026) Coding agents are effective long-context processors. arXiv preprint arXiv:2603.20432. Cited by: §1.

[21] Q. Meng, Y. Wang, L. Chen, Q. Wang, C. Lu, W. Wu, Y. Gao, Y. Wu, and Y. Hu (2026) Agent harness for large language model agents: a survey. Cited by: §2.1.

[22] C. He, X. Zhou, D. Wang, H. Xu, W. Liu, and C. Miao (2026) Harness engineering for language agents: the harness layer as control, agency, and runtime. Cited by: §2.1.

[23] Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, et al. (2023) Autogen: enabling next-gen llm applications via multi-agent conversation. arXiv preprint arXiv:2308.08155. Cited by: §2.1.

[24] P. Rajasekaran (2026) Harness design for long-running application development. Note: https://www.anthropic.com/engineering/harness-design-long-running-appsAnthropic Engineering Blog Cited by: §2.1.

[25] OpenAI (2025) Introducing codex. Note: https://openai.com/index/introducing-codex/OpenAI Research Preview Cited by: §2.1, §4.1.

[26] T. Chen, S. Lu, K. Zhao, W. Meng, H. Teng, T. Li, C. Li, X. Liu, J. Liang, Z. Zhang, et al. (2026) HarnessX: a composable, adaptive, and evolvable agent harness foundry. arXiv preprint arXiv:2606.14249. Cited by: §2.2.

[27] M. Chen, C. Lv, G. Zhang, H. Chang, and S. Zhou (2026) HarnessForge: joint harness and policy evolution for adaptive agent systems. arXiv preprint arXiv:2606.01779. Cited by: §2.2.

[28] M. Chen, J. Wang, Z. Liu, Y. Wang, and Q. Wang (2026) From failed trajectories to reliable llm agents: diagnosing and repairing harness flaws. arXiv preprint arXiv:2606.06324. Cited by: §2.2.

[29] T. Cherny-Shahar and A. Yehudai (2026) Repository intelligence graph: deterministic architectural map for llm code assistants. arXiv preprint arXiv:2601.10112. Cited by: §2.3.

[30] L. Pan, L. Zou, S. Guo, J. Ni, and H. Zheng (2026) Natural-language agent harnesses. arXiv preprint arXiv:2603.25723. Cited by: §2.3.

[31] Nex-AGI Team, Y. Cai, L. Chen, Q. Chen, Y. Ding, L. Fan, W. Fu, Y. Gao, H. Guo, P. Guo, et al. (2025) Nex-n1: agentic models trained via a unified ecosystem for large-scale environment construction. arXiv preprint arXiv:2512.04987. External Links: 2512.04987, Link Cited by: §4.1.

[32] M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y. Shin, T. Walshe, E. K. Buchanan, et al. (2026) Terminal-bench: benchmarking agents on hard, realistic tasks in command line interfaces. In The Fourteenth International Conference on Learning Representations, External Links: 2601.11868, Link Cited by: §4.1.

[33] S. Gandhi, L. Gao, and J. Callan (2025) Repository-level code search with neural retrieval methods. arXiv preprint arXiv:2502.07067.

[34] V. Lomshakov, A. Podivilov, S. Savin, O. Baryshnikov, A. Lisevych, and S. Nikolenko (2024) ProConSuL: project context for code summarization with llms. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 866–880.

[35] A. R. Oskooei, S. Yukcu, M. C. Bozoglan, and M. S. Aktas (2025) Repository-level code understanding by llms via hierarchical summarization: improving code search and bug localization. In International Conference on Computational Science and Its Applications, pp. 88–105.

[36] G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar (2023) Voyager: an open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291.

Appendix Contents

Appendix A Harness Handbook Construction Details

This appendix expands the three construction phases in Section 3.2. The selected leaf mode g and construction configuration Θ stay fixed through construction and later resynchronization. Both modes work from the same static facts but organize them differently: function-as-leaf starts from a trusted seed skeleton \mathcal{S}_{0} and places functions within it, whereas file-as-leaf describes files first and infers the skeleton. Both ultimately produce the same reader-facing handbook—an L1–L3 document tree, a cross-stage state-register view, and source locators for L3 entries. Algorithm 2 summarizes the shared workflow and its mode-specific branches.

Algorithm 2 Harness Handbook Construction

A.1 Phase I: Shared Static Fact Extraction

Phase I is a single deterministic analysis shared by both modes. Language-specific adapters parse \mathcal{R} and build the program graph \mathcal{G}. An internal-function node records a function’s qualified name, source file, signature, line range, enclosing class, and observed state access; a boundary node represents a named external target, such as a library function.

The graph keeps a call edge only when the target resolves to an internal function or a named boundary; unresolved calls are written to an audit log rather than assigned a guessed target. Because this phase uses no LLM, every later organization step builds on the same source-grounded facts.

A.2 Phase II: Behavioral Organization

Phase II organizes the facts in \mathcal{G} around execution behavior. Its output is the current stage skeleton \mathcal{S} together with the mode-specific organization state \mathcal{U}_{g}. The two leaf modes differ in both their starting point and their unit of organization.

A.2.1 Function-as-Leaf

Initial assignment.

The function-as-leaf branch (g=function) starts from the seed skeleton \mathcal{S}_{0}, whose stage and state-register declarations initialize \mathcal{S}. On the first pass, it considers every analyzable internal function—that is, every non-synthetic function with usable source locations—and attempts to assign it to one or more stages. The decision uses the function’s source in \mathcal{R}, its callers and callees in \mathcal{G}, the current neighboring assignments, and the stage descriptions. A function is normally placed as a whole. If different parts serve different behavioral roles, it can instead be divided into contiguous source regions and each region placed se...