Your AI Agents Finished the Task. Which Ones Actually Helped?

Your AI Agents Finished the Task. Which Ones Actually Helped?

You built a team of six AI agents.

One researches. One plans. One writes code. One reviews it. One runs tests. Another coordinates the workflow.

The demo looks impressive.

Then someone asks a question your dashboard cannot answer:

If we removed half the agents, would the result actually get worse?

Your traces show tool calls. Your dashboard reports latency. Your billing system counts tokens.

But which agent improved the outcome enough to justify its cost?

That is the question behind AgentLedger: a proposed contribution-analysis and team-optimization layer for multi-agent applications.

Not another framework for creating agents.

A system for deciding which agents deserve to run, on which tasks, and under what conditions.

Product status: This article describes a proposed architecture and MVP—not a released product, validated benchmark, or demonstrated cost-saving result.

Observability Is Not Contribution Analysis

Consider a hypothetical bug-fixing workflow:

Researcher → Planner → Coder → Reviewer → Tester → Coordinator 

The patch passes the test suite.

From the trace, you can reconstruct what happened:

• The Researcher retrieved documentation.
• The Planner proposed an implementation.
• The Coder produced a patch.
• The Reviewer suggested changes.
• The Tester executed tests.
• The Coordinator delivered the result.

What the trace alone does not establish is what mattered.

Perhaps the retrieved documentation was already available in the context. Perhaps the Reviewer suggested cosmetic changes. Perhaps the Tester caught the only serious defect.

Or perhaps the Reviewer prevented a failure that the visible tests would never have detected.

Activity is observable. Contribution requires an evaluation design.

AgentLedger would connect those two layers.

The Product in One Sentence

AgentLedger would run controlled variations of an agent workflow, evaluate their outcomes, and estimate the incremental value of individual agents and agent combinations.

It would answer four operational questions:

1. Which agents improve outcomes?
2. Which agents duplicate one another?
3. Which agents become valuable only when paired with others?
4. Which team configuration offers the best quality–cost–latency trade-off for a given task class?

The intended output is not a leaderboard.

It is an actionable, evidence-qualified recommendation:

“For low-risk documentation changes, test a workflow without the general reviewer. Retain security review for changes involving authentication, permissions, or external input.”

The qualification matters. An agent can be unnecessary for one task class and essential for another.

Start With the Counterfactual

The basic experiment is straightforward:

Run the task with an agent, then run a comparable version without it.

 ┌─ Full team ──────────── Evaluate │ Task + Environment ─┼─ Without Reviewer ───── Evaluate │ ├─ Without Researcher ── Evaluate │ └─ Other combinations ── Evaluate 

This produces a useful first approximation: leave-one-out ablation.

However, it does not fully capture interactions.

Imagine two agents that can independently retrieve the same critical document. Removing either one may have almost no effect because the other provides a substitute.

Now imagine a Researcher that finds a vulnerability and a Coder that fixes it. Neither may produce much value alone, while the pair substantially improves the outcome.

A useful attribution system must examine combinations—not just isolated removals.

Where Shapley Values Fit

The proposed attribution engine would use Shapley-style contribution estimates.

Let:

• (N) be the set of agents.
• (S) be a subset of those agents.
• (v(S)) be the expected utility of a precisely defined workflow using (S).

The Shapley value for agent (i) is:

[
\phi_i =
\sum_{S \subseteq N \setminus {i}}
\frac{|S|!(|N|-|S|-1)!}{|N|!}
\left[v(S \cup {i}) - v(S)\right]
]

Operationally, this averages the agent’s marginal contribution across different teammate combinations.

The crucial phrase is “precisely defined workflow.”

An agent subset is not executable by itself. We must specify how the system runs when particular agents are absent.

The mathematics cannot repair an ambiguous experiment.

Attribution Is Not a Fairness Certificate

A contribution score is relative to the chosen utility function, task distribution, and execution protocol.

It does not establish:

• Moral deservingness.
• Universal agent quality.
• Fair treatment of people.
• A context-independent causal effect.

If the evaluation rewards the wrong outcome, a sophisticated attribution method will explain the wrong objective with greater precision.

That is a product-design problem, not a missing formula.

The Hardest Question: What Does “Remove an Agent” Mean?

Suppose we remove the Planner.

Does the Coder receive the original task directly? Does the Coordinator create a replacement plan? Does the workflow stop because a required artifact is missing?

Each choice defines a different experiment.

AgentLedger should distinguish at least two modes.

Mode 1: Role Ablation

Remove a role without reallocating its responsibilities.

Question:

How dependent is this workflow on this role?

A failure may reveal a structural dependency rather than a uniquely capable agent.

Mode 2: Team Reconfiguration

Remove an agent and allow a predefined policy to redistribute its responsibilities.

Question:

Can a smaller team achieve comparable results?

This is often closer to the production optimization question, but the redistribution policy becomes part of the evaluated system.

Scores from these modes should not be mixed into one ranking.

Define Value Before Measuring It

For a coding workflow, an initial utility function might be:

[
v(S) =
\mathbb{E}
\left[
Q(S) - \lambda C(S) - \mu T(S)
\right]
]

Where:

• (Q(S)) measures task quality.
• (C(S)) measures execution cost.
• (T(S)) measures latency.
• (\lambda) and (\mu) express explicit trade-offs.

The quantities require appropriate scaling, and the weights require justification.

More importantly, some requirements should be constraints rather than rewards.

For example:

Maximize quality-adjusted efficiency Subject to: - No unauthorized external actions - No forbidden data access - Required security checks pass - Minimum acceptance criteria are satisfied 

A cheaper workflow should not receive a better recommendation simply because the scoring system lets cost savings compensate for a critical security failure.

AgentLedger should also report quality, cost, and latency separately. A single score is useful for optimization, but insufficient for inspection.

A Behavioral Lens—Without Anthropomorphizing Agents

A second design principle is to evaluate agents in context.

Rather than describing an agent as “stressed,” “dominant,” or “afraid to disagree,” measure the engineering conditions that might change its behavior:

These interventions generate testable questions:

• Does independent review detect more defects than review performed after seeing the coordinator’s preferred answer?
• Does a tighter planning budget reduce unnecessary steps without reducing task success?
• Does one agent’s early answer disproportionately shape later outputs?

These are hypotheses to investigate—not built-in claims about how language models behave.

The goal is not to diagnose artificial personalities. It is to design better collaboration environments.

Proposed Architecture

Agent Runtime │ ▼ Trace and Artifact Collector │ ▼ Experiment Specification │ ▼ Replay and Reconfiguration Runner │ ├── Sandboxed Tools ├── Coalition Sampler └── Repeated Trials │ ▼ Independent Evaluation │ ▼ Contribution Estimates │ ▼ Report and Candidate Routing Policy 

1. Capture Reproducible Evidence

Record more than messages:

• Model identifiers and settings.
• Prompt and workflow versions.
• Input and output artifacts.
• Tool requests and responses.
• Environment snapshots where practical.
• Cost and timing measurements.
• Evaluation configuration.

The objective is not perfect determinism. It is sufficient reproducibility to distinguish a workflow change from unrelated execution variation.

2. Replay Safely

An experimental replay must not resend a customer email, repeat a payment, or modify production data.

Side-effecting tools should be mocked, sandboxed, or blocked.

For dynamic tools, the replay policy must specify whether to use recorded responses or fresh sandbox execution. Those choices answer different questions and belong in the report.

3. Evaluate Independently

For the coding MVP, evaluation could combine:

• Held-out functional tests.
• Regression tests.
• Security checks.
• Explicit task acceptance criteria.
• Human inspection for selected cases.

A model-based judge could supplement these signals, but should not silently become the sole definition of success.

4. Estimate Contribution With Uncertainty

Exact evaluation across all subsets becomes expensive as team size grows.

The proposed engine would therefore begin with inexpensive ablations and expand into sampled agent orderings when the additional evidence is worthwhile.

Reports should distinguish:

• Variation between repeated executions.
• Uncertainty from sampling combinations.
• Variation across tasks.

An apparently negative contribution with wide uncertainty is a reason to investigate—not automatically disable an agent.

What the Report Should Actually Say

Avoid a seductive but misleading output:

Reviewer: 12% valuable Researcher: 8% valuable 

Prefer an evidence-qualified report:

Task class: Low-risk documentation updates Finding: Removing the general reviewer is a candidate optimization. Evidence: Summarize paired evaluation results, cost changes, quality changes, and uncertainty. Scope: Applies only to the tested workflow and task distribution. Exclusions: Security-sensitive changes and unfamiliar repositories. Next action: Run a shadow evaluation before changing production routing. 

Signed contribution estimates should also retain their units. They should not be forced into percentages that imply every agent owns a positive share of the result.

The MVP Should Be Deliberately Narrow

The first version should target small coding teams operating in sandboxed repositories.

Three to five roles are enough:

Planner → Coder → Reviewer → Tester 

Initial capabilities:

1. Import execution traces and artifacts.
2. Run predefined removal experiments.
3. Evaluate a small set of team combinations.
4. Compare quality, cost, and latency.
5. Produce a report with scope and uncertainty.

Explicit non-goals:

• Autonomous production deployment.
• Universal rankings of agent models.
• Automated financial rewards for agents.
• Claims of fairness based solely on attribution.
• One-click removal of “low-value” agents.

The first milestone is modest but meaningful:

Identify one repeatable workflow change that reduces resource use without an unacceptable loss of quality on a held-out task set.

Where the Product Could Become Defensible

The formula alone is not the moat.

Potential defensibility would come from:

• Reliable replay infrastructure.
• High-quality task-specific evaluations.
• Historical evidence about agent interactions.
• Efficient experiment selection.
• Integrations with existing runtimes.
• Safe conversion of findings into routing policies.

Over time, the system could learn a more useful mapping than “best agent”:

Task characteristics ↓ Recommended team configuration ↓ Expected trade-offs and uncertainty 

That turns attribution into an operational decision layer.

A Plausible Business Model

An open-source core could provide local trace ingestion, experiment definitions, and basic ablations.

A hosted product could add:

• Scheduled evaluations.
• Model and prompt regression analysis.
• Team access controls.
• Private trace storage.
• Experiment budgets.
• Cross-project reporting.
• Approval-gated routing recommendations.

The customer would not be buying another place to read agent conversations.

They would be buying evidence for changing the system.

The Next Breakthrough Might Be Subtraction

It is easy to make a multi-agent architecture look more sophisticated by adding another role.

It is harder to demonstrate that the role improves the outcome.

AgentLedger starts with a different assumption:

Every additional agent should be a testable design decision—not a permanent architectural commitment.

Before adding agent number seven, ask a more valuable question:

What evidence would justify keeping the first six?
Author:Seyed Alireza Alhosseini Almodarresieh
my site:https://almodarresieh.github.io/