A multi-agent run can pass a check on each agent and still ship a bad result, because a downstream agent can consume an upstream output, including when handoffs branch or run in parallel. Log the handoff, block a payload that fails the receiver's schema, and run the agent in a simulated environment before it may finish an action you would not want undone. At the AI Engineer World's Fair, Mikiko Chandrasekhar saw about 80% of the room raise a hand for building an agent, about the same people (maybe one or two fewer) for agents in production, and no hands at all for agents that worked 100% of the time on the first try.
Chandrasekhar, then a staff developer advocate at MongoDB, spoke in July 2025, the same month as Giovanna Carofiglio of Outshift by Cisco. Kris Lovejoy, global head of strategy at Kyndryl, spoke in June 2026. The useful unit across those conversations is the path between agents: trace it, score each agent on its own job, and constrain what it is allowed to finish.
Why does a working agent still produce a broken system?
A local miss becomes a system failure once the next agent treats that output as fact. A structured workflow can still return a different answer, Chandrasekhar said: one out of 100 runs, or one out of 20 runs when the task is more complex. The stakes she named run from an internal financial report a public company cannot get wrong, through triage for severe health conditions and underwriting for loans or insurance, where the consequences can be disastrous, to summer reading lists from reputable sites that named books that did not exist and took a reputation hit because nobody checked.
A paper she called "why multi agent systems fail" tries to taxonomize how these systems break. Her reading was that a lot of these failures are predictable enough to classify, and that techniques for making a single agent better have to be extended to a system of agents. The split she recommended is small agents, each equipped for one to three tasks, each with criteria for its own performance, then a look at the group together.
If you don't have a way of observing what these agents are doing, if you don't have a way of evaluating them, and if you don't have a way of putting guardrails around their behavior, you just don't have an agentic system that's gonna really survive.
Mikiko Chandrasekhar, Staff Developer Advocate at MongoDB, on Chain of Thought ep 33
Mastering Multi-Agent Systems … - Chain of Thought | AI Agents, Infrastructure & Engineering - Apple Podcasts
Podcast Episode · Chain of Thought | AI Agents, Infrastructure & Engineering · July 23, 2025 · 40m
podcasts.apple.com How do you check a handoff before the next agent acts?
Record the sender, the receiver, and the payload, then refuse the call if the payload does not match the schema that receiver is allowed to act on. A missing field is dropped context. A field that is present with the wrong type is bad input treated as fact. The check catches both.
Carofiglio wants that path visible: how agents communicate, how data passes from one agent to another, and how tools are called. That visibility serves both the developer and the team running production. Similar inputs should produce consistent output. You learn that by testing the space of solutions.
A minimal sketch to adapt, not a drop-in library.
class HandoffRejected(Exception): def __init__(self, receiver): super().__init__(receiver) def trace_span(name, fields): return {"name": name, "fields": fields} def schema_ok(payload, schema): # schema maps each field the receiver needs to its expected type return isinstance(payload, dict) and all( isinstance(payload.get(field), kind) for field, kind in schema.items() ) def call_agent(receiver, payload): return {"receiver": receiver, "payload": payload} def handoff(sender, receiver, payload, schema): trace_span( "agent_handoff", {"from": sender, "to": receiver, "payload": payload}, ) if not schema_ok(payload, schema): raise HandoffRejected(receiver) return call_agent(receiver, payload) That means that we want to explore by testing all the possible state of solution or space of solutions that these agents will work in and try to characterize patterns, make sure the output is consistent where the agents are provided with similar input.
Giovanna Carofiglio of Outshift by Cisco, on Chain of Thought ep 31
How do you limit autonomy before release?
Lovejoy treats agentic AI on mission-critical systems as a privileged user. Protection, on her account, depends on visibility. The sequence she listed: inventory the agent, evaluate it in simulation so it is not performing activities you do not want, generate policies and monitor them, keep audit and compliance reports, and put humans in supervisory roles.
The model she questioned is human in the loop. She asks whether it should be human on top instead, and she favors secondary evaluations and approvals before the agent can complete the action. These systems run tasks in parallel, and a person on every branch can stall the work. Policy as code stays one element of the architecture. Pasting deterministic rules onto an autonomous system is the other move she questioned. When those rules miss a case, an orchestration agent and many smaller, outcome-oriented agents keep performing their tasks and can take systems down, because they did not understand how those guardrails worked. She pointed to a recent car-rental case as an example.
We have to evaluate the agent within simulated environments to ensure that that agent is not going to be performing activities that we don't want it to perform.
Kris Lovejoy, Global Head of Strategy at Kyndryl, on Chain of Thought ep 62
FAQ
Should every agent share one scorecard?
Chandrasekhar said each agent should have criteria focused on its specific skill set. A single shared score hides which specialist drifted. Keep a separate check for the group so you can still see a bad handoff that every local score missed. For coding agents she named accuracy, runtime efficiency, and the number of times someone accepts a suggestion, and she agreed that lines of code is a terrible measure.
Does a schema check replace evaluation?
A schema check rejects a bad payload at one handoff. You still need observation, a score for each agent, and a limit on behavior. Lovejoy's pre-release step is a simulated environment, so the agent is exercised against activities you do not want before it faces real systems.
Where should the human sit?
Lovejoy argued for humans in supervisory roles, with secondary evaluations and approvals before an agent completes an action. She floated "human on top" as the alternative to human in the loop. A person placed in the middle of parallel work can stall that work.
Takeaway
- Record sender, receiver, and payload on every agent-to-agent handoff.
- Reject the handoff when the payload fails the receiver's schema.
- Keep each agent to a small task set (Chandrasekhar suggested one to three) and score it on its own criteria.
- Run the agent in a simulated environment before you widen what it can finish on its own.
- Keep an inventory, monitored policies, and an audit trail for every agent that acts as a privileged user.
The longer explainer, with the episode clips, is on Chain of Thought. It draws on this episode.
Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.
Drafted with AI assistance from the episode transcripts.
