Multi Agent Systems Fail in the Handoffs

The appeal is obvious. One agent that researches, one that writes, one that reviews. Specialisation, separation of concerns, the same instincts that make microservices attractive.

In practice the systems I have built and seen fail in a consistent place: not inside any agent, but in what passes between them.

The three handoff failures

1. Lossy summarisation

Agent A does the research and produces a summary for agent B. The summary is where the detail dies.

A read six documents and formed a nuanced view including caveats, source quality, and one contradiction it noticed. The summary says "the recommended approach is X". B acts on X with complete confidence, unaware that A rated its own conclusion as uncertain.

Confidence is the first thing lost in a handoff, and it is the thing B most needs.

The fix is to make uncertainty structural rather than prose:

{
  "recommendation": "use approach X",
  "confidence": "low",
  "conflicting_evidence": ["source 3 contradicts, published more recently"],
  "assumptions": ["assumes Postgres 16+", "assumes single region"],
  "sources": ["doc-1#section-4", "doc-3#section-2"]
}

B can now branch on confidence, and a human reviewing the trace can see what was actually known.

2. Error swallowing at the boundary

A fails partially, produces something plausible anyway, and B has no way to tell.

This is the swallowed exception pattern with a model in the middle. Models are strongly inclined to produce an answer rather than report failure, so partial failure becomes confident output by default.

Every handoff needs an explicit status, and the consumer needs to check it:

{ "status": "partial", "completed": ["search", "extract"], "failed": ["verify"], "data": {...} }

And the orchestrator, not the next agent, decides what to do with a partial result. Letting B decide whether A's failure matters puts the judgement in the wrong place.

3. Instruction leakage

A's output becomes B's input. If A processed untrusted content, an injected instruction can ride along into B's context.

This is the amplification problem in prompt injection. Every agent to agent boundary is a place where untrusted content crosses into a new context, potentially one with more privileges.

Treat inter-agent data as untrusted. Structured fields rather than free text, and never place another agent's raw output where your system prompt goes.

Why more agents usually makes it worse

The compounding is unforgiving. If each agent is 95 percent reliable on its step, a five step chain is 77 percent. Ten steps is 60 percent.

And the errors are not independent. A wrong assumption early gets elaborated rather than corrected, because each subsequent agent treats its input as given. By step four the system is confidently building on something that was a guess at step one.

Human teams handle this through disagreement. Someone says "wait, is that actually true?" Agents in a pipeline do not, because each one's job is to act on its input.

The practical implication: fewer agents with more context usually beats more agents with less. A single agent that can see the whole problem catches its own inconsistencies in a way a pipeline cannot.

I would default to one agent, and split only when there is a concrete reason: a genuine tool or permission boundary, a context window limit, or a need to run steps in parallel.

When separation is genuinely right

Privilege boundaries. A researcher that reads untrusted web content and a writer that has repository write access should be separate, specifically so the untrusted content never enters the privileged context. This is the strongest reason and it is a security architecture decision rather than a quality one.

Parallelism. Six independent lookups genuinely run faster in parallel. Note this is a fan-out with a join, not a chain, and fan-out is much safer because errors do not compound.

Context limits. A task that genuinely does not fit. Worth checking honestly, because the limit is often smaller in practice than the advertised window due to degradation over long sessions.

Genuinely different models. A cheap fast model for classification, an expensive one for the hard reasoning step. Real cost savings.

Notice that "specialisation improves quality" is not on this list. It is the most commonly cited reason and the one I have found least supported. A specialised prompt helps. A specialised agent, with its own context and its own handoff, mostly adds boundaries where information gets lost.

Making it debuggable

If you do build a pipeline, the difference between one you can operate and one you cannot is entirely instrumentation.

Log every handoff in full. Input, output, model, prompt version, token counts, duration, and a trace id linking the whole run. Storage is cheap and this is the only way to answer "where did this go wrong".

Trace the whole run as one unit. Individual agent logs are nearly useless. You need the sequence.

Validate at every boundary with a schema. A malformed handoff should fail loudly at the boundary rather than being interpreted generously by the next agent:

const result = HandoffSchema.safeParse(agentOutput);
if (!result.success) {
  metrics.increment("handoff.invalid", { from: "researcher", to: "writer" });
  throw new HandoffError(result.error);
}

Instrument each step's success rate separately. When end to end quality drops, you need to know which step. Without per step metrics you are guessing.

Cap the total work. Steps, tokens, wall clock time, and cost. Agents delegating to agents can loop, and a runaway pipeline running overnight is a real and expensive failure mode.

Make it replayable. Storing inputs at each step means you can rerun from step three with a fixed prompt rather than rerunning the whole chain. This is the difference between a five minute iteration and a twenty minute one.

The evaluation problem

The hardest part is that end to end quality is what matters and per step quality is what you can measure.

A pipeline where every step scores well individually can produce bad output, because the steps are optimised against their own inputs rather than against the final goal.

The only approach I have found that works: evaluate end to end on a fixed set of real cases, and use per step metrics only for diagnosis. If end to end quality drops, look at step metrics to find where. Do not optimise step metrics on their own, because you will improve a number and not the outcome.

That is the same discipline as testing LLM features generally: the eval set is the ground truth, and everything else is a diagnostic.

The summary I would give

Start with one agent. Add a second only when you can name the boundary and why it must exist.

For every boundary you do add, define the contract explicitly, validate it, log both sides, and decide who owns partial failure.

The systems that work are the ones where the handoffs are engineered as carefully as an API between services, because that is what they are. The ones that fail are the ones where the handoff is a paragraph of prose passed between two prompts and hoped for.