A single agent with a handful of tools can do a surprising amount. It reads a request, calls a search tool, calls a calculator, and writes an answer. The trouble starts when the work outgrows one context window, one permission set, or one way of failing.

Orchestration is the layer that decides who does what, with which information, under which limits, and what happens when a step goes wrong. It is less about clever prompting and more about the control plane around the models.

A working definition

Anthropic’s engineering guide Building effective agents draws a useful line. Workflows run language models and tools through predefined code paths. Agents let the model direct its own process and tool use. Most production systems sit somewhere between the two, and orchestration is how you choose where.

In practice, an orchestrator is responsible for five things:

  1. Routing: deciding which handler, model, or agent should take a request.
  2. Delegation: turning a large task into scoped subtasks with clear outputs.
  3. State: deciding what each participant can see and what it writes back.
  4. Recovery: handling timeouts, bad outputs, and partial failure.
  5. Authority: knowing which steps need a person before they take effect.

A single agent does all of this implicitly inside one loop. An orchestrated system makes each decision explicit, testable, and observable.

A grounded example

Consider an internal assistant that answers vendor questions for a finance team. A request arrives: “Can we renew the analytics contract, and what will it cost next year?”

A single agent could try to handle everything. An orchestrated version might look like this:

  • A router classifies the request as a contract question with a cost component. This can be a small model or plain rules; it does not need to be the most capable model in the system.
  • A contract worker retrieves the agreement and summarizes renewal terms. It can read the document store and nothing else.
  • A spend worker queries approved finance views for current and historical cost. It has read-only, row-scoped database access.
  • The orchestrator merges both results, checks that every figure traces back to a record, and drafts an answer.
  • If the draft recommends signing anything, the orchestrator stops and routes it to a person.

Nothing here is exotic. The value is that each worker has a narrow job, a narrow permission set, and a defined output, so a failure in one can be seen and handled without corrupting the others.

Routing and tool selection

Routing is the cheapest place to buy reliability. A misrouted request wastes every step after it.

Good routers are boring. They classify by intent, required data source, and required authority, and they have an explicit path for “unclear”, usually a clarifying question. When a route is chosen, the receiving agent should see only the tools relevant to that route. Long tool lists increase the chance of a plausible but wrong call, and descriptions of tools you never intended to use still consume context.

Shared versus isolated state

The most consequential design choice in multi-agent systems is what each participant can see.

Shared state, such as one conversation history visible to all agents, is easy to build and easy to debug in a demo. It also lets one agent’s mistake or injected instruction propagate to every other agent.

Isolated state gives each worker a fresh context containing only its brief: objective, inputs, tools, and required output format. Workers return structured results; the orchestrator decides what to keep. Anthropic’s write-up of its multi-agent research system describes this pattern: a lead agent decomposes a query and gives each subagent an objective, an output format, and guidance on tools and sources.

Isolation costs some redundancy. It buys containment, smaller contexts, and results you can validate at the boundary.

Errors, retries, and timeouts

Every step in an orchestrated system needs three numbers and one rule:

  • a timeout, after which the step is treated as failed;
  • a retry budget, small and usually with backoff;
  • a total budget for time, tokens, or cost across the whole task; and
  • a rule for whether the step is safe to repeat.

Reads are generally safe to retry. Writes are not, unless they carry an idempotency key and the orchestrator verifies the resulting state before trying again. A retried payment or a duplicated message is a business incident, not a transient error.

When a worker fails for good, the orchestrator should degrade explicitly: answer the part it can support, state what is missing, and avoid filling the gap with generated text.

Observability

If you cannot reconstruct why the system did something, you cannot improve it or defend it. At minimum, log for every task:

  • the route chosen and why;
  • each delegated brief and the structured result it returned;
  • every tool call with arguments, latency, and outcome;
  • retries, timeouts, and budget consumption; and
  • approval requests and who resolved them.

Agent frameworks increasingly ship tracing for this; the OpenAI Agents SDK, for example, documents handoffs between agents, guardrails, and built-in tracing. Whatever the tooling, the trace should be readable by the engineer who has to explain an incident.

Human approval boundaries

Approval should attach to effects, not to agents. “The finance agent needs approval” is vague. “Any action that commits spend, sends external communication, or changes a contract record requires approval of the exact payload” is enforceable.

The orchestrator is the natural place to enforce this, because it sees the proposed action before it executes. The approval request should show what will change, why, and the evidence behind it.

When you do not need multiple agents

Multi-agent architectures have real costs. In the same research-system write-up, Anthropic reports that in its data, agents used about four times more tokens than chat interactions, and multi-agent systems about fifteen times more. That trade can be worthwhile for broad, parallelizable work. It is wasteful for tasks a single well-scoped agent or a fixed workflow can handle.

A reasonable test before adding agents:

Question If yes
Can a fixed sequence of steps solve it? Use a workflow, not agents
Does one context window comfortably hold the task? Start with a single agent
Are the subtasks independent and parallel? Delegation may pay off
Do subtasks need different permissions or data? Isolate them as separate workers
Would a failure in one part corrupt the rest? Isolate state between workers

Orchestration is worth its complexity when it creates boundaries you actually need: different authority, different data, parallel work, or contained failure. Otherwise, the simplest system that solves the problem is usually the most reliable one.