--- title: 'The anatomy of a production multi-agent platform' excerpt: 'A production-ready multi-agent architecture: a cheap router up front, expensive specialists behind it, safety in code, and a bypass for the common case.' date: '2026-06-15' lastModified: '2026-07-04' author: 'Teo Deleanu' authorAvatar: '/team/teo.jpg' tags: ['Production AI', 'AI Engineering', 'Agents', 'Multi-agent'] keywords: ['multi-agent architecture', 'orchestrator specialist pattern', 'LLM routing economics', 'production AI agents'] featured: false tldr: 'An analytics platform we built for an IoT product runs a front-desk router on a mid-tier model (Claude Sonnet 4.6) in front of two specialists on a top-tier model (Claude Opus 4.8). The orchestrator splits and routes; it does not manage, because manager patterns produce chatty token-burning loops. Single-domain questions, the common case, skip synthesis entirely and cost one specialist call. Safety and cost control live in code: tenant IDs injected into SQL via AST manipulation, and Langfuse tracing with per-task cost attribution.' keyTakeaways: - 'Put a mid-tier model at the front desk and top-tier models behind it: the router fires on every request, specialists only on demand, so the price asymmetry compounds' - 'The orchestrator should split and route, never supervise; management patterns burn tokens in loops that add nothing to the answer' - 'Design the common path first: single-domain questions bypass synthesis and cost one routing decision plus one specialist call' - 'Enforce tenant isolation in code (sqlglot AST injection) and trace per task, or you cannot attribute cost or blame across agents' --- ByteByteGo has a solid conceptual tour of multi-agent patterns. This is the version from a system we run: an analytics platform we built for an IoT product, in production, with real tenants and a real bill. One shape held up. A front-desk orchestrator on a mid-tier model splits multi-domain questions, routes the pieces to specialists on a top-tier model, and synthesizes the results. Manager-style alternatives burn tokens without improving the answer. Two claims carry this post: the orchestrator should be a router rather than a manager, and most questions should never pay the multi-agent tax at all. ![Orchestrator-specialist architecture: a front-desk router on a mid-tier model routes to top-tier specialists, with a single-domain bypass that skips synthesis](/blog/multi-agent-anatomy.svg) ## A cheap model at the front desk The front desk runs Claude Sonnet 4.6. Its whole job is to read the question, decide which domains it touches, split it if it touches more than one, and dispatch the pieces. Behind it sit two specialists on Claude Opus 4.8: an Operational agent and an Analytical agent, the latter turning questions into SQL over telemetry. The price asymmetry is the point. The router fires on every request. The specialists fire on demand. Put the expensive model at the front desk and you pay top-tier rates for classification, the least interesting task in the pipeline, on 100% of traffic. Put it in the back and the expensive tokens only flow when a question needs deep reasoning. Classification and decomposition sit comfortably inside a mid-tier model's competence; tool-use reasoning over a live database is where the top-tier model earns its rate. ## Routing beats managing The word "orchestrator" invites the wrong design. Most frameworks ship a manager pattern: an agent that plans, delegates, reviews worker output, requests revisions, and loops until satisfied. That produces a predictable failure mode, chatty token-burning loops where agents negotiate with each other and the transcript grows faster than the answer improves. Nobody reads those transcripts. Everybody pays for them. Our front desk does exactly three things: split, route, and synthesize. It never critiques a specialist's answer or asks for a revision. It holds no opinion on how the specialists should do their jobs. If two specialists ran, it merges their answers into one response, and that is the entire management layer. Synthesis is also conditional, and this matters more than any model choice. When a question touches a single domain, the front desk routes it to one specialist and returns that answer directly. No merge pass, no second model call on the way out. Zero synthesis tax for the common case. Design that path first: most questions are single-domain, so "how many devices reported errors last week" should cost one routing decision plus one Opus call and nothing else. The multi-agent machinery exists for the minority of questions that genuinely span domains, and only those pay for it. ## One tool and a catalog, instead of 200 definitions Specialists reach the platform through tools, and tool definitions ride in the context window on every turn. Our earlier version carried 200+ hardcoded tool definitions, a tax on every single call. We replaced them with one generic tool plus a compressed catalog of available operations that fits in roughly 9k tokens. The model reads the catalog, picks an operation, and invokes the generic tool with the operation name and arguments. Per-turn context cost dropped, and adding an operation became a catalog entry instead of a schema deployment. The cost math behind this catalog diet is in [an earlier post in this series](/blog/free-because-open-source). ## Safety guarantees belong in code The Analytical specialist generates SQL from natural language, in a multi-tenant database. This is precisely where prompt-level safety falls apart. "Always filter by the tenant's application_id" in a system prompt is a request, and a model under adversarial input can decline requests. The prompt says don't. That is a suggestion, and a security boundary has to be stronger than a suggestion. So we stopped asking. Every generated query is parsed with sqlglot, and the tenant's application_id predicate is injected into the AST before execution. Whatever SQL the model writes, the query that reaches the database is scoped to one tenant. Cross-tenant leakage is impossible by construction. The general rule we took from this: if a property matters, enforce it in code where the model cannot vote on it. ## The plumbing that makes it operable Three pieces do the unglamorous work. Checkpointing. Agent runs checkpoint through AsyncPostgresSaver, so a run that dies mid-flight resumes from its last saved state instead of replaying every model call from the start. In a system where one user question can fan out to multiple expensive specialist runs, replay is real money. Model routing. Every model call goes through LiteLLM with a per-task primary and a fallback chain. We wrote this up as an ADR: one adapter, no per-provider SDKs scattered through the codebase. When a provider degrades, the fallback fires; when we want to swap a model for one task, it is a config change rather than a refactor. Tracing. Langfuse traces every request end-to-end with per-task cost attribution. In a single-agent system you can eyeball the bill. Across a router and multiple specialists you cannot, and without per-task attribution you can attribute neither cost nor blame. When Analytical spend moves, we can see which task class moved it and whether the router or the specialist misbehaved. ## Where this fits in the series The routing economics here connect directly to [evals as CI](/blog/evals-as-ci): a router you cannot regression-test is a router you cannot trust after a model swap. The platform behind this post is described with numbers in [the operator brief](/about), and the [live demo](/demo) shows the single-agent end of the same discipline if you want something to poke. The pattern is not exotic. A cheap router, expensive specialists, a bypass for the common case, and guarantees in code. That is the whole anatomy.