Table of contents
How to Design Reliable Multi-Agent AI Systems for Production
Introduction
The potential of multi-agent AI is very exciting, with autonomous, specialized components working together to accomplish tasks that a single model cannot complete. The demos create a view of the future, while the production feels like a 2 a.m. support ticket for a lot of the time.
The cause of issues in production is not bad models. The production systems have all been developed as prototypes rather than being developed as distributed systems. As such, agents have done what they are programmed to do, however, the architecture is the area that fails.
This document can be used as a tool to assist engineers that want to develop a system that is beyond the proof of concept. If an engineer requires a solid foundation for understanding how agents think and perform actions, the AI Training Course is an excellent place to begin, covering information that relates to chain of thought, use of tools, and agent behaviour and improving the design thought process.
This document is comprised of the following: Ten steps of design journey, three examples of failures within the real-world, and a check list for production. The goal of this document is to assist engineers with building the most reliable systems, rather than the most technologically sophisticated systems.
What Is a Multi-Agent AI System (In Production Terms)?
A smart LLM prompt is what many people think of when they hear about an AI agent, but that’s not the case in actual production where there are multiple parts making up an agent and running continuously through a cycle:
The boundaries of functionality for traditional LLMs (Large Language Models) are defined by a single context (i.e. input), one API call, one output. A multi-agent system (MAS) is a collection of AI agents that act collaboratively toward task accomplishment beyond the limitations of a single API call due to task dimensions or because multiple API calls must be executed sequentially.
These are the basic roles within a MAS.
- Planner: Defines sub-goals to be completed sequentially to achieve high-level goal(s).
- Executor: Executes sub-goal API calls utilizing tools and/or API calls.
- Verifier: Confirms validity of completed outputs before passing to downstream agents.
- Memory: Responsible for tracking scoped (local) & shared (global) context across all agents in a MAS.
The primary discovery regarding MAS is that they represent a type of distributed software system that is subject to the challenges of coordinated efforts, state consistency, failed propagation, etc., as well as having to build in/bake-in retry logic, timeouts, and circuit-breaking into every component of the MAS. Engineers who expect a MAS to operate like a large chatbot learn this lesson the hard way.
Why Most Multi-Agent Systems Fail in Production
Demos have everything put together for success because they have good data going in, they have stable functions to call, and they have no overlapping calls occurring at the same time. Yet, none of these conditions exist in a real production environment. Knowing the possible failure cases ahead of time before you build is the quickest way to avoid them; if you’d like more detail on each of those possible failure modes check out Why AI Agents Fail in Production for the major causes with real-world situations.
There are three key reasons for incidents in production environments. They are:
Assumptions Made at Design Time Become Failures at Runtime
Building prototypes or tests rely on a set of implicit or silent agreements/assumptions in relation to the architecture – for example, when the planner will provide a valid JSON response back; or that the response time of the tooling being used will be less than two seconds; and that the context will remain immutable across parallel executions. All of these assumptions create a dependency, and at some point in production all dependencies will fail.
Example of Issue: A team used their planner agent to assume that there would always be a valid JSON action plan returned to them when sending document-processing pipeline requests. In their staging environment, it did. However, when they moved to a production environment they encountered a case of invalid data – one edge case that returned an incomplete/partial JSON object. As a result, when the executor attempted to parse the action plan given the invalid response, it could not succeed – and because there was no schema validation check in place during the data interchange between the planner and executor, the executor completed the request/resubmitted twelve times (burning through API credits without any output). Adding a single schema validation check when sending requests from the planner to the executor would have completely eliminated this issue.
No Verifier Means Every Bad Output Propagates
Without permitting a dedicated verification step, a hallucinated tool call, malformed output, or out-of-scope action passes directly to the next phase. With systems that deal with real data or trigger real actions, this issue isn’t simply a reliability issue, it is also a safety issue.
An illustrative example: An internal HR automation agent was requested to generate amendments to contracts which are routed to the correct approver queue. The executor, working with an ambiguous employee record, hallucinated a clause that did not exist in the source document. There was no verifier between drafting and routing phases of the process which allowed the fictitious clause to be included in the approver queue. The following human reviewer identified the error but after the document had been sent. A verifier comparing the draft with the source record would have identified the discrepancy prior to routing the draft.
Design Failures, Not Model Failures
The change in perspective that really counts is that the models are generally going to do what they are told to do. The actual problem lies in the architecture that surrounds them – there are no backup plans when the tools do not work and there are no circuit breakers to stop the domino effect of errors from happening, nor is there a way for a human to get involved to deal with instances that the system cannot handle.
Basic principle: If you do not know the answer to the question, “What happens if this step does not work?”, then your design is incomplete.
The Design Journey - How to Build Reliable Multi-Agent Systems
Every action depends on previous steps. Missing a step is not a shorter route but the reason for the failures previously.
Step 1: Beginning with Business Results & KPIs
Creating “Our research workflow is automated” is a direction and not an objective. A reliable objective would like “given a support ticket I will classify the ticket, write a response, direct to a correct queue within 30 seconds at 95% success rate”. This example demonstrates the task, metric, boundary and level to verify accomplishment. You will need to determine your KPI’s prior to creating any agents; they will be task completion rates, error rates, latency rates, rate of escalation, and costs per task.
Step 2: Splitting the System into Distinctives Agents
All agents should have a single responsibility, a defined input format, a defined output format, and performance limitations. If two agents share state then you will have problems with maintaining consistency. Each agent’s responsibility should be viewed like an API contract, as functionally they are a contract/ promise between two agents.
Step 3: Design the orchestration and communication layer.
Establish a definitive messaging schema across each agent-to-agent relationship you create. Carefully select your topology (linear pipeline, fan-out, event-driven, etc.) depending on the failure modes of the topology. Model the states of your tasks (pending, working, verified, failed, escalated) and describe the valid transitions between task states. Do not count upon agents to create their own organisation. Establish a graph; allow agents to navigate through the established graph.
Step 4: Design your state and memory management with care.
Give each agent access to only that state necessary for its operation. Minimise shared memory because it causes coordination problems and can be problematic due to possible conflicts. Implement either explicit write locks or optimistic concurrency control for shared state. Think about where context window limits may be an issue; long term tasks will cause an overflow of the task state if you do not manage the information being passed forward via explicit summarisation or sliding window techniques.
Step 5: Add Layers to Verify and Guardrail
This is a step that many neglect and therefore fail to carry out. An output must be checked to verify it passes through the required barriers operating between agents. First is the schema check (i.e. is it in the expected form). Second, is semantic check (i.e. does it give off correct information), third is the safety guardrail (i.e. does it fall within the allowable scope). Before an irreversible real world action by an agent is carried out, the guardrail must check to ensure this action complies with the policies of the operator.
Design rule: Reasoning and execution are separated by a verification gate. The executor reasons why an action should occur, the verifier verifies whether the action has complied with the policy. Execution occurs after Reasoning and Verifying.
Stage 6: Develop a Plan for Failures, Retry Attempts, and Human Review
All elements in your agent’s architecture should have specified failure behaviours; this applies equally to success behaviours. Configure every agent and tool call to have set amount of time Once you’ve determined the length of the timeout for each activity, start establishing maximum retry counts by creating an escalation process based on the impact of the failure and request human review. Additionally establish circuit breakers configured for immediate failure when the dependent services fail continuously.
Stage 7: Create Visibility and Debugging Capabilities in The System
Each action taken by an agent, each call made to a tool, and each state change in your system should provide a well-defined log containing a task ID, agent ID, date / time and inputs, outputs, and the status of the operation. All tasks should generate an end-to-end distributed trace. Create real-time monitoring of the Key Performance Indicators (KPI) associated with your system and implement alerting to notify of any abnormal outcomes. Investigate investing in the capability to perform failure replays, which will allow for live simulations of the exact sequence of events that caused the failure occurring in a controlled environment; this functionality will be the most valuable error-debugging feature you can develop.
Step 8: Keep Tools and External APIs Stable
You should never directly call external APIs from within the prompts of the agent. It is important to wrap every single integration with a typed function to provide input validation, response schema validation, error normalization, and logging. During the development phase, you will want to test against mocked APIs. The response format of external APIs can change without notice, and any schema drift is a very issues in agentic systems.
Example Scenario: A research agent using a third party database to retrieve data was providing nonsensical results because the API provider changed their response schema without warning. The agent was looking for data within the field “results,” which the API provider changed to “data,” so the agent was still executing after having received no data (the agent still thought it was a valid response). If there had been schema validation in place, the agent would have thrown an error immediately, and there would not have been any chances of the agent hallucinating about the data it possessed.
Step 9: Designing Systems for Scale
Designing each agent instance as “stateless”, where all persistent state is stored externally, and designing the agent with message queuing to “decouple” the agents from each other and provide natural “Back Pressure”. Implement a rate limit on each agent and by each tool; if you want to avoid hitting the LLM API rate limit in production, you need to explicitly manage them. You should also model your estimated token consumption and total API call volume before deployment since the costs of having a multi-agent infrastructure scale non-linearly.
Step 10: Securing the System through Access Control, Permission and Auditability
Each agent has “Least Privilege” permissions based on their role in the system (they can only access tools and data needed to accomplish their assigned function). Log every significant action taken by an agent (including the agent’s ID, task ID, timestamp, and full context); all log files must be protected from modification. Sandbox any agents that execute code or access file systems; all external input to an agent should be “sanitized” to protect against prompt injection, a real and increasing attack vector for agentic systems.
Mistakes in Architecture
- Monoliths – You should avoid building your agents in a monolithic design where the agent is doing the entire process from Plan > Execute > Verify > Memory Management as it makes it impossible to correctly test, verify, scale or debug each function within the agent separately.
- No Verifier Layer – If you do not place a verifier between every phase within your output, then if you create a Hallucinatory Output from PhA1 it can potentially corrupt your entire pipeline.
- Fallback Strategy – Each Node or agent should have a defined fallback strategy, if you do not have one you have not finished designing the agent.
- Over-Autonomy – All agents should have appropriate rules established that limit their autonomy so that they do not behave in a dangerous way when given their broad permissions. If a system never escalates, it is because the system has not been created with the appropriate fail-safe mechanisms in place.
Where Multi-Agent Systems Actually Work Well
Be transparent about compatibility before committing to the architecture:
- The Narrower Your Automation Workflows: The more Limited the Type of Document Processing, Enriching of Data, Generating Reports from Structured Formatted Inputs – therefore, the More Defined and Predictable the Task; the More Reliable the System.
- Internal Enterprise Processes: Where You Have a Controlled Input Process, Hence There are Fewer Threats Resulting in Catastrophic Failure; and So You Can Easily Use Human Intervention When Needed.
- Semi-Autonomously Operated Co-Pilot Agents: When Co-Pilot Agents Perform Document Drafting, Suggesting, And Surface but (NOT) Performing Single Action Can Provide you fast value & least risk.
Production-Readiness Checklist
Architecture
- Defining measurable targets and key performance indicators with business outcomes.
- Documenting all roles for agents, identifying input/output/boundaries.
- Establishing message schema on all pairs of agents.
- (Example: defining state of tasks).
Reliability
- All critical actions must go through a verification layer before execution.
- Timeout configuration on all agent/tool calls.
- Retry logic with linear and exponential backoff, idempotent operations.
- Circuit breaker functionalities (for all external dependencies).
- Defined human escalation procedures (with testing).
Observability
- Structured log of all actions that have occurred, with all states being modified, including how they were modified.
- Distributed traceability of each task with end-to-end visibility.
- KPI Dashboard that is real-time and has alerts configured.
- Ability to replay failed actions via real-time tracking.
Security
- Least privileges must be enforced for each role of the agent.
- Unchangeable audit logs for each consequential action that occurs.
Sanitization and protection against prompt injections for user input data
Conclusion
The difference between examples of performing multi-agent AI systems (e.g., working systems) and those that do not work as well in real life is not just model performance. The difference is in system engineering discipline.
Engineers who develop productive multi-agent systems have reliable models than an engineer who has improved access to a better model. Engineers who have developed productive multi-agent systems clearly define the boundaries of roles, build verification elements prior to their need, instrument all elements, and design for escalation of human intervention rather than make those an admission of failure.
Structure is the basis of determination; intelligence without structure leads to complexity; however, complexity without visibility leads to risk. You must develop your structure before you can allow your agents to use their intelligence.
FAQs
The number of agents necessary depend on the distinct roles of:
- Planning agent
- Execution agent
- Verification agent
Use only enough agents to represent the core functions in production until the production data demonstrates otherwise. Three specific agents are more reliable than 15 with undefined functions.
While they dramatically accelerate development they add ambiguity regarding failure points. Evaluate the framework based on the requirements indicated in this guide. For example, if you lack schema validation between agents, distributed tracing, or a mechanism to escalate to a human when necessary, you need to augment the framework you choose or build from primitive code line constructs.
Prevent all incoming data from externally from entering an agent’s context through data sanitization processes. Normalize the data through the network. Use another validation scheme, such as a model, that will identify any prompt injection issues before the agent utilizes the injected prompt. Least-privileged account definition should also be used, preventing agents who have received a successful injection from performing any function outside of their defined role.
Determining the level of human approval required is based on the significance of the function being performed by agents. Drafts and reports would be considered reversible and have little value hence minimal human oversight would be appropriate. However, banking transfers, communications, or executing production code would require an inherent human approval path; hence there should be a clear escalation path from the start of system implementation. As your system earns trust over time you can decrease your human validation paths.
