Production AI Fails Outside the Model: How to Engineer Fallbacks, Observability, and Ownership
With each new model deployed, most teams creating AI-powered systems are getting extremely adept at answering this one question: is the model’s output correct? Many fewer teams have developed the ability to respond to a more important, more challenging inquiry: What if this output leads to independent action?
That distinction is no longer academic. The definition of “correctnes” has become even more fluid with the emergence of agentic systems in AI, which plan, call tools, change state and take real actions across distributed infrastructure, as opposed to one-off chat responses. A model can return a perfectly reasonable looking answer, and the system around the model does the opposite. It is a concept I’ve been thinking about a lot in my own work, where I’m creating large-scale distributed systems that are driven by ML models, and I’ve started to call this the reliability gap — the gap between “the model was correct” and “the system was correct.”
The reliability gap
Think of an autonomous agent which needs to update a record in a multi-tenant system. It reads the correct context, calls the proper tool and gives a clean confident summary. All the pieces of action seem to be appropriate. However, along the way, it became confused about tenant boundaries and wrote to the wrong scope. The tool was successful. The answer at the end seemed to be correct. This would be a passed final-answer evaluator. The system, in fact, did not work.
This is the main failure mode that teams should try to prevent. Traditional evaluation – that is, judging only the output – is structurally blind to a bad execution path and a good final answer could be the result.
From short-lived requests to long-running trajectories
Classic LLM interactions are straightforward to reason about: Input is fed into the model, the model generates an output, and the main question is whether the output is correct or not. The side effects can be restricted, the interaction is temporary and the response is obvious. All three of these assumptions are violated by agentic systems. They require a goal, a plan, a series of calls to tools, changing state, and real world effects that continue and accumulate. The main question has less to do with if the answer was right – it’s whether the system behaved properly throughout the actual trajectory. It is a completely different engineering problem and an evaluation model is needed.
A practical four-layer model for reliability
I found it to be helpful to think of reliability as being on four layers of a system of agents rather than as a single pass/fail indicator.
- Outcome — Useful, accurate and complete?
- Process — Was the sequence of actions used by the agent coherent and defensible, with the use of context and tools?
- Control — Was there control of all actions within policy, authority and risk bounds?
- Recovery — Was failure detected, were the blast radius contained and did the system recover safely?
A system can pass on outcome and still fail badly on control or recovery. If we think of these as independent different criteria, each scored separately, rather than as a monolithic “was it right” decision, then these silent failures will become more obvious before they become costly ones.
Treat the trace as the flight recorder
None of this is evaluable after the fact unless you capture it. The single highest-leverage investment a team can make is treating the entire execution trace — state changes, tool calls, policy checks, retries, and decisions — as the agent’s flight recorder. If the outcome is incorrect, the trace will tell you where reliability failed: as a context retrieval, invalid tool call, a policy boundary that prevented the execution or a recovery step that failed to run? If you don’t have a correlated end-to-end execution record, each failure appears to be a red failure. Failure is now localized – a particular segment, a particular dependency, a particular decision.
In practice that means recording model calls, tool calls, context changes, memory changes, policy decisions and changes to the state of the environment into a single, correlated sequence of events you can query — much like you would trace a distributed request. Next, rate the journey, rather than the destination: did the Goal and the Constraints persist through the steps; was the context retrieved that was actually used; were the tool calls valid; was each decision a consequence of the preceding one; was each action within the policy; if something went wrong, did recovery work?
Every autonomous action spends a risk budget
The next piece will be control. Not all of the actions an agent can take have the same repercussion, and applying the same treatment to all of them is a no-no. The following is a simple heuristic that I find helpful to rate each possible action that an agent can take on three scores: impact if it fails, uncertainty in the decision, and difficulty reversing the action, all on a low/medium/high scale; and take the worst of the three scores as the handling.
Low-risk actions: something that can be looked up — draft that can be reversed (read-only) should be done. Actions that are medium risk should be validated and have a limited scope.
Actions that are high-risk, high-impact and unreversible should go to a human approval gate or be halted. A confident model output doesn’t mean that something is irreversible and safe; confidence and risk are independent and it's the systems that mix them up that are the ones that cause real harm.
This inevitably results in a hierarchical autonomy model which has proven successful in production:
- Observe — Read, analyse and explain; there is no external mutation and evidence is recorded.
- Propose — the system proposes, simulates or suggests, but is human who approves before it is executed.
- Execute — bounded and narrow authority, prescribed rollback path, and a system action.
Transitions between these modes should only be made by a trace of evidence and not assumed for the default configuration. In more concrete terms, before a system moves to a new tier, it is important to be able to answer “yes” to all five of these questions (can we reconstruct what occurred over a full run; can we score process, control and recovery, as well as output; is authority bounded by scope, impact and time; can we stop, compensate, retry safely, or rollback; and will failures feed back into tests and policy)? If an answer to any question is no, the system has not gained the next level, even if the answers to the model are good in a demo.
Too many teams begin agents at “Execute” because it is the most impressive demo and then find out the price they have to pay for it after an incident.
Who owns it when the agent is wrong?
Questions that remain unanswered with tracing and bounding are: When a system full of agents makes a bad action, who is responsible for that action?
We’ve got answers for deterministic services for decades now: There is an owning team, there is an on-call rotation, there is an error budget, and there is a runbook. Agentic systems are often used and do not have any of that. The model is vendored (not open source), there is no developer control over the prompts that are deployed that are in the config repo, there are tools from three different teams, and the policy boundary has been configured by the guy who did the demo.
When things go wrong, the incident review gets hung up on a question without a question owner: Was that a model problem, a prompt problem, a tool problem or a policy problem?
Three promises are put in place to bridge that divide. First all agents have to be assigned to an owning team, not to a channel, but to an actual team, not just a channel, to which an agent belongs, and to which pager notifications are sent when an agent misbehaves, not just errors.
Secondly, the policy boundary should have its own documented owner, who is not the person that has the ability to ship, since the incentive to broaden the scope of a person’s authority and take on the responsibility for the resulting consequences should not be distributed across two people.
Third, the incident review process must also not assume that the finding of ‘the agent did what it was supposed to do, and it was bad’ is an unfortunate trait of AI, but rather a legitimate finding that has an action item. If there’s a bad action in a policy that was allowed to be in the policy, then it’s a control defect and should be submitted in the same queue as any other defect.
None of this is exotic. It’s the ownership model we already apply to anything that can page someone, extended to a class of systems that can now act on its own.
Reliability engineering, not prompt engineering
It’s worth naming what this actually is: distributed-systems engineering, for AI. Any single failure domain is more than a goal, plan, model, tool, state, effect — it’s several of them.
That translates to the practices that work in classic distributed systems are the same ones that agentic AI teams need to implement: end-to-end telemetry that tracks the entire journey, isolation so that one task or tenant doesn’t impact another, backpressure – where the system slows down or stops when there are degraded dependencies – and rollback designed before the action is taken, not after an incident.
The final answer is an output. Reliability is a system property.
The industry’s evaluation culture has been created for a world of model interactions in a single turn, that it hasn’t adapted to a world of AI systems planning and acting and persisting. This isn’t a new discipline; it’s simply the application of the discipline that is already familiar to us (distributed systems engineering) to a class of systems that is more and more deserving of it.
SD Times Q&A
What is the reliability gap in agentic AI systems?
The reliability gap is the difference between a model producing a correct-looking output and the overall system behaving correctly. In agentic systems, a model can return a plausible answer while the surrounding system takes a wrong action — for example, writing to the wrong tenant scope in a multi-tenant database. Evaluating only final output is structurally blind to bad execution paths.
How do you add observability to an autonomous AI agent in production?
Treat the full execution trace — model calls, tool calls, context and memory changes, policy decisions, and state mutations — as a correlated, queryable record, similar to distributed request tracing. This trace functions as a flight recorder: when an outcome fails, you can pinpoint whether the failure was in context retrieval, a tool call, a policy boundary, or a recovery step. Without end-to-end trace correlation, every failure appears as an undifferentiated red alert.
How should AI agent actions be risk-rated before execution?
Score each possible agent action on three dimensions: impact if it fails, uncertainty in the decision, and reversibility — each rated low, medium, or high. Take the worst score across the three as the required handling tier. Low-risk (reversible, read-only) actions can execute autonomously; medium-risk actions require validation and scoped authority; high-risk or irreversible actions should require human approval or be halted entirely.
Who owns an AI agent incident when the agent takes a wrong action?
Ownership should be assigned to a named engineering team — not a Slack channel — with an on-call rotation and pager alerts tied to agent misbehavior, not just hard errors. Policy boundaries need a separate documented owner distinct from whoever can ship the agent, to separate the incentive to broaden authority from accountability for consequences. Incidents where ‘the agent did what policy allowed but the action was bad’ should be treated as control defects with action items, not accepted as inherent AI unpredictability.
What is a hierarchical autonomy model for AI agents and when should each tier be used?
A hierarchical autonomy model defines three operating tiers: Observe (read-only, no external mutation), Propose (system suggests or simulates but a human approves before execution), and Execute (bounded authority with a prescribed rollback path). Promotion to a higher tier requires answering ‘yes’ to five criteria: full run reconstructability, scoreable process/control/recovery, time- and scope-bounded authority, safe rollback capability, and a feedback loop from failures into tests and policy.
The post Production AI Fails Outside the Model: How to Engineer Fallbacks, Observability, and Ownership appeared first on SD Times.
Tech Developers
No comments