The demo worked perfectly. The agent answered every question, routed every request, and summarised every document flawlessly. Then it hit real users. Within 48 hours it was hallucinating customer names, looping on ambiguous inputs, and occasionally emailing the wrong person. This is not a cautionary tale — this is a pattern we have seen on nearly every AI project we have shipped.
Why demos lie
Demos are curated. The inputs are clean, the context is complete, and the person running the demo knows what the agent can handle. Production is the opposite. Users write in fragments, switch topics mid-conversation, attach malformed files, and ask questions the agent was never designed to answer. If your agent was only tested on good inputs, you do not have a production agent — you have a demo.
The failure mode is not the LLM. The failure mode is the assumptions baked into the system around the LLM — the prompt, the context window, the routing logic, and the absence of fallbacks.
The three layers that actually break
In every production AI system we have built, breakage concentrates in three places:
- Context assembly — the code that builds the prompt. Off-by-one errors, missing delimiters, stale data from a cache, or a context window that silently truncates. None of this surfaces in a demo.
- Routing logic — the code that decides which tool or agent handles a request. Ambiguous inputs hit the wrong branch. Edge cases fall through to no branch at all.
- Output handling — the code that consumes the LLM response. Parsing brittle JSON, assuming a specific format, failing silently when the model diverges from the expected schema.
Guardrails are load-bearing, not cosmetic
Most teams treat guardrails as a post-launch checklist item. Block bad words, add a disclaimer, ship it. That is not guardrails — that is theatre. Real guardrails are structural. They define what the agent is allowed to do, what it must refuse, how it handles uncertainty, and what happens when confidence is low. They need to be designed in from the start, not bolted on after the first incident.
We use a simple rule: every action an agent can take that has side effects — sending a message, updating a record, making an API call — requires an explicit confidence threshold before it fires. Below that threshold, the agent surfaces the action for human review instead of executing it. This single pattern has prevented more production incidents than any other measure we have implemented.
Build the eject button first
Before we write a single line of agent logic, we design the human handoff path. What does the agent say when it cannot handle a request? Where does that request go? Who gets notified? How long before it escalates? This is not a fallback — it is a first-class feature. Users trust a system that knows its limits more than one that confidently answers everything.
Observability is not optional
Standard application monitoring does not work for agents. A 200 OK from your LLM call tells you nothing about whether the response was correct, helpful, or safe. You need to log inputs, outputs, tool calls, confidence scores, and escalations — and you need a way to replay and audit individual conversations. We treat every agent interaction as a structured trace, not a log line.
Every agent we ship includes a dashboard that shows live escalation rate, average confidence score, tool call distribution, and a conversation replay tool. This is non-negotiable. Without it, you are flying blind.
Latency is a design constraint, not an afterthought
LLM calls are slow. GPT-4 can take 5–15 seconds for a complex response. In a chat interface, that is tolerable. In a workflow that chains three agent calls, that is a 30–45 second wait. Users abandon. The system looks broken even when it is working correctly. Design for latency from day one: stream responses where possible, parallelize independent tool calls, cache deterministic lookups, and set clear timeout budgets for every step in the chain.