Designing reliable AI agents

An agent is a language model that decides its own next step. It is given a goal and a set of tools, it calls a tool, reads the result and chooses again until it judges the task complete. That loop is what lets it handle work that cannot be scripted in advance, and it is also the source of every reliability problem: the control flow is chosen at run time by a probabilistic component.

Our position is that agent reliability is mostly a systems problem. Better models reduce the error rate per step, but a long task multiplies per-step errors, so the surrounding engineering decides whether the system can be trusted with real actions.

If the steps are known in advance, write a workflow. A fixed sequence of model calls with code deciding the order is cheaper, faster, easier to test and easier to explain to an auditor. Routing, extraction, summarisation and most document processing fit this shape.

An agent is justified when the path depends on what is discovered along the way: investigating an incident, reconciling records across systems with inconsistent data, or working through a codebase. Many good designs are hybrids, with a deterministic workflow that hands one bounded sub-task to an agent.

The characteristic failure is the tool-call loop: the agent repeats the same failing call with trivial variations, or oscillates between two actions, consuming tokens until something external stops it. Every agent needs hard limits that are enforced by the harness and not requested in the prompt.

Agents retry. So does the harness when a network call times out with the outcome unknown. If a tool sends a payment, creates a ticket or emails a customer, a retry repeats the side effect. Every tool that changes state should accept an idempotency key derived from the task and step, so that a repeated call returns the original result instead of acting again.

Tool design matters as much as prompt design. A tool should do one thing, validate its arguments against a strict schema, and return errors the model can act on, such as which field was invalid and what values are allowed. A vague error string invites a blind retry. Prefer narrow tools over a general one that accepts arbitrary SQL or shell commands, because the tool boundary is where permissions are enforced.

Any text an agent reads can contain instructions: a web page, a support ticket, a file, the output of another tool. This is indirect prompt injection, and no prompt wording reliably prevents it. The practical defence is to assume the model can be steered and to limit the consequences.

Run the agent with the least privilege the task needs, using credentials scoped to the user it acts for, not a shared service account. Classify tools by reversibility. Reads can run freely; reversible writes can run with logging; irreversible or high-value actions need an approval step where a person sees exactly what will be executed. Be most careful when one run combines access to private data, exposure to untrusted content and a channel that can send data out.

These controls have a cost: approval gates slow work down and narrow tools limit what the agent can solve. In a sandbox with no production credentials, such as a coding agent working on a branch that a person reviews before merging, broad autonomy is reasonable because every action is reversible. The level of control should follow the blast radius of a mistake, not a general unease about agents.

Over a long task the context fills with tool output, and quality declines before the hard limit is reached. Early instructions lose influence, stale intermediate results are treated as current, and the agent drifts from the original goal. This context drift is gradual, which makes it hard to see in a short test.

The mitigations are structural. Truncate or summarise large tool results before they enter the context. Keep durable state such as the plan, decisions made and facts established in an external store that is re-read, not left to scroll away. Compact the history at defined points. Delegate self-contained sub-tasks to a sub-agent with a fresh context that returns only its conclusion.

A task that runs for minutes or hours will be interrupted by a deployment, a rate limit or a crash. Persist the state after every step so that a run resumes from its last checkpoint instead of starting again and repeating side effects. Durable execution engines such as Temporal fit this well, with each tool call as a recorded activity.

Record the full trajectory: every model input, tool call, argument, result, token count and timing, linked by a run identifier. When an agent does something unexpected, this trace is the only way to learn why.

A correct final answer can hide an unacceptable path, such as reading records the task did not require or taking forty steps where four would do. Evaluation therefore covers the whole trajectory against a fixed suite of tasks run in a sandbox with realistic tools.