AI security and prompt injection

Prompt injection is the defining security problem of applications built on large language models, and it heads the OWASP Top 10 for LLM Applications for good reason. An attacker places text where the model will read it, and the model follows that text instead of the developer's instructions.

Our position is that this should be treated as an unpatched vulnerability class with no complete fix. Detection helps and should be used. The design work that matters assumes the model will sometimes be successfully manipulated, and makes sure that outcome is contained.

Direct injection comes from the person typing into the application. They try to override the system prompt, extract it, or talk the model into behaviour the product does not allow. This is the familiar case, and its impact is usually limited to what that user could already reach.

Indirect injection is more serious. The malicious text arrives inside content the model processes on behalf of an innocent user: a web page it was asked to summarise, a document retrieved by a RAG pipeline, an email in an inbox an agent is triaging, a field in a support ticket, the output of another tool. The user never sees the instruction. The model does, and it acts with the user's privileges.

SQL injection was solved by parameterised queries, which give the database a structural separation between code and data. A language model has no equivalent. System prompt, user message and retrieved document are all tokens in one context, and the model's tendency to follow instructions does not depend on where an instruction came from. Delimiters and role markers are conventions the model usually respects, not a boundary it is unable to cross.

Input classifiers and guard models reduce the volume of successful attacks. They are probabilistic, and an attacker can retry indefinitely with paraphrases, other languages, encodings or instructions split across several documents. A control that stops most attempts is useful against casual misuse and insufficient against someone motivated. We use these filters as one layer and never as the reason a design is considered safe.

The damage an injection can do is bounded by what the model is able to invoke. An assistant that can only produce text for a human to read is a nuisance when hijacked. An agent that can send email, run queries and call internal APIs is a remote operator for whoever controls its input.

The most damaging combination is a model that has access to private data, reads untrusted content and has some way to send information outwards. Remove any one of the three and the attack loses most of its value. The outbound channel is often the easiest to overlook, because it does not look like a tool.

A chat interface that renders Markdown images will fetch any URL the model emits, and the query string can carry whatever the model was persuaded to put there. Clickable links, web-fetch tools, webhooks and even a search query sent to a third party do the same job. We restrict rendered images and outbound requests to an allow-list of domains, strip or neutralise model-generated URLs that carry parameters, and route agent network access through an egress proxy.

Everything the model produces should be handled like data from an anonymous user. If it is inserted into a page, it is encoded for that context. If it becomes part of a database query, the query is parameterised. If it is passed to a shell or an interpreter, that step runs in a sandbox with no credentials and no network. Structured outputs are parsed and validated against a schema, and a failed validation is an error, not something to repair by asking the model again without limits.

This also applies between agents. In a multi-agent workflow, one agent's output is another's input, and an injection that lands in the first can propagate through the chain unless each hand-off is typed and checked.

Actions that move money, send messages to third parties, delete data or change permissions should require a person to confirm them. The confirmation must show the real parameters as the backend will execute them, not the model's summary of what it intends to do, because the summary is produced by the component that may have been compromised.

The trade-off is that approval prompts degrade quickly. If every step asks for confirmation, people approve without reading. We reserve approval for irreversible or externally visible actions and let reversible, low-impact steps run automatically inside tight limits. A fully autonomous agent is a reasonable choice only where its inputs are trusted or its tools cannot cause harm.

Injection resistance changes with every prompt edit and model upgrade, so it needs regression tests. We keep a suite of adversarial inputs, including documents with embedded instructions, and run it in CI against the full pipeline with assertions on tool calls, not only on text. In production, every tool invocation is logged with the content that preceded it, so an incident can be traced back to the document that caused it.