AI Engineering
Why most AI prototypes fail in production
A prototype proves that a model can produce a good answer once. Production requires a system that produces acceptable answers every time, at a known cost, on data nobody curated. The gap between the two is engineering, not model quality.
· 4 min read · Starone Engineering
An AI prototype is usually a prompt, a model API and a small set of inputs chosen by the person who built it. It answers one question: can a model do this task at all. That is a useful question, and a notebook is the right place to answer it.
Production asks different questions. What happens on the inputs nobody chose, how the system behaves when the model is slow or wrong, what it costs per request, who is allowed to see which data, and how anyone will know when quality drops. Our position is that most prototypes stall because these questions were never part of the prototype, not because the model was inadequate.
The demo measures the wrong thing
A demo is evaluated by a person looking at a handful of outputs and judging them plausible. That is a test of the best case. Real traffic contains ambiguous requests, inputs in the wrong language, documents that were scanned badly, questions the system should refuse and questions that have no answer in the data.
Language models fail quietly. A conventional service that cannot answer returns an error; a model that cannot answer returns fluent text. Without a measurement that separates correct from fluent, the team has no evidence about how the system behaves outside the examples it was shown.
No evaluation set means no engineering
The first production artefact we build is not the pipeline. It is an evaluation set: a versioned collection of representative inputs with expected outputs or grading criteria, including the difficult and adversarial cases. Until it exists, every prompt change, model upgrade or retrieval tweak is a guess, and regressions are found by users.
The evaluation set should run in continuous integration like any other test suite. Exact-match checks cover structured outputs. Rubric-based grading by a second model covers free text, provided the grader itself is spot-checked against human judgement. The set grows from production: every reported failure becomes a test case.
The data is the system
In a prototype, the documents are a folder someone exported. In production they are a moving corpus with owners, permissions, duplicates, contradictions and deletion requirements. The unglamorous work lives here: parsing tables and scanned PDFs, deciding how to chunk, keeping the index in step with the source, and removing a document everywhere when it is withdrawn.
Access control is the part most often missed. If retrieval ignores the caller's permissions, the model will summarise a document the user was never entitled to read. Filtering must happen at query time against the caller's identity, not as an instruction in the prompt.
Latency, cost and the tail
A prototype tolerates a slow answer because one person is waiting and nobody is counting tokens. A production path has a latency budget and a unit cost, and both are dominated by the tail. A chain of several model calls multiplies the chance that one of them is slow, rate-limited or times out.
The engineering responses are ordinary ones applied to an unusual dependency:
- Streaming responses so perceived latency is the time to the first token, not the last.
- Timeouts, bounded retries with jitter, and a fallback model or a degraded answer when the primary is unavailable.
- Routing simple requests to a smaller model and reserving the large one for the cases that need it.
- Caching stable prompt prefixes and repeated lookups, with token usage tracked per feature and per tenant.
Non-determinism needs change control
The same input can produce different outputs, and the model behind an API alias can change without a deployment on your side. A change to any part of the request path is a release and needs the same control as a code change.
Where the output feeds another system, constrain it. Structured output against a schema, validated before use, turns a class of parsing failures into a retry instead of a corrupted record.
- Treat the prompt, model version, retrieval configuration and tool definitions as one release unit.
- Pin model versions explicitly instead of following a moving alias.
- Keep prompts in source control and review them like code.
- Promote a change only when the evaluation set shows it is no worse than what it replaces.
Security and observability arrive late
A model that reads untrusted text can be instructed by that text. Prompt injection through a retrieved document, an email or a web page is the defining security problem of these systems, and it has no complete fix at the prompt level. The dependable control is architectural: limit what the model can do, separate the privileges of reading from those of acting, and require confirmation for anything irreversible.
Observability is the other late arrival. A production system needs a trace per request that records the prompt version, retrieved context, tool calls, token counts, latency and the final output, with personal data handled according to policy. Without that trace, a complaint about a bad answer cannot be reproduced.
When a prototype is exactly right
None of this argues against prototyping. A fast, disposable experiment is the cheapest way to learn whether a task is feasible and whether users want the result. The mistake is promoting the experiment instead of rebuilding it. Our default is to treat the prototype as a specification: it shows what good output looks like, and the production system is engineered from that evidence.
For an internal tool used by a few people who review every output, a lightly hardened prototype may be all that is justified. The investment should scale with the cost of a wrong answer.