AI systems engineered for production, not for the demo

A production AI system is easier to reason about as layers. A user interacts with the application layer. The application calls an AI orchestration layer, which invokes models and agents. Those draw on a knowledge and data layer and act through enterprise systems and APIs, with infrastructure and observability underneath all of it.

Connecting an application directly to an LLM API collapses these layers into one function call. It works until the system needs something the call cannot provide: grounding in private data, per-user permissions, a cheaper model for simple requests, a record of why an answer was given, or protection against a document that contains instructions. Each of those belongs to a specific layer, and adding them later means restructuring the application.

The same path can be read as a lifecycle: data, models, orchestration, applications, evaluation, observability. Data quality bounds what retrieval can return, evaluation decides what may be released, and observability feeds real failures back into the evaluation set.

A model answers from whatever is in its context window. Context management is therefore a design problem: which instructions, retrieved passages, conversation history and tool results go in, in what order, and what is summarised or dropped as a conversation grows. More context is not better. Irrelevant passages lower answer quality and raise cost and latency.

Most hallucination in enterprise systems is a retrieval failure: the right passage was never retrieved, so the model filled the gap. We measure retrieval separately from generation, use hybrid search and reranking to improve recall, require citations to retrieved sources, and instruct the system to say when the sources do not contain an answer. For outputs that feed other systems, schema-constrained generation and validation replace free text.

Embeddings capture similarity of meaning, which suits semantic search and questions answered by a single passage. Questions that span entities and relationships, such as which contracts reference a superseded policy, are better served by a knowledge graph or a structured query, and a well-designed system routes between them.

Inference cost scales with tokens and model size; latency scales with output length and the number of sequential model calls. The main levers are routing simple requests to smaller models, caching stable prompt prefixes and repeated answers, trimming context, batching offline work, and streaming so the user sees the first tokens quickly. An agent that makes ten sequential calls will be slow whatever the model, so parallel tool calls and fewer steps matter more than a faster endpoint.

Model observability extends tracing to the AI path. Each request records the prompt version, retrieved documents, tool calls, model, token counts, latency and outcome, with sensitive content redacted. On top of traces we sample outputs for automated and human quality scoring, track user feedback, and alert on drift in quality, cost per request and refusal rate. Without this, a provider-side model change or a degraded index stays invisible until someone complains.