Reference Architecture
IllustrativeAI Knowledge Platform
A retrieval-augmented platform that answers questions over enterprise documents and systems, with citations, permission-aware retrieval and full observability.
The problem
Consider an organisation whose working knowledge is spread across documents, wikis, ticketing tools, databases and internal systems. People spend time finding the right source, then more time deciding whether it is current. Keyword search returns pages rather than answers, and the people who know where things live become a bottleneck.
A language model alone does not solve this. It has no knowledge of the organisation's content, it will answer confidently when it should not, and it has no concept of who is allowed to read what. The engineering problem is to ground a model in the organisation's own material, keep that material current, respect existing permissions and make every answer checkable.
Requirements the design must meet
- Every answer must cite the passages it was generated from, and the system must decline to answer when retrieval finds nothing relevant.
- Retrieval must enforce the source system's permissions, so a user never receives content they could not open directly.
- Changes in source systems must reach the index within a defined freshness window, including deletions and permission changes.
- Answer quality must be measurable against a maintained evaluation set before any change to prompts, models or retrieval is released.
- The model provider must be replaceable without rewriting the application, and sensitive data must stay within agreed boundaries.
- Every request must be traceable end to end, with latency, token cost and retrieval results recorded per step.
The design
Components
Ingestion
- 01
Sources
Documents, databases, internal APIs
- 02
Parse and chunk
Extract text, keep structure
- 03
Embed
Batch embeddings, versioned model
- 04
Index
Vector and keyword indexes with ACLs
Query
- 01
Question
Authenticated user and their scope
- 02
Retrieve
Hybrid vector and keyword search
- 03
Rerank
Keep only the strongest passages
- 04
Generate
Grounded answer with citations
- 05
Evaluate and log
Trace, score and capture feedback
Ingestion and connectors
Connectors pull content from document stores, wikis, databases and internal APIs on a schedule or in response to change events. Each document carries its source identifier, version and access control list through the pipeline. Deletions and permission changes are treated as first-class events, not as an afterthought of the next full crawl.
Parsing, chunking and embedding
Documents are parsed into text while preserving headings, tables and page references, then split into chunks sized for the content type. Embeddings are produced in batches by a versioned embedding model, so the index can be rebuilt deliberately when that model changes. Chunk metadata is kept rich enough to build a useful citation.
Vector database and keyword index
Chunks are stored in a vector index alongside a keyword index over the same content. Hybrid retrieval covers the cases where either method alone is weak: exact identifiers and product codes on one side, paraphrased questions on the other. Access control metadata is indexed with each chunk so that filtering happens inside the query, not after it.
LLM orchestration service
A dedicated service owns prompt templates, query rewriting, retrieval, reranking, model routing and guardrails. Prompts are versioned artefacts with tests, not strings scattered through application code. The service exposes a small, stable contract, which keeps the model provider and the retrieval strategy replaceable.
Enterprise API layer
An API gateway exposes the platform to web clients, chat tools and other internal systems through one authenticated interface. It applies rate limits and quotas per user and per application, and streams responses so that long answers feel responsive. Structured lookups against systems of record are offered to the orchestrator as typed tools.
Authentication and authorisation
Users sign in through the organisation's identity provider using OIDC or SAML. The user's identity and group membership are propagated to the retrieval query, where they are matched against the access control lists stored with each chunk. Service-to-service calls use short-lived credentials, not shared keys.
Evaluation harness
A curated set of questions with expected sources and reference answers is run against every candidate change. Retrieval is scored separately from generation, because the two fail for different reasons and are fixed in different places. User feedback from production is reviewed and promoted into the evaluation set over time.
Observability
Each request produces a trace covering query rewriting, retrieval, reranking and generation, with the retrieved chunk identifiers, token counts and latency attached. Dashboards track cost per request, retrieval hit quality and the rate of declined answers. This is what makes a poor answer diagnosable instead of merely reported.
Trade-offs
Design decisions
| Decision | Why |
|---|---|
| Retrieval-augmented generation before fine-tuning | Enterprise content changes daily and must be attributable to a source. Retrieval handles freshness, citations and permissions directly, where fine-tuning bakes knowledge into weights that cannot be cited or selectively revoked. Fine-tuning remains an option for tone or format once retrieval quality is established. |
| Hybrid retrieval with a reranking stage | Vector search alone misses exact terms such as part numbers and clause references, and keyword search alone misses paraphrase. Running both and reranking the merged candidates costs additional latency, which is accepted because retrieval quality sets the ceiling on answer quality. |
| Permissions enforced at retrieval time, inside the index query | Filtering after generation is too late, because the model has already read the restricted text. Storing access metadata with each chunk increases index size and makes permission sync a hard requirement, but it is the only placement that prevents leakage by construction. |
| An orchestration service separate from the application | Keeping prompts, routing and guardrails in one service adds a network hop and a component to operate. In exchange, several applications share one tested implementation, and a change of model or provider is a contained release instead of a rewrite. |
| Decline to answer when evidence is weak | A threshold on retrieval relevance means some answerable questions will be refused. That is preferable to a fluent answer with no supporting source, because trust in the platform is lost faster than it is built. |
Intended outcomes
- Answers that cite their sources, so reviewers can verify them
- Retrieval that follows existing permissions instead of creating a parallel access model
- Changes to prompts, models and retrieval that are evaluated before release
- Poor answers that can be traced to a specific retrieval or generation step
What the design is built to achieve. Illustrative, not measured results from a client project.
Representative technologies
- Python
- FastAPI
- TypeScript
- Next.js
- PostgreSQL
- pgvector
- OpenSearch
- Redis
- Kafka
- Kubernetes
- OpenTelemetry
- OIDC / SAML
Have a system that needs to work in production?
Tell us what you are building. An engineer will read it and reply within two working days.