What enterprise RAG actually requires
Retrieval-augmented generation is easy to demo and hard to operate. Here is the architecture, governance, and evaluation work that separates a pilot from a production system.
A retrieval-augmented generation prototype takes an afternoon. You embed some documents, drop them in a vector store, stuff the top matches into a prompt, and the demo works. Then it meets a real enterprise: ten million documents, strict access controls, auditors, and users who notice the moment an answer is wrong. That is where most RAG projects quietly stall.
This is a field guide to the parts that do not show up in the demo — the retrieval quality, governance, and evaluation work that decides whether enterprise RAG reaches production.
Retrieval is the product
The model is a commodity. Your retrieval layer is not. If the right passage never makes it into the context window, no amount of prompt engineering recovers it. Three things move the needle more than model choice:
- Chunking that respects structure — split on headings, tables, and semantic boundaries, not a fixed 500-token window that cuts sentences in half.
- Hybrid retrieval — combine dense vector similarity with sparse keyword (BM25) search, then rerank. Pure vector search misses exact terms like part numbers and policy codes.
- Metadata filters — every chunk carries source, department, effective date, and sensitivity so retrieval can be scoped before it ever ranks.
def retrieve(query: str, user: User, k: int = 8) -> list[Chunk]:
# Scope first: never rank documents the user cannot see.
filters = {"tenant": user.tenant, "acl": {"$in": user.groups}}
dense = vector_store.search(embed(query), k=40, filters=filters)
sparse = bm25.search(query, k=40, filters=filters)
# Reciprocal rank fusion + cross-encoder rerank.
fused = reciprocal_rank_fusion([dense, sparse])
return reranker.rank(query, fused)[:k]Grounding and honest failure
The most valuable answer a RAG system can give is sometimes "I don't have that." Confident fabrication destroys trust faster than a visible gap. Two practices matter:
- Require citations. Every claim maps to a retrieved chunk, and the UI links back to the source.
- Gate on retrieval confidence. If the top results are weak, refuse or escalate instead of guessing.
In regulated environments, a wrong answer is not a bug — it is a liability. Design for abstention.
— Yatzon engagement playbook
Evaluation you can trust
You cannot improve what you cannot measure, and "it looks good" is not measurement. Stand up an evaluation harness before you scale, not after.
Retrieval metrics
Measure recall@k and mean reciprocal rank against a labeled set of question-to-passage pairs. If the right passage is not in the top-k, the generation step is irrelevant.
Answer metrics
Use an LLM-as-judge rubric for faithfulness (is every claim grounded?), relevance, and completeness — validated against human review on a sample. Track these per release like any other regression suite.
type EvalCase = { question: string; groundTruthDocs: string[] };
async function scoreRetrieval(cases: EvalCase[]) {
let hits = 0;
for (const c of cases) {
const retrieved = await retrieve(c.question);
const ids = new Set(retrieved.map((r) => r.docId));
if (c.groundTruthDocs.some((d) => ids.has(d))) hits += 1;
}
return { recallAtK: hits / cases.length }; // ship-gate this number
}The shortlist
If you take three things from this: scope retrieval by permission before you rank, cite every claim and allow the system to abstain, and gate every release on a retrieval-and-answer evaluation suite. The model will keep getting better on its own. The system around it is the work — and the reason enterprises hire a team that has built one before.
