Inside the Seven-Stage RAG Pipeline: Where Quality Actually Breaks

Most "hallucinations" in RAG systems are retrieval failures dressed up as model failures. Walking the pipeline stage by stage makes that obvious.

Stage 1 — Ingestion

Source of truth breaks here first. PDFs with two-column layouts, tables split across pages, and scanned images without OCR silently drop content. Log per-document extraction coverage and alert when it dips.

Stage 2 — Chunking

Fixed-size chunking severs sentences and tables. Structure-aware chunking (by heading, paragraph, or semantic unit) preserves the reasoning context the model needs. Overlap of 10–20% smooths boundary loss.

Stage 3 — Embedding

Embedding model choice determines the ceiling of retrieval quality. Domain-specific embeddings (legal, medical, code) outperform general models by wide margins on their turf. Refresh embeddings when the model changes — never mix vintages.

Stage 4 — Retrieval

Hybrid retrieval (dense + BM25) reliably beats either alone. Tune top-k against recall on a golden set, not by feel.

Stage 5 — Reranking

A cross-encoder reranker over the top 50 candidates lifts precision substantially. Skip it and your generator sees noisy context, which is where hallucinations start.

Stage 6 — Generation

Constrain the prompt: "Answer only from the provided context; if not present, say so." Combine with structured output for citations.

Stage 7 — Citation and grounding

Every claim should link back to a chunk ID. Grounded, cited answers turn RAG from a black box into a reviewable artifact — the pattern we use in grounded answers with citations.

The failure heatmap