Inside hybrid search: why RAG needs both keywords and embeddings

Vector embeddings get the headlines, but production RAG systems almost always run hybrid search — embeddings plus traditional keyword indexing. Here's why neither works alone.

What each approach is good at

Dense (vector) search encodes meaning. Ask "can I work remotely?" and it finds a passage titled "telecommuting eligibility" even without shared words. It's forgiving of phrasing.

Sparse (keyword) search like BM25 matches tokens exactly. Ask for "Pantone 2685" or drug code "J3490" and it finds the row that contains those characters. It doesn't guess.

Where each fails alone

Pure vector search quietly drops exact-match queries. The embedding for "J3490" sits near other medical codes, so the top result may be the wrong code with similar vector geometry — delivered with confidence.

Pure keyword search fails the opposite way. A user asking "what's our refund window?" gets nothing if the policy says "return period." Synonyms, paraphrases, and plain-English questions bounce off the index.

How hybrid retrieval works

1. Run the query through both indexes in parallel.
2. Normalize scores (reciprocal rank fusion is a common choice).
3. Merge the two result lists into one ranked set.
4. Pass the top passages to a reranker for a final ordering before the model sees them.

What to tune

• Weighting. Give keyword matches more weight in domains full of codes (healthcare, finance, retail SKUs). Give semantic more weight in policy and knowledge-base content.
• Reranking. A cross-encoder reranker on the merged list reliably lifts answer quality, at modest cost.
• Filters first. Apply metadata filters (region, effective date, document type) before fusion so neither index wastes slots on ineligible content.

How to know it's helping

Split your evaluation set into "semantic" and "exact-term" questions. Measure recall@k on each with vector-only, keyword-only, and hybrid. In most enterprise corpora, hybrid wins both categories once reranking is in place.

For the end-to-end grounding story, see how to reduce AI hallucinations in enterprise answers.