All Posts
AI Engineering 9 min read

Why our first RAG system failed (and how we fixed chunking and hybrid search)

#rag#vector-search#ai

Every tutorial on Retrieval-Augmented Generation (RAG) makes it sound trivial: split documents into 500-token chunks, embed them with OpenAI or Cohere, toss them into Pinecone or pgvector, and query with top-k cosine similarity. We followed that playbook for an enterprise knowledge-base client. Two weeks into staging, users reported answers that were technically fluent, confidently stated, and completely wrong.

The naive chunking trap

The root cause was blind chunking. Splitting documents strictly by character count or fixed token windows cuts off critical context in the middle of sentences or splits tables in half. If a financial report says "Operating expenses decreased by 12% in Q3" and the next chunk starts with "However, one-off legal penalties totaled $14M", vector retrieval would pull one chunk and miss the qualification.

We replaced fixed chunking with semantic, document-aware partitioning:

- Markdown and structural parsing that respects section hierarchies - Small-to-big retrieval: chunking small sentences for embedding precision, but returning the enclosing parent paragraph to the LLM - Preserving metadata such as table headers, document source, and publication dates directly within chunk headers

Why semantic search alone falls short

Cosine similarity is great at conceptual similarity, but terrible at keyword precision. When a user searched for exact part number "SKU-9021-B", vector distance matched dozens of similar product descriptions but missed the actual SKU because embedding models compress exact strings into generalized semantic vectors.

The breakthrough came from combining BM25 keyword search with dense vector embeddings via Reciprocal Rank Fusion (RRF). Dense vectors find conceptual answers; BM25 catches specific terminology, SKUs, and exact names. We weighted the results dynamically using a cross-encoder reranker (Cohere Rerank) that scores only the top 25 candidate chunks.

Measuring retrieval accuracy

You cannot improve what you cannot measure. We built an automated evaluation harness using Ragas and TruLens against a golden dataset of 150 client questions:

- Context Precision jumped from 54% to 91% - Context Recall improved from 62% to 94% - Faithfulness (reduction in hallucinated claims) rose to 98%

By measuring every prompt and retrieval tweak against automated metrics, we stopped relying on gut feel and started treating retrieval tuning like compiler optimization.

Key takeaways

1. Never use fixed-size token splitting without context window awareness. 2. Always pair vector search with traditional sparse/lexical search (BM25 or PostgreSQL tsvector). 3. Invest in reranking; cross-encoders filter out the noise that dense retrieval inevitably drags in. 4. Log every retrieval miss. What users actually query will surprise you.

Have a similar challenge?

We build production-grade software for companies that need it done right.

Let's Talk