What actually breaks when RAG meets production

Chunking is a retrieval decision, not a preprocessing step

Most teams treat chunking as plumbing — split at 1,000 characters, add an overlap, move on. It is actually the single highest-leverage decision in the pipeline, because a chunk is the smallest unit your system can ever retrieve. If the answer spans two chunks and only one is returned, the model will confidently give you half an answer.

What has worked consistently for us is chunking along the document's own structure rather than by character count. Headings, clauses, table rows. Each chunk then carries its ancestry — document title, section path, effective date — as metadata that goes into the embedded text as well as the filter.

• Split on structural boundaries, not fixed character windows
• Embed the heading path along with the chunk body
• Keep tables whole; splitting a table destroys the relationship between header and cell
• Store effective dates as filterable metadata so superseded content can be excluded

Retrieval quality is measurable, so measure it

The thing that separates systems that survive from systems that quietly degrade is an evaluation set. Not a benchmark — your own questions, with the passages that should be retrieved, written down by someone who knows the domain.

A hundred question-and-source pairs is enough to start. Run retrieval against them on every change and track recall at k. You will discover that the embedding model upgrade everyone wanted actually made recall worse on your corpus, which is the sort of thing that is embarrassing to learn from a customer instead.

If you cannot say what your retrieval recall is this week compared to last week, you do not have a RAG system. You have a RAG demo that has been running for a while.

Hybrid search, because embeddings cannot spell

Vector search is excellent at meaning and poor at exact strings. Ask about error code E4102 or clause 7.3(b) and a pure embedding search will cheerfully return semantically similar but entirely wrong passages.

Running BM25 alongside vector search and fusing the ranks fixes most of this for very little effort. In our own evaluation sets, hybrid retrieval has consistently beaten either method alone — usually by ten to fifteen points of recall on corpora containing identifiers, part numbers or legal references.

Make the model able to say no

The failure mode that damages trust fastest is not a wrong answer. It is a confident wrong answer with a citation attached to a document that says something else. Users forgive an assistant that says it does not know. They do not forgive one that invents policy.

Two mechanisms carry most of the load here: a relevance threshold below which the system declines to answer rather than reaching for whatever ranked highest, and a grounding check that verifies each claim in the response is actually supported by the retrieved text. Both cost latency. Both are worth it.

What we do on every build now

None of this is novel and none of it is glamorous, which is precisely why it gets skipped in the rush to demo. The pattern below is what we now treat as the minimum bar before a retrieval system goes near real users.

• An evaluation set of real domain questions, versioned in the repository
• Recall and grounding scores reported in CI on every pipeline change
• Hybrid retrieval with reciprocal rank fusion as the default
• An explicit abstention path when confidence is below threshold
• Per-query cost and latency logging, so regressions are visible before the invoice