RAG in production: what actually breaks

You can build a RAGdefinitionRetrieval-augmented generation. The assistant runs a search over your company's files first, then a language model writes the answer from the passages it got back. Feed it the wrong files and you get a wrong answer, said with confidence. demo in an afternoon. A RAG system takes an engineering program, because the demo never has to answer one question. When the retriever hands back the wrong context, does anyone notice before the user does? If your company is putting an assistant in front of staff or customers, make sure someone can answer that before it goes live.

A wrong answer from an assistant doesn't look wrong. It reads well, it sounds sure of itself, and someone acts on it. An agent quotes last quarter's pricing to a customer. A manager follows a policy clause without the exception that sat in the next paragraph. The cost shows up later as a refund, a complaint or an audit finding, and nobody connects it back to the search step.

The worse failure is a leak. If the search index ignores who's allowed to read which document, anyone who can type a question can pull salary data, legal drafts or another customer's records out of it. OWASP's Top 10 for LLM applications lists this as its own risk, vector and embedding weaknesses (LLM08:2025), and it lists permission-aware vector stores among its recommended defenses. Treat the assistant as a new way into every document you load into it.

For engineers: retrieval fails silently, and the model papers over it. The wrong answers that cause trouble tend to be confident ones, written on top of the wrong passages. Check these causes first, before anyone touches the prompt.

• Bad chunking splits a policy clause from its exceptions. Someone asks about the rule, gets the clause, and never sees when it doesn't apply.
• Pure dense retrieval misses the exact identifiers, part numbers and acronyms keyword search would have caught.
• A stale index serves last quarter's pricing with this quarter's confidence, and the model writes fluently over the wrong evidence.

The fixes are unglamorous and well understood. Use hybrid retrieval that combines BM25definitionA classic keyword-ranking method. It reliably finds exact part numbers and acronyms that embedding search can miss. with dense embeddings, and put a rerankerdefinitionA second model that sorts whatever the search pulled back. Search grabs plenty of near misses, and the reranker moves the passages that actually answer the question to the top. over the candidate set. Chunk along the document's structure instead of a fixed token count. Treat index freshness as an SLO with an owner, so someone gets paged when it goes stale.

Watch for embedding driftdefinitionSwitch embedding models without re-indexing and this is what you get. The stored vectors and the new queries no longer share a coordinate system. as well. Swap the embedding model without re-indexing the corpus and every stored vector now sits in a foreign coordinate system. None of this is research, and it's what turns recall you assume into recall you can state.

Without an eval harness, every change to a RAG system is a guess that a demo signed off on. Pull the golden setdefinitionA fixed collection of real questions from users, with known source passages and reference answers. It's what every system change is scored on. from actual user queries. What the team imagines users ask is a different, easier test. Score each question on both axes separately, or you'll spend a week tuning prompts to fix what's actually a chunkingdefinitionCutting documents into pieces before indexing. Cut by a fixed token count and a policy clause can end up in one chunk while its exceptions sit in the next one. bug.

LLM-as-judge scoring keeps the harness affordable, with the usual caveats. Calibrate the judge against a human-labeled sample and pin its version. Then audit the cases where it disagrees with people. Nobody should read a passing harness as proof the system is good. What it does catch is a change that made things worse, and it catches it before your users do.

# golden set runs on every change, like tests, because it is one
$ rag-eval run --golden-set v9 --diff main
retrieval  recall@5: 0.87  context precision: 0.74
generation faithfulness: 0.91  vs main: −0.04
regression: 12 questions worse · chunking suspected
✗ merge blocked · evidence attached to PR

What makes a RAG assistant dependable is everything around the model. Here's what we'd put in before anyone adds a second feature, roughly in the order we'd build it. Most of it is ordinary pipeline work.

We put an evaluation harness into NLP and LLM work from day one, so nobody has to take a demo on faith. The golden set and the CI gate usually come first. Then we fix retrieval permissions and ingestion around whatever model the team already picked. If your pilot looks fine in demos and nobody can say how often it's wrong, ask us to start there.