RETRIEVAL-AUGMENTED GENERATION: WHERE IT WORKS, WHERE IT BREAKS, HOW TO FIX IT

Retrieval-Augmented Generation (RAG) is an architecture in which a system retrieves relevant documents from a knowledge base and feeds them to the model as context before it generates an answer. It lets a model reason over proprietary or recent information it was never trained on, but its usefulness rests entirely on choices about retrieval strategy, ranking, and fallback — get one of those wrong and the whole thing quietly breaks.

What RAG Actually Is

At its core, RAG is a four-step loop: a user asks a question, the system retrieves documents relevant to that question, the question and documents are passed to the model together, and the model composes an answer grounded in what it was given. The appeal is obvious — the model can cite information it never memorized, from last week's release notes to an internal policy handbook. The constraint is equally simple: the answer can only be as good as the documents retrieval manages to surface. RAG sits downstream of prompt engineering, because whatever gets retrieved becomes part of the context the model reads, and it exists precisely to extend a large language model beyond the boundary of its training data.

Where RAG Works

RAG is at its best when the knowledge base is high-quality, current, and reasonably small. If the underlying documents are accurate and well-organized, retrieval reliably finds what matters and the model answers from real, verifiable material — which is why customer support, internal question-answering, and product documentation are the canonical success stories. A corpus under roughly ten thousand documents is forgiving: precision stays high, you can afford to pull many candidates per query, and ranking barely has to work hard.

The other condition that favors RAG is the shape of the query. Exploratory, fact-seeking questions like "what is the refund policy?" map cleanly onto retrieval — there is a document that answers them. Analytical questions like "analyze our customer churn over the last six months" do not, because they require computation over data rather than the location of a passage. Recognizing which kind of question you are actually answering is often the difference between a system that feels magical and one that feels evasive.

Where RAG Breaks

The failure modes are the mirror image of the successes. As a corpus grows past roughly a million documents, vector-search precision degrades: a query returns a mix of genuinely relevant passages and plausible-looking noise, and the model is left to filter signal from distraction, which it does imperfectly. The retrieval architecture chosen at that scale is what separates a system that holds up from one that silently rots. Staleness bites just as hard — RAG is only ever as fresh as its corpus, so a knowledge base refreshed once a quarter answers with last quarter's truth.

The most dangerous failure is the empty retrieval. When a query has no relevant documents, the system faces a fork: admit it does not know, or fill the silence with a fabricated answer. Most systems, given no explicit instruction otherwise, choose to hallucinate by default — and a confident wrong answer is far more damaging than an honest "I don't know." Ambiguous queries create a milder version of the same problem, forcing the model to guess which of several partially-relevant documents the user meant. And even with perfect context in hand, models sometimes ignore what was retrieved and answer from training data instead, more often than most teams assume.

Deciding Whether To Use RAG

The decision itself is narrower than the hype suggests. It comes down to a short chain of questions:

Do you have proprietary or recent data needed to answer the question?
├─ YES: Is the knowledge base good (>90% accurate, < 6 months old)?
│  ├─ YES: Use RAG
│  └─ NO:  Fix the knowledge base first, then use RAG
└─ NO:  Don't use RAG. Rely on the model's training data or fine-tune.

The trap worth naming is adopting RAG because it is the default architecture of the moment rather than because the problem calls for retrieval. If the answer already lives in the model's training data, retrieval adds latency and cost while introducing a new surface for failure; the honest choice is to skip it.

Making RAG Hold Up

The techniques that keep RAG stable at scale are all about not trusting any single path. Rather than depending on vector search alone, robust systems run vector search, keyword search, and structured queries in parallel and rank across the combined results — hybrid ranking that weighs semantic similarity alongside document freshness, click-through, and relevance feedback rather than cosine distance in isolation. An explicit fallback chain handles the empty-retrieval case deliberately: broaden the search, then fall back to training knowledge with a stated caveat, and finally admit ignorance rather than inventing an answer.

Two disciplines make the difference between a system that improves and one that drifts. The first is a feedback loop that logs which retrieved documents users actually engage with and re-ranks accordingly over time. The second is verification: because a model will occasionally discard excellent context and answer from memory, a grounding check that confirms the answer is supported by what was retrieved is not optional (see verification). None of this is free — every extra document in the prompt carries an inference cost in latency and spend, so ranking well is also a budget decision. RAG is best understood not as a standalone feature but as one component of a larger system, and the fuller picture of how those components fit together is laid out in Building AI Systems That Actually Work.

Open Questions

• As corpora scale into the tens of millions of documents, can retrieval quality be maintained without hand-tuned hybrid ranking, or is fully learned end-to-end retrieval the only durable path?
• What is the right default when retrieval comes back empty — and can a system reliably learn its own confidence threshold for when to answer versus abstain rather than having it hand-set?
• As models grow context windows large enough to hold entire knowledge bases, where does the boundary between retrieval and raw long-context reasoning actually fall, and which problems still genuinely need a retrieval step?

Part of the knowledge graph at The Best Blog Ever — reference definitions for ideas that matter.