Confident, wrong answers: fixing retrieval before you blame the model

A RAG assistant sounds confident because a language model writes fluently over whatever context it is given, and it goes wrong when that context is missing or incorrect. Retrieval-augmented generation works in two moves: it retrieves passages related to the question, then generates an answer conditioned on them. The approach exists precisely because grounding output in retrieved text produces, in the words of the original RAG paper, "more specific, diverse and factual" language than a model working from its training data alone.

The failure mode is baked into the same mechanism. If retrieval surfaces the wrong passage, or half of the right one, the model still produces a clean, assured paragraph around it. Language models are, as a broad survey of hallucination puts it, "prone to hallucinate unintended text" — fluent, well-formed, and not supported by any source. A larger model writes an even more convincing version of the same wrong answer, which is why upgrading the model tends to raise costs without moving accuracy.

So the useful question is not how smart the model is - it is what the model was looking at when it answered. That shifts the investigation from the generation step, where the symptom appears, to the retrieval step, where most of the cause sits.

Before changing anything, capture what retrieval returned for a question that failed. For each wrong answer, log the passages the system pulled and read them against the answer. If the correct fact is absent from every retrieved passage, retrieval is the problem and no prompt change will save it. If the fact is present and the model still got it wrong, the issue is in generation or in how the passages were ranked.

This is measurable, not a matter of intuition. Microsoft's guidance on evaluating RAG systems defines groundedness — whether the answer is supported by the retrieved context — and relevance — whether retrieval surfaced the right context — as distinct evaluators, run before production and monitored after. Scoring the two separately tells you which half of the pipeline to fix. The common symptoms map cleanly to causes:

Two of these rows point at siblings in this series — keeping an index in sync with changing content, and adding keyword search for exact identifiers — which are covered in their own articles. The rest come down to the three retrieval failures below.

Most retrieval misses trace to one of three things: how documents were split, how they were labelled, and how results were ordered. Each is fixable without touching the model.

Chunking that breaks the meaning

Documents get split into chunks before they are embedded and indexed, and where you cut matters. A chunk that ends mid-table or mid-clause hands the model half a fact, and a chunk that blends three unrelated sub-topics dilutes the vector so the right passage never ranks. Microsoft's note on chunking documents for vector search makes the point that content "poorly represented as a single vector" does better when chunked at a finer grain that respects the document's structure. Splitting on sections and sentences, rather than a fixed character count, removes a large share of wrong answers on its own.

Metadata the ranker can actually use

A passage with no source, date, or section tag is a passage the system cannot reason about. Without a date it cannot prefer the current policy over the retired one; without a section or document type it cannot scope a search or apply access rules. Attaching that metadata at indexing time gives retrieval the filters it needs to return the right passage rather than a plausible neighbour.

Ranking that buries the right passage

Sometimes the correct passage is retrieved and then lost, ranked below near-duplicates or generic boilerplate. Semantic similarity alone rewards passages that sound like the question, which is not the same as passages that answer it. A reranking step, or a blend of keyword and semantic search, pushes the passage that actually contains the fact back to the top; the keyword-and-hybrid case has its own article in this series.

Losing user trust to confident, wrong answers? Have your retrieval diagnosed by engineers who have brought internal assistants back from exactly this. A short diagnostic on real failing questions usually shows whether the model or the retrieval is at fault, before any budget goes to a larger model.

Put a retrieval check in front of generation, so quality is measured rather than assumed. The tool is a small evaluation set: real questions paired with the passages that should be retrieved and the answer a person would accept. You run retrieval against it and score two things — whether the right passages come back, and whether the generated answer stays grounded in them.

A useful set is blunt and quick to build:

• Twenty to fifty real questions, including the ones that already failed.
• For each, the source passage that contains the correct answer.
• A short note on what a correct, grounded answer looks like.

Run that set on every change to chunking, metadata, ranking, prompt, or model, and a regression shows up as a score drop instead of a user complaint weeks later. Microsoft's built-in RAG evaluators support exactly this rhythm: evaluate against the set before production, then keep the same checks running as monitors afterward. The set costs a day or two to assemble, which is far less than the cost of quietly shipping a worse retriever.

Diagram showing a RAG pipeline where a question passes through retrieval, chunking, metadata, ranking, retrieval evaluation, generation, and answer citations, with fixes looping back to retrieval.

Show the source behind each answer, so a wrong retrieval becomes visible instead of silent. When every claim carries a link to the passage it came from, a reader can check it in one click, and a bad pull is caught at the point of use rather than discovered after a decision. Google Cloud describes this in its account of RAG and grounding: grounded responses can have "sources attached" to individual sentences, giving support for each stated claim.

Citations do more than expose errors; they rebuild the trust that confident wrong answers destroy. An assistant that says where its answer came from invites verification, and an assistant users can verify is one they keep using. The habit also disciplines the system, because a claim with no retrievable source is a signal that the answer was generated rather than grounded, and worth suppressing or flagging.

Sometimes retrieval is clean and the answer is still wrong, and it is worth naming that case so the diagnosis stays honest. If the correct passage was retrieved, ranked at the top, and the model still misread or contradicted it, the fix is in generation: a clearer prompt, a model better suited to the task, or a structured output the model must fill from the passage. Reranking and better chunking will not help there, and chasing them wastes time.

This is why the diagnostic step comes first. It keeps you from tuning retrieval when the model is at fault, and from swapping models when retrieval is starving them. Most of the time the evidence points at retrieval, but the point is to follow the evidence rather than the reflex.

• A confident tone comes from the language model's fluency and says nothing about whether the underlying fact is correct.
• For any wrong answer, log the retrieved passages first; if the fact is absent from them, the problem is retrieval, not the model.
• Most retrieval misses trace to chunking that breaks meaning, missing metadata, or ranking that buries the right passage.
• A small evaluation set of real questions and their source passages catches a worse retriever before users do.
• Citations users can open turn silent retrieval errors into visible ones and rebuild trust in the assistant.

An assistant that answers with confidence and gets the fact wrong is a retrieval story far more often than a model story. The model is doing what it was built to do, which is write fluent text over the context it receives; the context is where the answer is won or lost. Fixing chunking, metadata, and ranking, then grounding answers in openable sources, addresses the cause instead of the symptom, and it does so without the recurring cost of a larger model.

The order is the whole point: diagnose, fix retrieval, verify, and only then look at generation. If your internal assistant has lost users to confident, wrong answers, RAG and knowledge-assistant development from our team covers the retrieval diagnosis, the fixes, and the evaluation harness that keeps them in place, with the work done alongside your own engineers.