Your RAG isn't hallucinating. Your retriever is missing.

A team shows me a support assistant that invents a refund policy. The ticket says hallucination. The fix on the roadmap says better model. Neither is right, and the evidence is usually one query away.

Ask what the model was actually given#

Before touching the prompt, log the retrieved chunks alongside every bad answer. In the reviews I run, the split is consistent: the model was never shown the right passage in roughly three cases out of four. It did not invent an answer over good evidence — it wrote fluent prose over nothing.

AnswerLog::create([
    'question' => $question,
    'chunk_ids' => $chunks->pluck('id'),
    'answer' => $answer,
]);

Once that log exists, the two failure modes separate cleanly.

• Retrieval miss. The correct passage is not in the context at all. A bigger model cannot fix this, and a longer prompt makes it worse.
• Grounding failure. The correct passage is right there and the model contradicted it. This one is a prompting and evaluation problem.

There is a third case worth naming so it stops polluting the other two: the corpus simply does not contain the answer. The correct behaviour there is to say so, and a system that answers anyway is failing at grounding, not at search.

Telling them apart, one bad answer at a time#

The log tells you what the model saw. To call something a miss you also need to know what it should have seen, which means a person opening the source documents and marking the chunk that answers the question. That is the labelling step from fifty labelled failures, and it is the only expensive part. The sorting itself is a few lines:

from typing import Callable, Sequence

from evals.dataset import FailureMode

Retriever = Callable[[str, int], Sequence[str]]


def triage(
    question: str,
    shown_ids: Sequence[str],
    gold_chunk_ids: frozenset[str],
    retrieve: Retriever,
    depth: int = 50,
) -> tuple[FailureMode, int | None]:
    """Name the failure, and for a miss, say where the answer actually ranked."""
    if not gold_chunk_ids:
        return FailureMode.OUT_OF_SCOPE, None

    if gold_chunk_ids & set(shown_ids):
        return FailureMode.GROUNDING_FAILURE, None

    rank = next(
        (i for i, chunk_id in enumerate(retrieve(question, depth), start=1)
         if chunk_id in gold_chunk_ids),
        None,
    )
    return FailureMode.RETRIEVAL_MISS, rank

The rank is the part people skip, and it is the part that tells you what to build. A miss where the answering chunk sat ninth, just below a top-six cut, is an ordering problem: a reranker or a slightly wider window will usually recover it. A miss where the answer appears nowhere in the top fifty is a recall problem, and no amount of reordering will find a chunk the retriever never returned. Those two fixes cost very different amounts, so it is worth knowing which one you have before choosing.

Why the miss happens#

Pure vector search is a similarity function, not a search engine. It has no notion of an exact string, so a policy number, an SKU or a customer name is matched on vibes. Add a keyword index beside it and the miss rate on that class of query drops sharply.

The other common cause is upstream of the retriever entirely. If the identifier and its meaning were split into different chunks at ingestion, neither chunk is findable by the question a person would ask — which is why chunking is a retrieval decision and not a preprocessing step.

Combining two retrievers is simpler than it sounds. Their scores live on different scales — a cosine similarity and a BM25 score cannot be added — so merge them by rank instead:

from collections import defaultdict
from typing import Sequence


def reciprocal_rank_fusion(*rankings: Sequence[str], k: int = 60) -> list[str]:
    """Merge ranked lists by position, never by raw score.

    A chunk both retrievers rank highly beats one that only a single
    retriever put first, which is exactly the behaviour you want.
    """
    scores: dict[str, float] = defaultdict(float)
    for ranking in rankings:
        for rank, chunk_id in enumerate(ranking, start=1):
            scores[chunk_id] += 1 / (k + rank)
    return sorted(scores, key=scores.__getitem__, reverse=True)

The k of 60 is the constant from the paper that introduced the method. It flattens the top of each list so that one retriever’s first place cannot drown out the other’s agreement. Leave it alone until your own labelled set gives you a reason to move it.

Negation stays weak in both, which is worth knowing before anyone promises it. “Plans that do not include support” and “plans that include support” share almost every word, so they embed close together and score alike under BM25. If negated questions matter in your product, treat them as their own class in the eval set and handle them with query rewriting or metadata filters, not with a better retriever.

Grounding failures are a different job#

When the right passage was in the context and the answer still contradicted it, the retriever did its job and the fix lives elsewhere. Two changes carry most of the weight.

First, make the model cite the chunk behind each claim, and check those citations in code rather than trusting them:

from typing import Sequence


def unsupported_citations(cited_ids: Sequence[str], shown_ids: Sequence[str]) -> set[str]:
    """Chunks the answer claims to rely on that it was never actually shown."""
    return set(cited_ids) - set(shown_ids)

An answer citing a chunk it never saw is a grounding failure you can catch before the user does, with a set difference.

Second, grade grounding with a judge whose agreement with a human you have measured. An unchecked judge will happily score a confident contradiction as a pass — check the judge before you trust it.

Why a bigger model makes it worse#

Upgrading the model is the most common response to a hallucination ticket, and for a retrieval miss it is the wrong direction. A more capable model, handed the same six passages without the answer in them, does not find the one that is missing. It writes a more persuasive answer from what it was given. The failure gets harder to spot, not rarer — the opposite of what the upgrade was bought for.

The limiting case is a context with nothing in it at all, which is what every request gets when the vector store goes down. That one deserves a design of its own: what your assistant does when the vector store is down.

What to do this week#

1. Log retrieved chunks with every answer. One migration, one line.
2. Label fifty bad answers by failure mode. An afternoon.
3. For every miss, record the rank the answering chunk actually sat at.
4. If misses dominate, add keyword retrieval before touching the model — and if they cluster just below the cut, try a reranker first.

The order matters. Every hour spent on prompt wording while the retriever is missing is an hour spent teaching the model to guess more confidently.