The support ticket arrives at 09:14 on a Tuesday. The user asks about refund processing timelines. The assistant responds in full sentences. It cites a policy. It is polite, it is confident, and it is wrong. The refund window it cites does not exist in the documentation. The user replies: "This is incorrect. Check section 4."
You open the logs. The model received a context window. That window was empty of the correct answer. It answered from nothing, or from a nearby, irrelevant paragraph that shared similar vocabulary.
The usual diagnosis is a failure of the large language model. The team proposes a swap. A larger model. A more expensive tier. A better prompt. These are not wrong solutions, but they are the last thing you should do. If the retriever has not found the document, the generator cannot invent it from the absence. I have seen teams spend weeks tuning the prompt of a system that was simply failing to find the right chunk. The model was a high-fidelity speaker in a room with no book on the table.
This article is about the specific stage between the search index and the prompt template. It is about the second pass. Reranking is often treated as an optimisation to be added later, a luxury once the baseline is working. In production, it is often the first place a failure hides. If your system is live and answering confidently but incorrectly, the bottleneck is almost certainly retrieval, not generation. This is the premise that runs through Your RAG isn't hallucinating. Your retriever is missing.. We are isolating the bottleneck before touching the LLM prompt or swapping model tiers.
Why reranking is the first variable to test
Two-stage retrieval is the standard architecture for any serious RAG system. The first stage is broad. It is a vector similarity search or a BM25 keyword match. It is designed to be fast. It returns the top fifty or top one hundred documents that might be relevant. This stage is imprecise. It is a net that casts wide to ensure it does not miss the needle.
The second stage is a precision filter. This is the reranker. It takes the broad list from the first stage and reorders it based on semantic relevance to the specific query. It uses more sophisticated scoring, often cross-attention mechanisms, to understand the relationship between the query and the document in depth. It is slower. It is more accurate. Its job is to take the top fifty and say: "No, this one is number one, and this one is number three."
Here is where the failure mode lives. If the correct document is in the top twenty of the first stage, but it drops out of the top three passed to the LLM, something between the two stages has failed. Either the reranker scored it below where the first stage had it, or it lifted it but not far enough to make the cut. A larger language model will not fix this. You cannot prompt your way into a better ranking. The model only sees what you hand it. If you hand it the wrong chunk, it will write a confident, fluent paragraph about the wrong chunk.
The common practice is to assume that if the first stage returns a document, the second stage will handle the selection. This is rarely true. The second stage is a separate model with its own biases, its own training data, and its own definition of relevance. It is a different instrument. It does not share the same understanding of the corpus as the embedder that produced the initial vectors.
How to isolate the failure using logs
You cannot fix what you cannot see. Most teams log the final output. They log the prompt and the response. This is insufficient. You need to log the intermediate states. Specifically, you need to log the IDs of the chunks returned by the first stage, and the IDs of the chunks returned by the second stage.
The diagnostic workflow is simple.
First, take a set of known-good queries. These are questions where you know, with certainty, which chunk contains the answer. You do not need a thousand of them. You need ten. Ten queries where the ground truth is unambiguous.
Second, run these queries through your system.
Third, look at the Stage 1 logs. Is the correct chunk in the top fifty? If it is not, the problem is upstream. It is your embeddings, your chunking, or your indexing. The reranker is not at fault because it never saw the document. It cannot rank what it does not have.
Fourth, look at the Stage 2 logs. Is the correct chunk in the top three? If it was in the top fifty in Stage 1, but is not in the top three in Stage 2, look at the reranker next. Missing the cut does not prove it did harm: compare the chunk’s rank before and after reranking. If it moved down, the reranker’s idea of relevance disagrees with the actual relevance of the document to the query.
This isolation prevents blind tuning. It tells you exactly where the failure is. If the document is missing from Stage 1, stop looking at the reranker. Look at the chunking. As I argued in Chunking is a retrieval decision, not a preprocessing step, the form of the document determines its discoverability. If the document is in Stage 1 but missing from Stage 2, stop looking at the chunking. Look at the reranker.
Measuring the impact: A Python snippet for hit@k
You need a number. "It feels slow" is not a metric. "It feels wrong" is not a metric. You need hit@k. This is a boolean indicator of whether the correct document is in the top k results.
Below is a short Python snippet that demonstrates how to calculate this. It assumes you have a list of ground-truth chunk IDs for a given query, and a list of returned chunk IDs from both stages.
def hit_at_k(ground_truth_ids: set, returned_ids: list, k: int) -> bool:
"""True if any ground-truth chunk ID is in the top k returned IDs."""
return bool(ground_truth_ids & set(returned_ids[:k]))
hit_at_k({"chunk_5", "chunk_9"}, ["chunk_9", "chunk_2", "chunk_7"], 3) # True
hit_at_k({"chunk_5", "chunk_9"}, ["chunk_2", "chunk_7", "chunk_1"], 3) # False
hit_at_k({"chunk_5", "chunk_9"}, ["chunk_2", "chunk_9", "chunk_7"], 1) # False
This code is simple. It does not measure "quality." It measures presence. Did the right document make the cut? Run this over your ten known-good queries. If your Stage 1 hit@50 is low, your retriever is broken. If your Stage 1 hit@50 is high, but your Stage 2 hit@3 is low, your reranker is broken.
This is the metric that tells the truth. It is binary, and it is traceable to the specific chunks returned. You do not need a complex evaluation harness for this. You need a loop and a log.
Common pitfalls in reranker implementation
The most common pitfall is assuming that the reranker is a drop-in replacement for the initial similarity score. It is not. It is a different model. It expects a specific input format. Usually, it expects the query and the document to be fed into a cross-encoder, where it can see the interaction between every token in the query and every token in the document.
If you are using a bi-encoder for retrieval and a cross-encoder for reranking, you must ensure the text is formatted correctly. The cross-encoder is sensitive to the order and the concatenation of the query and the document. A slight change in the delimiter, or the inclusion of extra metadata in the document text, can alter the score.
Another pitfall is latency. Rerankers are computationally expensive. They are slow. If your p99 latency budget is tight, adding a heavy reranker might degrade the user experience even if accuracy improves. This is the trade-off I discussed in Write the latency budget before you write the cache. You cannot add a stage to the pipeline without accounting for the time it takes. You must measure the cost of the precision you are buying.
A specific failure mode here is passage truncation. Every cross-encoder has a maximum input length, shared between the query and the chunk; the figure is on your reranker’s model card. If a chunk is longer, the reranker silently cuts it, and anything past the limit is never seen. It will score based on the beginning of the chunk, which may be irrelevant. This is a silent failure. The chunk is present, the reranker runs, but the score is based on truncated text. Check your chunk lengths against your reranker’s max input limit.
What to log and measure this week
This is the actionable part. You can do this this week. You do not need a new team. You do not need a new model. You need a log line and a spreadsheet.
- Add a log line for Stage 1. In your retrieval code, after the initial vector search, log the IDs of the top fifty chunks. Do not log the full text, just the IDs. This is enough to identify the chunk later.
- Add a log line for Stage 2. After the reranker, log the IDs of the top three chunks.
- Create a test set. Pick ten queries from your production traffic that you know are answered correctly. For each of these ten queries, identify the specific chunk ID that contains the answer. This is your ground truth.
- Run the diagnostic. Send these ten queries through your system. For each query, check the Stage 1 logs. Is the ground-truth chunk in the top fifty? Check the Stage 2 logs. Is it in the top three?
- Analyse the rank movement. If the chunk is in Stage 1 but not Stage 2, look at its rank before and after. If the reranker moved it down, the reranker is harmful. If it moved it up, but not into the top three, widen the top-k passed to the prompt or fix the first stage.
This process will take an afternoon. It will tell you where the failure is. It is the most efficient way to diagnose a RAG system that is live but misbehaving.
If you want a structured way to go through these checks, the 30-point production readiness checklist covers retrieval, evals, cost and latency. It includes the specific metrics you need to log to diagnose retrieval failures, each with the number it should produce and the failure it catches.
When to stop and look at chunking
There is a limit to what reranking can do. A reranker is a precision filter. It can only work on what the recall stage provides. If the correct information is never in the top-50 of Stage 1, the problem is not reranking. It is that the chunk doesn’t exist in a retrievable form.
This is where chunking becomes a retrieval decision. If your chunks are too large, they may contain too much noise, diluting the signal. If they are too small, they may lack context. If they are split in the wrong place, the semantic meaning may be broken. The embedder represents the text as it is. If the text is broken, the embedding is broken.
I have written about this in detail in Chunking is a retrieval decision, not a preprocessing step. The point is that you cannot have a good reranker if your chunks are bad. The reranker is the second link in a chain. If the first link is weak, the whole chain fails.
When you look at your diagnostics, and you see that the correct chunk is missing from Stage 1, stop looking at the reranker. Start looking at the chunk boundaries. Look at the content. Look at the length. Look at the split points. This is where the real work is.
The goal is not to have the most sophisticated model. The goal is to have the right document in the context window. Everything else is decoration. Measure the retrieval. Fix the retrieval. Then, and only then, worry about the generation.
If you are struggling with retrieval quality, go look at your logs. The answer is there. It is just not where the model thinks it is.
