The eval set most teams build is generated: feed the corpus to a model, ask for a thousand question-and-answer pairs, score against them nightly. It produces a number that goes up, a dashboard nobody reads, and no warning at all on the morning a prompt change breaks identifier lookups.
The eval set that works is smaller, uglier and hand-made. Fifty items, each one a question a real person asked and the system got wrong, labelled with why.
Start from failures you already have
You do not need a labelling budget. You need a week of production traffic, the retrieval logs from the last post, and an afternoon.
Pull every interaction where a user rephrased their question, abandoned the session, or hit the thumbs-down. Those three signals find most of your failures without anyone reading the whole log. For each one, write down the question, the answer you gave, and the answer you should have given.
Why not generate them? A generated question is written from a chunk that exists, in the vocabulary of the document. Your users write from the outside, in their own vocabulary, about things your corpus may not cover at all. The failure modes that matter live in that gap, and a generator cannot reach it.
Label by failure mode, not by score
A single quality score tells you something broke. A failure mode tells you which team fixes it. Four labels carry most of the weight:
retrieval_miss— the answer was not in the retrieved context.grounding_failure— it was there, the model contradicted it. Telling these first two apart is most of the work, and it is usually the first one.refusal— the system declined something it could have answered.out_of_scope— the corpus genuinely does not contain the answer, and the correct behaviour is to say so.
That last one is not a failure of the system; it is a failure of the test set if you score it as one. Keeping it as an explicit label is what stops your eval from rewarding a system that bluffs.
from __future__ import annotations
import json
from dataclasses import asdict, dataclass
from enum import StrEnum
from pathlib import Path
class FailureMode(StrEnum):
RETRIEVAL_MISS = "retrieval_miss"
GROUNDING_FAILURE = "grounding_failure"
REFUSAL = "refusal"
OUT_OF_SCOPE = "out_of_scope"
@dataclass(frozen=True, slots=True)
class EvalItem:
id: str
question: str
gold_chunk_ids: tuple[str, ...]
expected: str
mode: FailureMode
added_on: str # when this failure was first seen
source: str = "production" # production | support | synthetic
def is_answerable(self) -> bool:
return self.mode is not FailureMode.OUT_OF_SCOPE
def load(path: Path) -> list[EvalItem]:
rows = json.loads(path.read_text())
return [EvalItem(**{**row, "gold_chunk_ids": tuple(row["gold_chunk_ids"])}) for row in rows]
def save(items: list[EvalItem], path: Path) -> None:
path.write_text(json.dumps([asdict(item) for item in items], indent=2, sort_keys=True))
Keep the file in the repository, next to the code it tests. It is a test fixture, it belongs in review, and a diff on it should be as readable as a diff on a migration.
The runner is two functions
Retrieval and generation fail for different reasons, so score them separately. A combined score hides which half moved.
from dataclasses import dataclass
from typing import Callable, Sequence
from evals.dataset import EvalItem, FailureMode
Retriever = Callable[[str, int], Sequence[str]]
Answerer = Callable[[str, Sequence[str]], str]
Judge = Callable[[str, str, str], bool] # question, expected, actual -> passes
@dataclass(frozen=True, slots=True)
class Report:
retrieval_recall: float
answer_accuracy: float
refusal_rate: float
by_mode: dict[str, float]
def summary(self) -> str:
modes = " ".join(f"{mode} {score:.2f}" for mode, score in sorted(self.by_mode.items()))
return (
f"recall {self.retrieval_recall:.2f} "
f"accuracy {self.answer_accuracy:.2f} "
f"refusals {self.refusal_rate:.2f}\n{modes}"
)
def evaluate(
items: Sequence[EvalItem],
retrieve: Retriever,
answer: Answerer,
judge: Judge,
k: int = 10,
) -> Report:
retrieval_hits = 0
answer_hits = 0
refusals = 0
per_mode: dict[str, list[bool]] = {}
for item in items:
chunks = list(retrieve(item.question, k))
found = bool(set(chunks) & set(item.gold_chunk_ids))
retrieval_hits += found and item.is_answerable()
response = answer(item.question, chunks)
declined = response.strip().lower().startswith(("i don't", "i do not", "no answer"))
refusals += declined
passed = (
declined
if item.mode is FailureMode.OUT_OF_SCOPE
else judge(item.question, item.expected, response)
)
answer_hits += passed
per_mode.setdefault(item.mode, []).append(passed)
answerable = [item for item in items if item.is_answerable()] or items
return Report(
retrieval_recall=retrieval_hits / len(answerable),
answer_accuracy=answer_hits / len(items),
refusal_rate=refusals / len(items),
by_mode={mode: sum(results) / len(results) for mode, results in per_mode.items()},
)
The out_of_scope branch is the important line in that function. For those
items, declining is the pass condition. Without it, every eval quietly
teaches your system to answer questions it has no business answering.
Check the judge before you trust the judge
If a model grades the answers, its agreement with you is a number you have to know. Sample thirty items, grade them yourself, and compare:
from typing import Sequence
def cohens_kappa(human: Sequence[bool], model: Sequence[bool]) -> float:
"""Agreement above chance. Below ~0.6, the judge is not usable as a gate."""
n = len(human)
observed = sum(h == m for h, m in zip(human, model)) / n
human_true = sum(human) / n
model_true = sum(model) / n
expected = human_true * model_true + (1 - human_true) * (1 - model_true)
return (observed - expected) / (1 - expected) if expected < 1 else 1.0
Re-run that whenever you change the judge prompt or the judge model. A judge that drifts turns every other number in the report into decoration.
Make it a gate, not a report
An eval that emails a score is an eval that gets ignored by March. Run it in CI, on every pull request that touches the pipeline, and fail the build when a mode regresses.
import { readFile } from "node:fs/promises";
type Report = {
retrieval_recall: number;
answer_accuracy: number;
refusal_rate: number;
by_mode: Record<string, number>;
};
/** How far a mode may fall before the build fails. Tightened as the set grows. */
const TOLERANCE = 0.05;
async function readReport(path: string): Promise<Report> {
return JSON.parse(await readFile(path, "utf8")) as Report;
}
const [baseline, current] = await Promise.all([
readReport("evals/baseline.json"),
readReport("evals/current.json"),
]);
const regressions = Object.entries(current.by_mode).filter(([mode, score]) => {
const before = baseline.by_mode[mode];
return before !== undefined && score < before - TOLERANCE;
});
for (const [mode, score] of regressions) {
console.error(`✗ ${mode}: ${baseline.by_mode[mode].toFixed(2)} → ${score.toFixed(2)}`);
}
if (current.refusal_rate > baseline.refusal_rate + TOLERANCE) {
console.error(`✗ refusal rate rose to ${current.refusal_rate.toFixed(2)}`);
regressions.push(["refusal_rate", current.refusal_rate]);
}
if (regressions.length > 0) {
console.error("\nEval gate failed. Either fix the regression or move the baseline on purpose.");
process.exit(1);
}
console.log("✓ no mode regressed");
Either fix it or move the baseline on purpose is the whole discipline. A baseline that moves in the same commit as the change, with a sentence in the message saying why, is a decision. A baseline that drifts silently is how a system gets worse over six months while every dashboard stays green.
When has a score actually moved?
On fifty items, one item is two points. Before you celebrate or panic, ask whether the difference survives resampling:
import random
from typing import Sequence
def bootstrap_interval(
results: Sequence[bool],
rounds: int = 2_000,
confidence: float = 0.95,
) -> tuple[float, float]:
"""A crude confidence interval for a pass rate on a small set."""
n = len(results)
means = []
for _ in range(rounds):
sample = random.choices(results, k=n)
means.append(sum(sample) / n)
means.sort()
tail = (1 - confidence) / 2
return means[int(tail * rounds)], means[int((1 - tail) * rounds)]
If the intervals overlap, the score did not move. Say so out loud in the pull request; it saves the next person a week of chasing a difference that was sampling noise wearing a lab coat.
Keeping the set honest
Three habits, and the set stays useful for years:
- Add every new production failure, the week it happens, with the date.
- Never delete an item because it is hard. Move it to a
known_failinglist with a reason, so the count of known failures is itself a metric. - Re-read the oldest ten items each quarter. Corpora change; some items stop being correct, and an eval set that asserts last year’s answer is worse than no eval set at all.
Fifty items maintained this way will tell you more on a Tuesday afternoon than a thousand generated ones ever did.
