What’s in it
Thirty checks, grouped the way systems actually fail. Each one names the number it should produce, the threshold worth arguing about, and the failure it catches if you run it before your users do.
Retrieval quality — recall@k on your own questions rather than a public benchmark, chunk boundary audits, the embedding drift check nobody runs after a model upgrade, and what to do when hybrid search loses to plain BM25. The reasoning behind the boundary checks is in chunking is a retrieval decision.
Evaluation — the smallest eval set that still catches regressions, how to label it once and reuse it, judge agreement against a human sample, and the rule for when a score has moved and when it has only wobbled — the method from fifty labelled failures.
Cost and latency — p50, p95 and p99 measured separately, token accounting per request, the cache hit rate that decides your bill, and the timeout budget that keeps a slow retrieval from becoming a failed request. Start from the latency budget.
Failure modes — what the system does when the vector store is down, when the model returns nothing, and when a user pastes ten thousand words into a box sized for a sentence. The first of those has a post of its own.
How to use it
Run it as an audit on a system you already have in front of users. Anything that fails is a number, not an opinion, which is what makes it possible to argue about priorities with people who do not read the retrieval code.
It is the same list I work through on the first week of a production readiness review, minus the parts that only make sense with your codebase open.