I’m Nitish Kumar. I work on the part of an LLM product that nobody demos: the hundredth user, the 3am page, the retrieval that quietly stopped returning the right chunk three weeks ago.
The review
Most of what I publish here comes out of production-readiness reviews I run through Noetic. A review is two weeks: I read the system, measure it, and hand back the list of things that will break, ordered by when.
It covers the same four areas this site writes about:
- Retrieval quality — whether the right passage reaches the model at all, and what the boundaries of a chunk are doing to that.
- Evaluation — a labelled set built from your own failures, and a gate in CI that fails the build when a failure mode regresses.
- Cost and latency — p50, p95 and p99 per stage, and a written budget the pipeline can be held to.
- Failure modes — what the system does, and stops claiming, when a dependency is gone.
The 30-point checklist is the first week of that review, written down so you can run it yourself.
How this site works
Every figure on this site carries the date it was measured and, where there is code behind it, the commit it was measured at. If a number here is older than ninety days, the page says so rather than hiding it.
If you would rather start with the reasoning than the checklist, the writing is where it lives — why most hallucination is a retrieval miss is a good first read.
Get in touch
Email nitish@nits.dev with a line about the system you are running and where it hurts. The links below are where else I am.