What your assistant does when the vector store is down

Ask a team what their assistant does when the vector store is unreachable and you get one of two answers. The first is “it returns an error”, which is at least a decision. The second, far more common, is a pause — and then someone opens the code and discovers that an empty result set is treated exactly like a successful search that found nothing relevant. The model is handed no context and asked to answer anyway — the same retrieval miss that gets reported as hallucination, except now it happens to every request at once.

It answers. Fluently. That is the failure mode that costs you a customer: not the outage, but the confident paragraph produced during it.

Decide what degraded means before you need it#

Write the table before the incident. For each dependency, two columns: what the user sees, and what the system stops promising.

The right-hand column is the one that matters and the one that never gets written. Every product claim your interface makes — grounded in your documents, sourced, up to date — is a promise that some dependency is keeping. When that dependency is gone, the claim has to go with it, in the same deploy, automatically.

The rule. A degraded answer must look different to a healthy one. If your users cannot tell the difference, neither can your support team, and the incident review will be archaeology.

Timeouts first, then a breaker#

A timeout protects one request — and the deadlines should come from a written latency budget, not from whatever the client library defaults to. A circuit breaker protects the system from spending its entire capacity on a dependency that is already gone.

type State = "closed" | "open" | "half-open";

export interface BreakerOptions {
  failureThreshold: number;   // consecutive failures before opening
  resetAfterMs: number;       // how long to stay open before a trial call
  timeoutMs: number;          // per-call deadline
}

export class CircuitOpenError extends Error {
  constructor(readonly dependency: string) {
    super(`${dependency} circuit is open`);
  }
}

export class Breaker {
  private state: State = "closed";
  private failures = 0;
  private openedAt = 0;

  constructor(
    private readonly dependency: string,
    private readonly options: BreakerOptions,
    private readonly onStateChange?: (dependency: string, state: State) => void,
  ) {}

  get isOpen(): boolean {
    return this.state === "open" && Date.now() - this.openedAt < this.options.resetAfterMs;
  }

  async call<T>(run: (signal: AbortSignal) => Promise<T>): Promise<T> {
    if (this.isOpen) {
      throw new CircuitOpenError(this.dependency);
    }

    if (this.state === "open") {
      this.transition("half-open");
    }

    const controller = new AbortController();
    const timer = setTimeout(() => controller.abort(), this.options.timeoutMs);

    try {
      const value = await run(controller.signal);
      this.failures = 0;
      if (this.state !== "closed") {
        this.transition("closed");
      }
      return value;
    } catch (error) {
      this.failures += 1;
      if (this.failures >= this.options.failureThreshold || this.state === "half-open") {
        this.openedAt = Date.now();
        this.transition("open");
      }
      throw error;
    } finally {
      clearTimeout(timer);
    }
  }

  private transition(next: State): void {
    this.state = next;
    this.onStateChange?.(this.dependency, next);
  }
}

Wire onStateChange to whatever writes your logs. The transition to open is the single most useful line your system will emit during an incident: it is the moment the system stopped trusting a dependency, and it will be minutes ahead of the first customer report.

A fallback that is allowed to say less#

The fallback path has one job: answer what it honestly can, and be visibly smaller than the healthy path. It is not a second-rate copy of the main pipeline, and it should never reach for the model to paper over the gap.

from dataclasses import dataclass
from enum import StrEnum
from typing import Protocol, Sequence


class Mode(StrEnum):
    HEALTHY = "healthy"
    KEYWORD_ONLY = "keyword_only"
    PASSAGES_ONLY = "passages_only"
    UNAVAILABLE = "unavailable"


@dataclass(frozen=True, slots=True)
class Answer:
    text: str
    citations: tuple[str, ...]
    mode: Mode

    @property
    def is_grounded(self) -> bool:
        """The interface may only claim 'from your documents' when this is true."""
        return self.mode in (Mode.HEALTHY, Mode.KEYWORD_ONLY) and bool(self.citations)


class Search(Protocol):
    def __call__(self, question: str, k: int) -> Sequence[str]: ...


def answer(
    question: str,
    vector_search: Search | None,
    keyword_search: Search | None,
    generate: "Generate | None",
    k: int = 8,
) -> Answer:
    """Each dependency that is missing removes a claim, never adds a guess."""
    chunks: Sequence[str] = ()
    mode = Mode.HEALTHY

    if vector_search is not None:
        chunks = vector_search(question, k)
    elif keyword_search is not None:
        chunks, mode = keyword_search(question, k), Mode.KEYWORD_ONLY

    if not chunks:
        return Answer(
            text=(
                "I could not search the documents just now, so I would rather not "
                "guess. Try again in a few minutes, or contact support."
            ),
            citations=(),
            mode=Mode.UNAVAILABLE,
        )

    if generate is None:
        return Answer(
            text="Search is working but summarising is not. Here is what I found:",
            citations=tuple(chunks[:3]),
            mode=Mode.PASSAGES_ONLY,
        )

    return Answer(text=generate(question, chunks), citations=tuple(chunks[:3]), mode=mode)

Notice what the function never does: call the model with an empty context. That single prohibition removes the majority of confident-nonsense incidents, and it costs one if.

Never let a failed retrieval look confident#

Three enforcement points, in increasing order of how much they save you:

1. At the prompt. When context is empty, do not send the prompt at all. A system prompt saying “only answer from the context” is guidance; not calling the model is a guarantee.
2. At the response. Carry the mode all the way to the interface, and render a degraded answer differently — a banner, a different label on the sources block, no citation chips.
3. At the API boundary. Put mode in the response body. Every integrator downstream then gets to make their own decision, and your own mobile client stops being the only place the distinction exists.

If you do only one, do the first.

Alert on the promise, not the process#

Most alerting for retrieval systems watches infrastructure: the vector store’s CPU, the queue depth, the error rate of a container. Those tell you a component is unwell. They do not tell you the product broke its promise.

Alert on these instead:

• degraded share — requests served in any mode other than healthy, as a share of all requests, over five minutes;
• empty-context rate — retrievals returning nothing, which climbs long before a dependency fails outright;
• breaker transitions — every open, by dependency;
• citation rate — answers carrying at least one source. A silent drop here is the earliest signal that grounding has stopped working while everything else looks healthy.

That last one catches the outage that has no outage: a re-index that wrote an empty collection, a credential rotation that quietly downgraded a search client to returning nothing, a schema change that broke the filter. No error rate moves. The citation rate falls off a cliff.

Rehearse it#

The fallback path you have never run is a hypothesis. Run it deliberately:

• a test that boots the service with the vector store pointed at a dead host, and asserts the response mode is unavailable and no model call was made;
• a staging switch that opens each breaker by hand, so support has seen the degraded interface before a customer describes it to them;
• one scheduled game day a quarter, where someone pulls a dependency and the rest of the team works out which alert fired first.

You will find the same class of bug every time: something in the chain treats empty as fine. Finding it on a Thursday afternoon with a stopwatch is considerably cheaper than finding it at 3am with a customer on the phone.