Embedding Pipeline Service  ·  View 22 of 22  ·  Assurance

Failure Classes and Their Handling

Seven of the thirteen named failure classes, chosen because each one changed a design decision — with the residual risk stated.

Editable source SVG draw.io All views
What fails How it shows Immediate handling Recovery Residual risk Model endpoint drift Same name, new weights Reference probe diverges Rollout failed, not pipeline Pinned digest re-pulled Undetected if probe set is unrepresentative Partial document Some chunks fail Version never marked visible Previous version serves Retry, then quarantine A document stuck one version behind Change-feed gap Events never emitted Sweep finds discrepancy Re-ingest from source Ledger reconciled Up to one sweep interval blind Bad index build Low recall or corrupt Recall gate fails Alias never flips Rebuild or roll back Gate blind to a shift the frozen set misses Vector tier down Qdrant unavailable Retrieval errors spike Lexical fallback, flagged Snapshot restore, RTO 4 h Semantic recall lost while degraded Erasure not confirmed A store does not ack Unconfirmed erasure alert Suppressed on read path Retry to completion Bytes present though unreachable Backlog after an outage Hours of edits queued Oldest-unembedded age climbs Lane priority, bulk shed Surge GPU capacity Standard lane breaches while draining Assurance — Failure Classes and Their Handling Six more classes are named in the requirement and handled the same way. These seven are the ones that have changed a design decision. v 1.0 · owner Data & AI Platform Architecture · date 2026-10

What this table is for

  • Each row names what fails, how it becomes visible, what happens immediately, how it recovers, and what remains wrong afterwards. The last column is the one usually missing.
  • Model endpoint drift justifies addressing replicas by digest. Partial document justifies atomic version visibility. Change-feed gap justifies the reconciliation sweep.
  • Six further classes in the requirement — source unavailable, poison chunk, orphaned vectors, migration abandoned, deletion propagation, permission failure — are handled by the same mechanisms.

The posture

  • Degrade a dimension rather than the service: freshness widens, retrieval goes lexical, re-ranking is skipped, telemetry samples — before anything returns an error.
  • A failure is never silently absorbed: quarantine carries a typed reason, erasure carries an unconfirmed alert, withholding carries a count.
  • No failure is answered by "it will catch up next sweep" where a user would notice in the meantime.

Residual risks worth naming

  • A frozen reference set that is unrepresentative lets model drift through the probe, and an unrepresentative evaluation set lets a worse contract through the gate. Both gates are only as good as their sets.
  • Erasure can be unreachable rather than absent while a retry is outstanding — correct on the read path, imperfect on the disk.
  • Draining a large backlog breaches the standard lane while protecting the interactive one. That is the intended ordering, and it is still a breach.
◀  Identity and Access — Authorising One Retrieval All views
Solution Architecture v1.0 · open source, self-hosted on Kubernetes · Data & AI Platform Architecture · 2026-10 · 22 views · 17 architecture decision records22-failure-classes