Embedding Pipeline Service  ·  View 07 of 22  ·  Structure

Layered Architecture

Eight layers, and the asymmetry that lets most of them be down without retrieval failing.

Editable source SVG draw.io All views
Consumers Workspace search Ask-your-docs Similar documents Duplicate detection Serve Retrieval API p99 180 ms Query embedding Hybrid fusion vector + lexical ACL filter fails closed Suppression list 5 s revocation Index Index builder Vector index Qdrant Lexical index OpenSearch Alias resolver etcd Embed Batcher lane-aware Embedding fleet Ray Serve, TEI Vector cache Redis Reference probe frozen set Prepare Extract + OCR Tika, Tesseract Normaliser versioned Chunker versioned Chunk ledger PostgreSQL Normalised text MinIO, 30 d Capture Connector workers Change log Kafka Reconciliation sweep Lane admission 3 lanes Corpora Document service File store Comment threads Connected SaaS Control + ops Contract registry Index catalogue Quality harness ClickHouse Workload identity SPIRE Observability Prometheus which index changed chunks vectors accepted change Embedding Pipeline Service — Layered Architecture External / third party Interface / broker Application we own Security / platform Data store Queue / topic Decision point synchronous event / async Every layer below Serve may be down without retrieval failing: an index already built keeps serving. That asymmetry is why the control plane carries a lower availability target. v 1.0 · owner Data & AI Platform Architecture · date 2026-10

The asymmetry this view exists to show

  • Everything below Serve is a build path. An index that is already built keeps serving with capture, preparation and embedding all stopped.
  • That is why retrieval carries 99.95% monthly, ingest 99.9% and the control plane 99.5% (stated assumptions): a pipeline outage widens freshness rather than removing retrieval.
  • The one exception is the permission authority, which is on the hot path by design and fails closed — the single place the platform accepts a hard dependency.

Layer boundaries that matter

  • Capture never calls Prepare directly. The change log between them is what makes a backlog survivable and a replay possible.
  • Embed never writes to an index. The index builder owns contract enforcement, so a vector cannot land in an index built under a different contract.
  • The control plane spans every layer and is reached by none of them on a request path.

Why bands, and only here

  • This is the only view in the set drawn as stacked layers. Every other view uses the form its question needs — a context view drawn as bands reads as a layer diagram and answers nothing.