Embedding Pipeline Service  ·  View 08 of 22  ·  Structure

Platform Components

The container view: four boundaries, the components inside each, and the three external systems they talk to.

Editable source SVG draw.io All views
Capture and preparation Ingest Connector workers per source type Admission quotas, lanes Change log Kafka, 13 mo Reconciliation sweep Argo CronWorkflow Prepare Extraction workers sandboxed, bounded Chunker versioned Normalised text MinIO Chunk ledger PostgreSQL, Patroni Embedding and index build Inference Batcher 3 lanes Embedding fleet KubeRay + TEI Vector cache Redis Reference probe rollout gate Index Index builder per contract Vector index Qdrant Lexical index OpenSearch Snapshots MinIO, RTO 4 h Serving Query path Retrieval gateway Envoy Query embedder same contract Hybrid fusion Citation resolver offsets Access control ACL filter fails closed Suppression list 5 s p99 Control plane Authority Contract registry PostgreSQL Index catalogue aliases in etcd Migration orchestrator Argo Workflows Quality and cost Quality harness ClickHouse Drift monitors Cost attribution showback Corpus sources 4 kinds Permission authority query-time Product surfaces 14 teams change feed retrieval can read? alias Platform Components — Container View Interface / broker Application we own Queue / topic Security / platform Data store External / third party event / async synchronous Omitted for clarity: the observability stack, secrets distribution, and the warehouse export. Each appears on its own view. v 1.0 · owner Data & AI Platform Architecture · date 2026-10

Decisions

  • Extraction runs as its own sandboxed workload with no egress, because it is the least-trusted compute in the system and processes bytes a tenant uploaded.
  • The chunk ledger is a relational store with synchronous standby. It is the platform's own system of record, so it is the one data store that must not lose a write.
  • Access control is its own boundary inside Serving, holding the ACL filter and the suppression list, so revocation is a property of the read path rather than of indexing.

Build and buy

  • Bought as open source and operated: Kafka, PostgreSQL with Patroni, Redis, Qdrant, OpenSearch, MinIO, etcd, Argo Workflows, KubeRay, ClickHouse, Envoy, SPIRE, OpenBao.
  • Built: admission and lane assignment, the chunker, the index builder's contract enforcement, the alias resolver, the ACL filter and the quality harness.
  • The pattern: buy the storage and the transport, build the semantics. Everything built is something the architecture's correctness depends on.

Deliberately omitted

  • The observability stack, secrets distribution and the warehouse export: each has its own view rather than twelve more edges here.
  • The re-ranker, which is Phase 2 and would otherwise read as committed.
◀  Layered Architecture All views Integration Surface  ▶
Solution Architecture v1.0 · open source, self-hosted on Kubernetes · Data & AI Platform Architecture · 2026-10 · 22 views · 17 architecture decision records08-platform-components