Embedding Pipeline Service  ·  View 09 of 22  ·  Structure

Integration Surface

Three inbound surfaces with three different availability targets, and the dependencies each of them reaches.

Editable source SVG draw.io All views
Consumers and producers Workspace search Ask-your-docs Product engineer Corpus change feed Embedding Pipeline Service Retrieval gateway Envoy, mTLS Ingest API idempotent Control API contracts, corpora Dependencies Permission authority Corpus fetch APIs SPIRE retrieve retrieve register events ACL check fetch SVID Integration Surface — Consumers, Platform, Dependencies External / third party Person or role Interface / broker Security / platform synchronous event / async Three surfaces, three availability targets: retrieval 99.95%, ingest 99.9%, control 99.5%. A consumer never calls the control plane on the hot path. Omitted: OpenBao, the admin console, the observability sink and the nightly warehouse export. v 1.0 · owner Data & AI Platform Architecture · date 2026-10

Decisions

  • Retrieval, ingest and control are separate surfaces with separate targets (99.95% / 99.9% / 99.5%) because they fail differently and matter differently.
  • The retrieval gateway resolves an alias from a cached pointer. A control-plane outage therefore cannot take retrieval down — it only freezes cut-overs.
  • Ingest is idempotent on (document id, source version), so a redelivery is free and an out-of-order redelivery cannot overwrite newer state.

Dependency posture

  • Permission authority: hard dependency, fails closed. The only one.
  • Corpus fetch APIs: soft — a fetch failure delays a document and never infers a deletion.
  • SPIRE: hard for new workload identity, soft for the lifetime of an issued SVID.
  • Warehouse export: fully asynchronous, and omitted from this view for clarity.

Open question carried forward

  • Whether the platform owns the vector store or treats it as a replaceable sink (requirement question 8) is not settled here. The integration surface is drawn so that either answer stays possible.
◀  Platform Components All views Data Flow — Edit to Retrievable  ▶
Solution Architecture v1.0 · open source, self-hosted on Kubernetes · Data & AI Platform Architecture · 2026-10 · 22 views · 17 architecture decision records09-integration-surface