Embedding Pipeline Service  ·  View 12 of 22  ·  Data

Data Model

Thirteen entities, and the compound primary key that is the critical design decision expressed as a constraint.

Editable source SVG draw.io All views
tenant tenant_id PK residency_region quota_chunks quota_queries_s corpus corpus_id PK tenant_id FK source_kind freshness_tier embedding_contract contract_id PK normaliser_version chunker_version model_digest pooling_rule dimensions vector_index index_id PK contract_id FK tenant_id FK state built_at index_alias alias PK index_id FK previous_index_id flipped_at document document_id PK tenant_id FK corpus_id FK acl_ref deleted_at document_version document_id PK FK version PK source_version visible_at chunk chunk_hash PK document_id FK version FK ordinal offset_start heading_path vector chunk_hash PK FK contract_id PK FK index_id FK status migration migration_id PK from_contract FK to_contract FK oldest_unmigrated state suppression document_id PK FK reason effective_at erasure_confirmed_at normalised_text document_id PK FK version PK FK normaliser_version text_uri expires_at quarantine chunk_hash PK FK reason attempts first_seen evaluation_run run_id PK contract_id FK corpus_id FK recall_at_10 mrr 1 : N N : 1 1 : N 1 : N 1 : N 1 : N 1 : N 1 : N 1 : N 1 : N N : 1 1 : 1 1 : 1 1 : 1 N : 1 Data Model — Contract, Document, Chunk, Vector A vector's primary key is (chunk_hash, contract_id). That compound key is the critical design decision written as a constraint. v 1.0 · owner Data & AI Platform Architecture · date 2026-10

The key that matters

  • A vector's primary key is (chunk_hash, contract_id). A model change therefore produces new rows rather than updating existing ones, and the schema makes an in-place upgrade unexpressible.
  • chunk_hash is derived from normalised chunk text, so chunk identity survives an edit elsewhere in the document — this is what makes the diff cheap.
  • index_alias carries previous_index_id, which is what makes rollback a single write rather than a rebuild.

Modelling choices

  • A chunk points at the contract that produced it, because the chunker version is a contract element — a chunker change is as much a rebuild as a model change.
  • normalised_text carries expires_at, so the 30-day retention is a property of the data rather than of a cleanup job.
  • quarantine and suppression are separate optional entities rather than status columns, so a typed failure and a revocation each carry their own evidence.

Not modelled here

  • Permissions. The platform holds acl_ref on a document and resolves it against the authority at query time; mirroring the ACL model would be a second source of truth.
  • Lexical index state, which is derived from the same chunk rows and adds nothing to this view.
◀  Storage Zones All views Critical Flow — One Edit Becomes Retrievable  ▶
Solution Architecture v1.0 · open source, self-hosted on Kubernetes · Data & AI Platform Architecture · 2026-10 · 22 views · 17 architecture decision records12-data-model