Search Indexing Service  ·  View 07 of 21  ·  Structure

Layered Architecture

Eight layers, and the one seam that the whole design rests on.

Editable source SVG draw.io All views
Sources Catalogue Inventory Merchant Content Capture Stream Capture DMS / Streams Push Ingest API API Gateway Dedup + Version Guard Poison Quarantine Change log Change Log — system of record for what changed MSK · partitioned by entity key Assembly Join by Declared Keys Parent Fan-out rate-limited, resumable Enrichment timeout + fallback Writers Fast Lane Writer partial update Slow Lane Writer whole document Reindex Writer throttled share Indices idx_v41 live idx_v42 building idx_v40 retained 72 h Alias + gate Alias Acceptance Gate count · diff · judgement Query Query Service Owner-Write Overlay Cost Ceiling replay swap or refuse Layered Architecture — Eight Layers and One Seam External / third party Interface / broker Application we own Risk / gap Queue / topic Data store Decision point Security / platform event / async synchronous The seam: everything above the change log is about what changed; everything below is a projection of it. v 1.0 · owner Data Platform Architecture

The seam

  • Above the change log: what changed, captured, deduplicated, version-guarded, and quarantined if it cannot be understood.
  • Below it: projections — assembled documents, indices, aliases, caches — every one of which can be thrown away and rebuilt.
  • The acceptance gate sits between the index layer and the alias layer, which is the only reason a bad index can be refused rather than discovered.

Decisions

  • Three writers share the cluster under a declared capacity split: fast lane, slow lane, and reindex, the last throttled against a reserved query share.
  • The previous index version is a layer member, not a backup: idx_v40 stays mounted for 72 hours so rollback is an alias move.

Assumptions

  • Quarantine is a normal operating state with a 30-day payload retention, not an error path that is expected to stay empty.