Search Indexing Service  ·  View 10 of 21  ·  Data

Data Flow — One Change, Seven States

From a source commit to a reconciled document, and the two states that are authoritative.

Editable source SVG draw.io All views
Committed Source commit authoritative Captured Change event key + version Rejected poison / drift Logged Log record durable, acked Assembled Document joined + enriched Degraded stale enrichment Written Indexed fast or slow lane Visible Searchable behind the alias Verified Reconciled sampled vs source quarantine Data Flow — One Change, Seven States External / third party Interface / broker Risk / gap Queue / topic Application we own Data store Security / platform failure / alternate batch Only two of the seven are authoritative: the source commit and the log record. The dotted return is the reconciler re-emitting a key it found missing. v 1.0 · owner Data Platform Architecture

Decisions

  • A change is acknowledged at 'logged' and no later. Assembly, enrichment and the cluster are all downstream of the promise already made.
  • 'Degraded' is a first-class state: a document indexed with a stale enrichment value is searchable and flagged, never withheld.
  • The dotted return from 'reconciled' is the only path that repairs a change which never arrived — the failure with no error anywhere else in the flow.

Assumptions

  • 25,000 change events/second steady, 100,000/second for a 10-minute burst.
  • ≤ 0.01% of documents divergent from source at any time, measured by sampling 1,000,000 documents a day.
◀  Integration Surface All views Storage Zones  ▶
Solution Architecture v1.0 · Amazon Web Services · Data Platform Architecture · 2026-1010-indexing-data-flow