Search Indexing Service  ·  View 14 of 21  ·  Runtime

Reindex and Swap

Build beside the live index, catch up from the log, gate on evidence, then move one alias.

Editable source SVG draw.io All views
Trigger Breaking definition change Scheduled rehearsal Plan Dry Run duration · storage · cost Budget Ack above threshold Build Snapshot Replay ≥ 8 k docs/s Partition Checkpoints resume, not restart Catch up Log Catch-up to fast-lane budget Gate Doc Count Sampled Diff Judgement Score Swap Alias Move atomic per reader Settle idx_v40 retained 72 h Rollback = alias move RTO 1 min refuse, resume lag > budget: hold Reindex and Swap — Build Beside, Gate, Move the Alias Decision point Security / platform Application we own Data store Interface / broker Opportunity failure / alternate Reindex writes take a reserved share of cluster capacity: a rebuild may not raise query p99 by more than 15%. v 1.0 · owner Data Platform Architecture

Decisions

  • Progress is checkpointed per source partition, so a dead worker costs minutes rather than restarting a four-hour rebuild.
  • The swap is refused while the new index's lag exceeds the fast-lane budget: a correct index that is 40 seconds behind is not a safe index to serve.
  • The gate is four independent detectors, because each catches a different failure — count catches a truncated rebuild, sampled diff catches a mapping change, judgement score catches a relevance regression, and a canary catches what none of them predicted.
  • The gate refuses rather than escalating. Waking a human at 2am to approve a swap they cannot evaluate is not a control.

Assumptions

  • Largest index 38,000,000 documents, rebuilt in ≤ 4 h at ≥ 8,000 docs/second, while query p99 rises by no more than 15%.
  • Dry run reports duration, storage and cost before promotion; a rebuild above a declared cost threshold needs an explicit acknowledgement.

Risks

  • The 72-hour rollback window is the real limit on how long a bad relevance change can hide. A regression discovered on day four needs a rebuild, not an alias move.
◀  Critical Flow — A Price Change Becomes Searchable All views Freshness Lanes  ▶
Solution Architecture v1.0 · Amazon Web Services · Data Platform Architecture · 2026-1014-reindex-and-swap