Search Indexing Service  ·  View 15 of 21  ·  Runtime

Freshness Lanes

Five write classes, five budgets, and no shared queue between them.

Editable source SVG draw.io All views
Capture Assemble Write Visible Budget Fast lane — caught instantly Stock, price, hours Partial update no re-enrichment Fast Lane Writer idx_v41 p95 ≤ 2 s Suppression — no budget Delete, moderation No join needed Priority write Removed from candidates p99 ≤ 5 s Slow lane — invisible for a minute Titles, attributes Full assembly + enrich Slow Lane Writer idx_v41 p95 ≤ 60 s Parent fan-out — tracked job Brand, category rename Resolve children rate-limited Batched writes Progress + ETA 2 M children ≤ 30 min Reindex — reserved capacity Snapshot + log Deterministic rebuild Throttled writer idx_v42, not yet read 38 M docs ≤ 4 h Freshness Lanes — Two Budgets Through One Pipeline The lanes do not share a queue: a four-hour rebuild and a 4x burst of slow-lane edits cannot touch the fast lane's budget. v 1.0 · owner Data Platform Architecture

Decisions

  • Staleness is not one budget. Fields a user catches instantly get seconds; fields nobody notices for a minute get a minute; suppression gets no tolerance at all.
  • Suppression is its own lane rather than the fast lane's tail, so a moderation action cannot queue behind a price storm.
  • Parent fan-out is a tracked job with progress and an ETA, because a brand rename touching two million listings cannot be an inline side effect of one change.
  • Reindex writes take a reserved share of capacity, which is what makes a rebuild a scheduled operation rather than a latency incident.

Assumptions

  • Fast lane p95 ≤ 2 s; suppression p99 ≤ 5 s; slow lane p95 ≤ 60 s, p99 ≤ 5 min; fan-out of 2,000,000 children complete within 30 min at p95.
  • Degraded mode may widen the slow lane to 30 min, and the response says so.

Risks

  • Two lanes writing the same document means a partial update can land on a document the slow lane is about to replace. Version-guarded field-level writes are mandatory, not an optimisation.
◀  Reindex and Swap All views Deployment Architecture  ▶
Solution Architecture v1.0 · Amazon Web Services · Data Platform Architecture · 2026-1015-freshness-lanes