Search Indexing Service  ·  View 08 of 21  ·  Structure

Platform Components

Four planes in one region, and the two components that are allowed to face outwards.

Editable source SVG draw.io All views
AWS eu-west-1 — VPC, private subnets Control plane Admin & Definition API ECS Fargate Definition Registry Aurora PostgreSQL Reindex Orchestrator Step Functions Evaluation Service judgement sets Write plane Capture Tier DMS + Streams Change Log MSK Assembly Workers ECS Fargate Assembly State DynamoDB Index Writers fast / slow / reindex Search plane OpenSearch Domain 3 AZ · 220 GB · 2 replicas Alias Manager Query Service ECS Fargate Recent-Writes Table DynamoDB · TTL 60 s Assurance Reconciler 1 M docs/day Quarantine Store S3 · 30 d Audit Log S3 Object Lock Source Systems Snapshot Store S3 · daily Enrichment Services changes rebuild base best-effort bulk overlay sample Platform Components — One Region Application we own Data store Interface / broker Queue / topic Security / platform Risk / gap External / third party event / async batch failure / alternate synchronous Only the capture tier faces the sources, and only the query service faces callers. Twelve edges omitted; later views carry them. v 1.0 · owner Data Platform Architecture

Decisions

  • Control, write, search and assurance are separate planes: the admin console cannot take the indexing path down, and the query path does not depend on the registry being writable.
  • Only the capture tier faces the sources and only the query service faces callers. Assembly, writers and the cluster have no public surface.
  • Assembly state — applied source versions, join partners, parent→children mapping, enrichment cache — is a strongly consistent key-value store, separate from both the log and the index.
  • The reconciler is its own component with its own schedule, not a mode of the assembly workers it is checking.

Assumptions

  • One OpenSearch domain across three availability zones, 220 GB primary with two replicas, sized for a second copy of the largest index during a rebuild.
  • ECS Fargate for every tier, chosen so that reindex capacity can be rented for four hours rather than owned.

Omitted

  • Twelve internal edges, carried by views 10, 13, 14 and 19.
  • VPC endpoints, subnet layout and the bastion path, which view 16 and view 19 cover between them.