Service Mesh Platform  ·  View 12 of 31  ·  4 · Data

Five Stores, Five Recovery Obligations

Where each kind of state lives, what it costs to lose it, and which stores need backups that are actually restored in drills.

Editable source SVG draw.io All views
Mesh state, by what losing it costs Intent · source of truth · RPO 0 Intent repositories Forgejo · signed commits Applied objects etcd · per cluster Identity · RPO 0 OpenBao Raft intermediates SPIRE datastore entries · agents Issuance records ClickHouse · 13 mo Derived · rebuildable xDS snapshots istiod memory · ≤ 5 min Rendered bundles Harbor OCI · cosign Discovery · current only EndpointSlices per cluster Remote endpoints declared exports only Telemetry · loss-tolerant Thanos 15 d raw · 13 mo Tempo traces · 7 d Access logs ClickHouse · 30 d Five Stores, Five Recovery Obligations Only the first two columns need backups that are restored in drills. The third is rebuilt by a restart, the fourth by the clusters themselves. v 1.0 · owner Platform Networking Architecture · date 2026-09

Decisions

  • Git is the source of intent; each cluster's etcd holds the applied copy. Losing a cluster's etcd costs a resync from Git. Losing Git would cost history, so Forgejo runs on a synchronous PostgreSQL standby with read-only mirrors at every site.
  • Compiled xDS snapshots are never stored. istiod rebuilds them from the Kubernetes API in minutes, and a persisted snapshot would only be a second thing that could disagree with intent.
  • Rendered, signed bundles are stored in Harbor. They are what a one-step revert pins to, so a revert never waits on a recompile.

Targets

  • Intent RPO 0, RTO ≤ 15 min. CA material RPO 0, intermediate reissue ≤ 30 min. Snapshot rebuild ≤ 5 min. Telemetry RPO ≤ 5 min.
  • Retention: metrics 15 days raw and 13 months rolled up, traces 7 days, access logs 30 days, issuance records and audit trail 13 months.

Risks

  • Issuance records are written from SPIRE's audit log through the telemetry path, which drops under backpressure. A nightly reconciliation against SPIRE's datastore finds gaps; an Object Lock copy makes them immutable once found.
◀  Anatomy of One Meshed Call All views Intent and Evidence — Data Model  ▶
Solution Architecture v1.0 · Platform Networking Architecture · 2026-09 · 31 views · open source, fully on-premises12-five-stores