Autoscaling Controller  ·  View 12 of 37  ·  3 · Data

Data Architecture

Two stores whose loss is loss of control, and two whose loss is only loss of explanation.

Editable source SVG draw.io All views
Who touches the authoritative stores Decision plane Decision Engine writes desired Actuation plane Reconciler reads desired only Authoritative — losing this loses control Target and policy registry PostgreSQL transactional Version History kept in full Desired-state record Desired Capacity RPO 0 · sync HA Transitions 30 days Derived — losing this loses history, not control Signal store Raw Series 10 s · 15 days Rollup 5 min · 13 months Blob Backend Mimir blocks Decision journal Event Hubs transport · RPO 60 s Azure Data Explorer query · 13 months Blob WORM immutable copy Audit log Config Change Log tamper evident the only writer resume after crash ingest immutable copy Data Architecture — Four Stores, Four Durability Contracts Application we own Data store Queue / topic Security / platform synchronous event / async A controller restart re-derives nothing. It reads the desired-state record and resumes reconciling towards the number already there. v 1.0 · owner Data & AI Global Practice · date 2026-09

Decisions

  • The desired-state record is authoritative. A controller restart re-derives nothing and simply resumes reconciling towards the stored value.
  • Signals are inputs, not state. Losing 15 days of signal history costs a counterfactual replay and nothing else.
  • The journal is append-only and immutable, with the queryable copy in Data Explorer and the evidential copy in WORM Blob.

Why four stores and not one

  • Their durability requirements differ by three orders of magnitude, and one store sized for the strictest of them would cost more than the fleet it manages.
  • 200,000 samples/second and 150 journal records/second have nothing in common as write profiles.

Risks

  • Two authoritative stores in one PostgreSQL instance makes that instance the control plane's single point of failure. Zone-redundant HA with a synchronous standby is the mitigation, and RTO ≤ 60 s the accepted exposure.
  • During that RTO the fleet holds capacity and cannot change size. That is the deliberate trade: no change beats a wrong change.
◀  Authority and Arbitration All views Target and Policy Model  ▶
Solution Architecture v1.0 · Data & AI Global Practice · 2026-09 · 37 views · Azure and open source12-data-architecture