Autoscaling Controller

Architecture Views

37 views, in reading order. Every view ships three ways: an HTML page, an SVG that re-opens in diagrams.net fully editable, and draw.io source.

A closed-loop controller that observes demand on a fleet of heterogeneous workloads and drives capacity to the smallest number that still meets the service objective. Read the set in order: acts 1 and 2 fix the boundary and the plane separation everything else follows from; act 3 says which store is the truth and which is rebuildable; act 4 shows what actually happens in one 15 second interval, on the way up and on the way down; act 5 covers running it and paying for it; act 6 covers why it is safe and what it does when it is wrong or blind. One decision governs all of them — the decision plane writes a number, the actuation plane converges on it, and they are joined only by a durable record.

1 · Context and scope

The boundary, who feeds it and what it drives, and the loop that is the whole system.
01 Signal sources Managed Prometheus utilization, custom Azure Monitor platform metrics Service Bus backlog, msg age Forecast Feed calendar, prediction People Service Owner SRE On-Call FinOps Analyst Platform Admin Capacity providers VM Scale Sets stateless fleets AKS Node Pools node groups Workload Replicas Deployments Warm Pool Fleet pre-warmed Autoscaling Controller declare, then converge Governance and evidence Microsoft Entra ID workload identity Azure Data Explorer decision journal Grafana + Alertmanager alerts, dashboards declares target + bounds pins, pauses, unfreezes reads cost of stability sets fleet ceilings remote read metric query queue depth capacity floor set capacity set node count set replicas attach, recycle authenticates journals every decision exports metrics Autoscaling Controller — System Context External / third party Queue / topic Person or role Security / platform Data store synchronous batch Out of scope: running the workloads, scheduling them, and any customer payload. The controller decides how much capacity a target should have and drives the fleet to that number. v 1.0 · owner Data & AI Global Practice · date 2026-09 System Context What sits inside the controller's boundary, what feeds it, and what it is allowed to change. HTML page SVG draw.io
02 Observe Collectors 4 signal classes Normaliser one common shape Freshness Gate 90 s · plausible Decide Policy Evaluator target, step, sched Damping + Bounds hysteresis, limits Shard Leader one per shard Declare Desired-State Record PostgreSQL · RPO 0 Converge Reconciler idempotent Provider Adapters breaker, backoff Drain Coordinator 120 s deadline Serve Managed Capacity VMSS, pools, pods Serving Verifier healthy + registered Explain Decision Journal 13 months Cost Attribution per team, per hour observed actual every evaluation Autoscaling Controller — High-Level Architecture Application we own Decision point Security / platform Data store Interface / broker External / third party event / async The spine is the control loop. Actual capacity re-enters as a signal, never as an assumption. v 1.0 · owner Data & AI Global Practice · date 2026-09 High-Level Architecture Six stages: observe, decide, declare, converge, serve, explain — and one feedback edge that matters. HTML page SVG draw.io
03 Observe read signal store Qualify fresh, complete, sane Decide policy + damping Declare desired capacity Converge reconcile the delta Verify Serving healthy + registered Control Loop 15 s · per target raw samples qualified inputs proposed capacity one durable number provider calls observed actual The Control Loop — One 15 Second Interval Application we own Decision point Data store Six steps, one direction. A step that cannot complete holds the last declaration rather than guessing the next one. v 1.0 · owner Data & AI Global Practice · date 2026-09 The Control Loop The six steps of one 15 second interval, and the fact that it closes. HTML page SVG draw.io
04 Configuration and control Target Registry versioned Policy Catalogue immutable versions Bounds and Ceilings outside the policy Overrides expiry mandatory Signal plane Collectors authenticated sources Normaliser Derived Signals named expressions Freshness Gate Signal Store Mimir · 13 months Decision plane Policy Evaluation sharded Combination Rule max out, min in Stability Controls hysteresis, limits Oscillation Guard freeze and alert Desired-state record Desired Capacity per Target strongly consistent · the truth Actuation plane Reconciler no policy logic Provider Adapters Warm-Pool Manager Drain Coordinator Registration Check Managed capacity Service Groups VMSS Worker Pools queue driven Node Groups AKS Warm Pools attach in 10 s Observability Decision Journal append only Controller Metrics Alerts Cost Reporting Counterfactual Replay Layered Architecture Data store Security / platform Application we own Decision point Interface / broker External / third party A layer may call the layer below it and read the layer above it. Bounds live above the policy that would otherwise compute them. v 1.0 · owner Data & AI Global Practice · date 2026-09 Layered Architecture Seven layers, and the one rule about which way they may call. HTML page SVG draw.io

2 · Structure

The plane separation that makes it safe, the deployable units inside each plane, every interface, and who wins when four things want to scale one fleet.
05 Decision plane — reads, never calls Signal Reader read-only identity Policy Evaluation versioned policy Damping and Bounds the only place Journal Writer inputs and arithmetic The join — a written record Desired-State Record one integer per target Decision Id 1 : 1 with actuation Actuation plane — calls, never decides Reconciler delta from observed Provider Adapters actuate identity Serving Verifier counts what serves never actual capacity Decision Plane and Actuation Plane, Joined by a Record Application we own Data store Interface / broker failure / alternate event / async The red path is the one the architecture forbids. A decision that could call a capacity API would make crash recovery, idempotency and audit all unprovable at once. v 1.0 · owner Data & AI Global Practice · date 2026-09 The Critical Design Decision The decision plane never calls a capacity API. The red arrow is the path the architecture forbids. HTML page SVG draw.io
06 Azure Region · control plane on AKS Signal services Collector Set Go · one per class Normaliser common shape Qualification Gate fresh + plausible Decision services Shard Manager lease per shard Decision Engine Go · 15 s tick Journal Writer at-least-once Actuation services Reconciler idempotent Adapter Set VMSS, AKS, K8s Warm-Pool Manager Drain Coordinator lease aware Control surface Config API targets, policies Override API expiry enforced Explain API why this capacity Regional state and evidence Authoritative · strongly consistent PostgreSQL Flexible Server registry + desired Derived · rebuildable or append-only Grafana Mimir signal store Event Hubs journal transport Azure Data Explorer journal query Blob · immutable WORM · 13 months Microsoft Entra ID workload identity Azure Key Vault no standing creds Azure Resource Manager capacity APIs desired write desired read scoped identity decision record Container Architecture — Deployable Units (C4 L2) Application we own Decision point Security / platform Interface / broker Data store Queue / topic External / third party synchronous event / async Every service is stateless. All regional state is in the row below it, and no service reads another region's copy. v 1.0 · owner Data & AI Global Practice · date 2026-09 Container Architecture Every deployable unit, and the regional state it is allowed to touch. HTML page SVG draw.io
07 Acquisition — four classes of source, each authenticated Resource utilization · lags CPU and Memory Managed Prometheus Accelerator Use DCGM exporter Work backlog · leads Queue Depth Service Bus Oldest Message Age the better signal Consumer Lag Event Hubs Service level · second order Latency Percentile p95, p99 SLO Burn Rate slow, noisy External declarations Schedule floor or ceiling Forecast Feed signed, hourly Qualification and storage Normalise Normaliser target, unit, window Derived Signals versioned expression Qualify Freshness Check reject over 90 s Plausibility Clamp declared range Quarantine hold capacity Store Mimir 10 s · 15 days Rollup 5 min · 13 months Last-Known-Good 3 intervals preferred stale qualified Signal Plane — Acquisition, Qualification, Storage Application we own Queue / topic External / third party Decision point Risk / gap Data store synchronous failure / alternate Freshness and completeness travel with the value as first-class attributes. A policy binds to a named authenticated source, so source identity is part of the signal definition. v 1.0 · owner Data & AI Global Practice · date 2026-09 Signal Plane Four classes of source, one common shape, and a gate that qualifies every sample. HTML page SVG draw.io
08 Sharding — decision throughput scales with target count Shard 0 · leader Decision Engine holds valid lease Shard 1 · leader Decision Engine holds valid lease Shard 2 · standby Decision Engine no lease, no write Lease authority Lease Store K8s Lease · 10 s What one shard does in one interval Inputs Qualified Signals with freshness Policy Version immutable Bounds and Overrides Propose Target Tracking hold a setpoint Step / Threshold capacity per band Scheduled Floor time window Forecast Floor may not undercut Constrain Combination Rule max out · min in Hysteresis + Sustain 15% · 2 / 6 Rate Limiter +100% / -10% / 60 s Hard Bounds min, max, ceiling Emit Desired Capacity durable write Journal Record acted on or not lease denied clamped last Decision Plane — Sharded, Leader per Shard Application we own Risk / gap Security / platform Data store failure / alternate synchronous Bounds are applied after every policy has spoken, by a component no policy can reach. A policy cannot breach a limit it also computes. v 1.0 · owner Data & AI Global Practice · date 2026-09 Decision Plane Sharded with a lease per shard, and what one shard does in one interval. HTML page SVG draw.io
09 Reconcile loop — holds no policy and no opinion about demand Read Desired Capacity the declaration Observed Actual what is serving Derive Delta re-derived each run Safety Bound Check refuse and journal Apply VMSS Adapter Node Pool Adapter Replica Adapter Warm-Pool Attach under 10 s Confirm Health Check LB Registration then it counts Drain Coordinator 120 s deadline Provider protection — every capacity API is rate-limited and failure-prone Throttling Concurrency Cap per provider Backoff + Jitter exponential Circuit Breaker 5 failures Scarcity Zone Fallback ordered list Shape Fallback declared shapes Unmet Demand a signal, not a retry counted as serving open Actuation Plane — Converge Actual Towards Desired Data store Decision point Security / platform Interface / broker Application we own Risk / gap synchronous failure / alternate Actual capacity is observed, never assumed from intent. Repeating a run for unchanged desired capacity changes nothing. v 1.0 · owner Data & AI Global Practice · date 2026-09 Actuation Plane Converge actual towards desired, and survive a capacity API that is rate-limited and failure-prone. HTML page SVG draw.io
10 Inbound — read only Managed Prometheus PromQL Azure Monitor REST Service Bus Mgmt Git · policy repo GitOps Forecast Feed Regional control plane Autoscaling Controller one per region Outbound — scoped identities VMSS API AKS Node Pool API Kubernetes API Event Hubs Alertmanager 10 s pull 60 s pull backlog, age versions signed hourly set capacity set count set replicas journal alerts Integration Surface — Every Interface In and Out External / third party Queue / topic Application we own Security / platform synchronous batch event / async Five inbound interfaces, five outbound. Nothing inbound can change capacity, and nothing outbound can change a policy. v 1.0 · owner Data & AI Global Practice · date 2026-09 Integration Surface Five inbound interfaces, five outbound, and the rule that separates them. HTML page SVG draw.io
11 Claimants Workload Policy target tracking Scheduled Policy floor or ceiling Forecast Floor may not undercut Operator Override expires in 4 h Single admission point Precedence Rules per target Combination Rule max out · min in Hard Bounds no claimant exceeds Declared Desired Capacity the workload layer Derived Node Layer Follows purely derivative Unmet Demand when it cannot cannot supply back as a signal Authority — Four Things Want to Scale One Fleet Application we own Person or role Decision point Security / platform Data store Risk / gap failure / alternate event / async Authority is strictly layered and arbitrated in one place. The infrastructure layer is derivative of the workload layer, and says so out loud when it cannot supply what was declared. v 1.0 · owner Data & AI Global Practice · date 2026-09 Authority and Arbitration A workload policy, a schedule, a forecast and a human can all target the same fleet. One of them wins, and it is decided in one place. HTML page SVG draw.io

3 · Data

Which two stores are authoritative, which two are rebuildable, the model behind an explainable decision, and why nothing crosses a region.
12 Who touches the authoritative stores Decision plane Decision Engine writes desired Actuation plane Reconciler reads desired only Authoritative — losing this loses control Target and policy registry PostgreSQL transactional Version History kept in full Desired-state record Desired Capacity RPO 0 · sync HA Transitions 30 days Derived — losing this loses history, not control Signal store Raw Series 10 s · 15 days Rollup 5 min · 13 months Blob Backend Mimir blocks Decision journal Event Hubs transport · RPO 60 s Azure Data Explorer query · 13 months Blob WORM immutable copy Audit log Config Change Log tamper evident the only writer resume after crash ingest immutable copy Data Architecture — Four Stores, Four Durability Contracts Application we own Data store Queue / topic Security / platform synchronous event / async A controller restart re-derives nothing. It reads the desired-state record and resumes reconciling towards the number already there. v 1.0 · owner Data & AI Global Practice · date 2026-09 Data Architecture Two stores whose loss is loss of control, and two whose loss is only loss of explanation. HTML page SVG draw.io
13 policy_version policy_version_id PK target_id FK form setpoint dead_band_pct sustain_out / sustain_in immutable_at fleet fleet_id PK region capacity_ceiling spend_ceiling_month default_degraded_mode signal_definition signal_id PK target_id FK source_identity expression lead_or_lag plausible_min / max staleness_horizon safety_bound bound_id PK target_id FK kind value second_approver effective_from scaling_target target_id PK fleet_id FK provider_kind min_capacity max_capacity control_interval_s drain_deadline_s scale_to_zero_ok warm_pool pool_id PK target_id FK NULL topology size_cap_pct max_member_age shape override override_id PK target_id FK kind pinned_capacity expires_at NOT NULL actor justification desired_state target_id PK desired_capacity decided_at decision_id FK dry_run 1 : N 1 : N 1 : N 1 : N 1 : N 1 : N 1 : 1 Target, Policy and Bound — Data Model A policy version is immutable once written, so a journalled decision can always be re-read against the exact policy that produced it. v 1.0 · owner Data & AI Global Practice · date 2026-09 Target and Policy Model The registry: what a target is, what may constrain it, and what expires. HTML page SVG draw.io
14 decision decision_id PK target_id interval_start policy_version_id proposed_capacity final_desired_capacity outcome decision_input input_id PK decision_id FK signal_id value unit freshness_ms sample_count was_last_known_good adjustment adjustment_id PK decision_id FK kind before after reason actuation actuation_id PK decision_id FK UNIQUE provider requested_delta applied_delta result zone_or_shape_used observed_capacity observation_id PK target_id observed_at instances_serving instances_provisioning instances_draining audit_event audit_id PK actor target_id field before / after justification signed_at 1 : N 1 : N 1 : 1 informs context Decision Journal — What Makes a Decision Explainable Every adjustment — clamp, suppression, rate limit, dead band — is a row with a reason. "Why did this target have this much capacity?" is answered from these tables alone. v 1.0 · owner Data & AI Global Practice · date 2026-09 Decision Journal Model What has to be recorded for "why did this target have this much capacity?" to be answerable without replaying a metric. HTML page SVG draw.io
15 Emit Authenticated Source named in the policy Stamped at Source not at ingest Normalise Common Shape target, dim, unit Window and Count completeness Derived Expression named, versioned Qualify Age Check over 90 s: reject Range Check clamp, then flag Clock Skew Check alert if persistent Serve Signal Store Mimir Last-Known-Good 3 intervals only Consume Decision Input value + freshness Marked Stale policy degrades rejected horizon passed quarantined Signal Lifecycle — Sample to Decision Input External / third party Application we own Decision point Data store Risk / gap failure / alternate Rejection is recorded, never silent. A signal that has aged past three control intervals stops being an input and becomes a degradation trigger. v 1.0 · owner Data & AI Global Practice · date 2026-09 Signal Lifecycle A sample from emission to decision input, and the three ways it can fail to get there. HTML page SVG draw.io
16 Consistency Replication and RPO Recovery Retention Target and policy registry Strongly consistent transactional Zone-redundant sync RPO 0 RTO under 60 s Current + full history Desired-state record Strongly consistent the truth Synchronous replica RPO 0 RTO under 60 s read SLO 99.95% Current + 30 days transitions Signal store Eventually consistent loss tolerant Replicated blocks RPO 5 min Rebuildable control unaffected 15 d raw · 13 mo rollup Decision journal Append only immutable Buffered transport RPO 60 s RTO 15 min not on control path 13 months WORM copy Audit log Append only tamper evident Signed batches RPO 60 s RTO 15 min 13 months Durability and Retention by Store The two authoritative stores carry RPO 0 because losing them loses control. The two derived stores are allowed to lose minutes because losing them only loses explanation. v 1.0 · owner Data & AI Global Practice · date 2026-09 Durability and Retention The contract each store signs, side by side. HTML page SVG draw.io
17 Azure · Sweden Central Control path — entirely local Decision Engine Desired State local writer Reconciler Local evidence Signal Store Journal Azure · West Europe Control path — entirely local Decision Engine Desired State local writer Reconciler Local evidence Signal Store Journal Shared, and deliberately off the control path Declared intent Policy Repository Git · source of truth Reporting only Cross-Region Cost View read of journals Fleet Rollup eventually consistent policy versions policy versions async export async export Regional Partitioning — No Cross-Region Read on the Control Path Application we own Data store External / third party batch event / async Adding a region means deploying an independent control plane. The only shared state is the replicated registry, and it is never read inside a control interval. v 1.0 · owner Data & AI Global Practice · date 2026-09 Regional Partitioning Every store partitioned by region, and nothing on the control path crossing one. HTML page SVG draw.io

4 · Runtime

One interval end to end, how signals are ranked, how policies combine, what stops oscillation, and what happens on the way up, on the way down and when the capacity is not there.
18 Interval Tick Signal Store Decision Engine Desired State Journal Reconciler Capacity Provider 1. t0 · shard tick 2. lease still valid? 3. read signals for target 4. values + freshness 5. evaluate every policy 6. combine, damp, clamp 7. write desired capacity 8. committed · p99 2 s 9. decision + all inputs 10. read desired 11. observe actual 12. instances serving 13. apply delta · p99 3 s 14. actuation outcome One Control Interval, End to End The engine never calls the provider and the reconciler never reads a signal. The two halves meet only at the desired-state record. v 1.0 · owner Data & AI Global Practice · date 2026-09 One Control Interval Tick to actuation call, with the two halves meeting only at the record. HTML page SVG draw.io
19 Preferred — leads demand Oldest Message Age rank 1 · queued work Concurrency arrival x service Fallback — lags demand Utilization rank 2 · always there SLO Burn Rate rank 3 · slow, noisy Resolve within the interval Fresh Signals Only stale ones drop out Highest Rank Wins for the setpoint Lower Ranks Vote scale-in must agree Result One Proposed Capacity with its driver named Driving Signal Recorded in the journal if fresh if rank 1 stale no lone scale-in Which Signal Drives the Loop — A Declared Hierarchy Queue / topic Application we own Decision point Data store synchronous Conflict inside one interval is resolved by rank on scale-out and by unanimity on scale-in. Being late to add capacity costs latency; being wrong to remove it costs availability. v 1.0 · owner Data & AI Global Practice · date 2026-09 Which Signal Drives the Loop A declared hierarchy, and how conflicting signals are reconciled inside one interval. HTML page SVG draw.io
20 Active policies Target Tracking hold a setpoint Step / Threshold steps per band Scheduled floor or ceiling Forecast Floor independent floor Each proposes Proposed Capacity integer per policy Arithmetic Recorded shown, not implied Combine Scale-Out: Maximum one may raise Scale-In: Minimum Agreed none may lower alone Clamp Target [min, max] Fleet Ceiling serve, do not grow Spend Ceiling per region Declare Desired Capacity Dry-Run Target journalled, not applied raises only reached: hold Policy Evaluation and Combination Application we own Data store Decision point Security / platform Risk / gap synchronous failure / alternate A forecast enters as a floor, never as a bias on the reactive loop. A wrong forecast then costs money, not an outage. v 1.0 · owner Data & AI Global Practice · date 2026-09 Policy Evaluation and Combination Three policy forms plus a forecast, one proposed number, then the clamps. HTML page SVG draw.io
21 Dead band Hysteresis 15% of setpoint Noise Below Band no change at all Persistence Sustain Out 2 intervals Sustain In 6 intervals Rate limit Growth Cap +100% / 60 s Reduction Cap -10% / 60 s Cooldown Scale-In Suppressed while still arriving Pending Capacity not yet serving Guard Oscillation Detector 3 reversals / 15 min Freeze and Alert names the signal Record Suppressed Decision a distinct outcome still becoming serving reversal 3 no change suppressed Stability — What Stops the Loop Fighting Itself Application we own Security / platform Decision point Risk / gap Data store synchronous event / async Scale-out and scale-in are deliberately asymmetric. Every suppression is journalled as its own outcome, so "nothing happened" is a fact with a reason rather than an absence. v 1.0 · owner Data & AI Global Practice · date 2026-09 Stability and Oscillation Control Five mechanisms in series, and the record that every suppression leaves behind. HTML page SVG draw.io
22 Work Queue Decision Engine Desired State Reconciler Warm Pool VMSS Load Balancer 1. backlog age rising 2. interval 1: breach, hold 3. interval 2: sustained 4. desired 40 -> 72 5. read desired 72 6. claim 8 warm members 7. attach · under 10 s 8. provision 24 cold 9. boot, warm, health 10. registered: counts now 11. scale-in suppressed Scale-Out — Sustained Breach to Serving Capacity p95 to serving capacity: 60 s for the warm-pool-backed share, 240 s for the cold share. The warm pool exists to cover the first of those windows, not the second. v 1.0 · owner Data & AI Global Practice · date 2026-09 Scale-Out A sustained backlog breach, through the warm pool and past it, to registered capacity. HTML page SVG draw.io
23 Decision Engine Desired State Reconciler Drain Coordinator Instance Load Balancer Journal 1. 6 intervals under band 2. desired 72 -> 66 3. delta capped at -10% 4. select 6 by lease state 5. may I stop? 6. lease held: refuse 7. deregister the rest 8. stop new work 9. wait to 120 s deadline 10. terminate 11. refusals + drains Scale-In — A Request the Workload May Refuse An instance holding a work lease refuses removal, and the reconciler re-derives the delta next interval rather than forcing it. Removing an instance still doing work is the failure this design exists to survive. v 1.0 · owner Data & AI Global Practice · date 2026-09 Scale-In How removal is made safe, and the instance that is allowed to say no. HTML page SVG draw.io
24 Size burst x time-to-serve Pre-Warm image, deps, cache Hold cost accounted apart Attach under 10 s Measure hit rate, penalty Recycle never a stale image Warm Pool 5% of steady state target size ready members burst arrives serving instantly resize evidence age limit reached Warm Pool Lifecycle Application we own Data store Security / platform A pool miss is not a failure, it is a recorded latency penalty against the target. Empty pool falls back to cold provisioning. v 1.0 · owner Data & AI Global Practice · date 2026-09 Warm Pool Lifecycle Size, warm, hold, attach, measure, recycle — and what it costs to hold it. HTML page SVG draw.io
25 Driving signal Policy form Actuation Counted when Stateless service Concurrency arrival x service Target tracking VMSS capacity LB registered Queue worker pool Oldest message age leads demand Target tracking age setpoint Replica count Subscription active Batch consumer Consumer lag plus schedule Scheduled + step Replica count Partition assigned Node group Unschedulable pods derived, not raw Derivative of above no policy of its own AKS node count Node Ready Four Classes of Target, One Loop The node group has no independent opinion about demand. It supplies what the workload layer declared, or reports unmet demand. v 1.0 · owner Data & AI Global Practice · date 2026-09 Four Classes of Target The same loop, four driving signals, four definitions of "serving". HTML page SVG draw.io
26 Request Provision Request preferred shape Refusal Zonal Stockout Quota Exhausted Accelerator Scarce API Throttled different problem Declared fallback Next Zone ordered list Next Shape declared only Spot then On-Demand above the floor Exhausted Unmet Demand Signal surfaced, not retried Alert capacity constrained Meanwhile Hold What Serves no scale-in Time Constrained reported per target breaker, not fallback degrade, keep serving When the Capacity Is Not There Application we own Risk / gap Security / platform Data store failure / alternate synchronous Throttling and scarcity look alike and are handled differently: throttling is retried with backoff behind a breaker, scarcity is a fallback list and then an honest signal. v 1.0 · owner Data & AI Global Practice · date 2026-09 When the Capacity Is Not There Stockout, quota and scarcity are one problem; throttling is a different one. HTML page SVG draw.io

5 · Operations

Where it runs, how a region is added, how a policy is promoted without disturbing the fleet, what is watched, and what stability actually costs.
27 Azure · Sweden Central · control plane Zone 1 AKS System Pool 3 nodes Decision Pods shards 0-2 Reconciler Pods 4 replicas Zone 2 AKS System Pool 3 nodes Decision Pods shards 3-5 Reconciler Pods 4 replicas Zone 3 AKS System Pool 3 nodes Decision Pods standby shards Signal Collectors 6 replicas Zone-redundant regional services Authoritative state PostgreSQL Flexible sync standby · RPO 0 Lease Store etcd via AKS Signal and evidence Mimir 3 AZ · Blob backend Event Hubs zone redundant Data Explorer leader-follower Platform Key Vault private endpoint Grafana Alertmanager Managed VMSS Fleets 50k -> 200k instances Managed Node Pools the fleet under them desired write actuate remote write Deployment — One Region, Three Availability Zones Application we own Data store Security / platform Queue / topic External / third party synchronous event / async Losing one zone loses one third of the decision shards and no desired state. The remaining zones take the leases; already-declared capacity keeps converging throughout. v 1.0 · owner Data & AI Global Practice · date 2026-09 Deployment Architecture One region, three zones, and what surviving the loss of one of them actually means. HTML page SVG draw.io
28 Region A · Sweden Central Control plane Full Control Plane 2,000 targets Regional State no remote read Managed fleet 200k Instances design ceiling Region B · West Europe Control plane Full Control Plane identical build Regional State no remote read Managed fleet 200k Instances Global · declarative and reporting only Intent Policy Repository reviewed, versioned Argo CD per-region sync Evidence Fleet Cost View async, lagging Fleet SLO View never a dependency declared config declared config export export Multi-Region — Independent Control Planes Application we own Data store External / third party Security / platform batch event / async Adding a region is a deployment, not a design change. Region B failing entirely has no effect on Region A's control loop. v 1.0 · owner Data & AI Global Practice · date 2026-09 Multi-Region Topology Two identical control planes that share intent and nothing else. HTML page SVG draw.io
29 Author Controller Code Go Policy Change declarative Review Pull Request CODEOWNERS Second Approver ceilings and max Prove Unit and Sim synthetic demand Counterfactual Replay against real history Capacity Diff cost and latency Stage Dry Run on Target journalled only Canary Targets 5% of the fleet Promote Argo CD Sync region by region Version Pinned immutable in journal Revert Roll Back Policy capacity unchanged unacceptable reversals rise Shipping the Controller, and Shipping a Policy Application we own Security / platform Decision point Data store failure / alternate A policy roll-back changes the next decision, never the current capacity. The fleet is not disturbed by the deployment that changes how it will be decided. v 1.0 · owner Data & AI Global Practice · date 2026-09 CI/CD and Policy Promotion Two things ship here — the controller, and the rules it decides by — and only one of them can move a fleet. HTML page SVG draw.io
30 Signal plane Decision plane Actuation plane Managed capacity Metrics Ingest lag p99 target 5 s Decision latency p99 target 2 s Actuation latency p99 target 3 s Desired vs actual per target Health Signal freshness Lease held per shard Breaker state Time-to-serving distribution Quality Samples rejected stale or implausible Suppressed decisions by damping Partial actuations Warm-pool hit rate Alerts Stale past horizon Pinned at max 10 min 3 failed intervals Oscillation freeze Explain Signal history 13 months rolled Why this capacity journal only Actuation trace 1 : 1 to decision Cost of stability currency per day Observability — Signal Type by Loop Stage Every alert names a condition an operator can act on. "Why this capacity" is answered from the journal without replaying a single metric. v 1.0 · owner Data & AI Global Practice · date 2026-09 Observability Signal type by loop stage, and the alerts an operator can actually act on. HTML page SVG draw.io
31 Attribute Serving Capacity per hour, per team Warm-Pool Cost accounted apart Controller Cost under 1.5% Measure Capacity Efficiency served vs provisioned Policy Headroom what the setpoint holds Price the stability Dead Band Sustain Windows Scale-In Rate Limit Cooldowns Report Cost of Stability currency per day The Tuning Number latency vs cost Act Budget Approached per target, monthly Scale-In Sharper inside safety bounds Alert, Never Breach Min within bounds floor holds argue with this Cost — Making the Price of Stability Visible Data store Application we own Decision point Risk / gap Security / platform synchronous Capacity retained by damping is not waste and not free. Pricing it in currency per day is what turns a tuning argument into a decision. v 1.0 · owner Data & AI Global Practice · date 2026-09 Cost and the Price of Stability Attribution, efficiency, and the one number a policy is actually tuned against. HTML page SVG draw.io
32 Alert Fires named condition Ask the Journal why this capacity Override expires in 4 h Draft New Policy as a version Replay Counterfactual against history Promote or Discard cost diff shown Operate tune against evidence a target, a time the driving signal capacity held candidate version what it would have done policy or nothing The Operator's Loop Security / platform Data store Application we own Decision point An override buys time to think; it never becomes the fix. Expiry is mandatory, so control returns to the policy whether or not anyone remembers. v 1.0 · owner Data & AI Global Practice · date 2026-09 The Operator's Loop Alert, ask the journal, buy time, propose a policy, prove it, promote or discard. HTML page SVG draw.io

6 · Assurance

Trust zones, three separate identities, who may change what, what it does when blind, and every named failure mode with its handling.
33 Untrusted · human and external Operator Entra + MFA Forecast Vendor signed payload Compromised Source assumed possible Control surface · private Control API RBAC · private only Policy Repository reviewed changes Signal plane · read only Collectors read identity Qualification Gate reject unsigned Decision plane · writes state Decision Engine write identity Desired State no capacity access Actuation plane · changes capacity Reconciler actuate identity Capacity APIs scoped to groups OIDC + MFA signed feed rejected config only qualified signals desired write desired read scoped mTLS Trust Zones — Every Crossing Authenticated Person or role External / third party Risk / gap Interface / broker Application we own Decision point Data store synchronous failure / alternate The signal path and the capacity path never share an identity. Compromising a metrics source cannot change a single instance. v 1.0 · owner Data & AI Global Practice · date 2026-09 Trust Zones Five zones, and every crossing authenticated and labelled. HTML page SVG draw.io
34 Controller Pod K8s Service Account Entra ID Key Vault Signal Source Capacity API Audit Log 1. projected token 2. federated exchange 3. token · 1 h · scoped 4. read as signal-reader 5. no secret to fetch 6. actuate as capacity-writer 7. scoped to this group only 8. another group: denied 9. every exchange Three Identities, No Standing Credentials Reading signals, writing desired state and actuating capacity are three separate workload identities. None of them is a fleet-wide administrative role. v 1.0 · owner Data & AI Global Practice · date 2026-09 Identity Separation Three workload identities, federated, short-lived, and scoped to the capacity groups they manage. HTML page SVG draw.io
35 Roles Viewer read everything Policy Author propose versions Operator override, pause Administrator bounds, ceilings Change path Proposed Change with justification Approval Gate role dependent Extra scrutiny Second Approver ceiling or max Mandatory Expiry overrides only Effect Versioned Config effective from Still Inside Bounds no exception Evidence Audit Record actor, before, after 13 Months tamper evident 4 h default always Who May Change What Person or role Application we own Decision point Security / platform Data store synchronous An operator can pin a target but cannot raise its maximum. Raising a maximum needs an administrator and a second approver, and is recorded either way. v 1.0 · owner Data & AI Global Practice · date 2026-09 RBAC and Change Control Four roles, one change path, and the two things that need a second pair of eyes. HTML page SVG draw.io
36 Loss One Signal Stale past 3 intervals Signal Store Down completely blind Decision Engine Down Immediate Fail Static hold last declared No Scale-In While Blind never guess down Actuation Continues desired is durable If it persists Past a Diurnal Turn declared threshold Schedule Baseline opt-in per target Stay Frozen the default Always Alert with Cause names the source Journalled as Degraded a real outcome if declared otherwise unaffected What It Does When It Cannot See Risk / gap Security / platform Application we own Decision point Data store synchronous Freezing is right for an outage measured in minutes and increasingly wrong past a diurnal turn. Which of the two applies is a per-target declaration, not a runtime guess. v 1.0 · owner Data & AI Global Practice · date 2026-09 What It Does When It Cannot See Fail static, then — only if declared — decay towards a schedule. HTML page SVG draw.io
37 Signal failures Assumed to occur Staleness source stops Corruption implausible value Store Outage Clock Skew Handling Mark Stale, Degrade Clamp, Quarantine Fail Static no scale-in Reject, Alert Actuation failures Assumed to occur API Throttling Capacity Scarcity Partial Actuation Downstream Saturation Handling Backoff, Breaker 5 failures Zone / Shape List Re-Derive from Actual Dependency Ceiling inhibit scale-out Controller failures Assumed to occur Crash Mid-Decision Split Brain two replicas Policy Misconfigured Poison Target crash-looping Handling Durable Desired State resume, do not redo Lease per Shard no lease, no write Hard Bounds, Rate Caps No Throughput Gain 3 intervals: freeze Named Failure Modes and Their Handling Risk / gap Security / platform synchronous Twelve failure classes, each with a named handling. None of them is "retry until it works", and none of them removes capacity that is currently serving. v 1.0 · owner Data & AI Global Practice · date 2026-09 Failure Modes Twelve classes assumed to occur, each with its named handling. HTML page SVG draw.io
Open svg/<view>.svg or drawio/<view>.drawio in draw.io Desktop or at app.diagrams.net to edit. The SVG carries the diagram inside it, so it is both the picture and the source. This folder is self-contained — copy it whole and every link still resolves.
Solution Architecture v1.0 · Data & AI Global Practice · 2026-09 · 37 views · Azure and open source37 views · HTML / SVG / draw.io