Autoscaling Controller  ·  View 37 of 37  ·  6 · Assurance

Failure Modes

Twelve classes assumed to occur, each with its named handling.

Editable source SVG draw.io All views
Signal failures Assumed to occur Staleness source stops Corruption implausible value Store Outage Clock Skew Handling Mark Stale, Degrade Clamp, Quarantine Fail Static no scale-in Reject, Alert Actuation failures Assumed to occur API Throttling Capacity Scarcity Partial Actuation Downstream Saturation Handling Backoff, Breaker 5 failures Zone / Shape List Re-Derive from Actual Dependency Ceiling inhibit scale-out Controller failures Assumed to occur Crash Mid-Decision Split Brain two replicas Policy Misconfigured Poison Target crash-looping Handling Durable Desired State resume, do not redo Lease per Shard no lease, no write Hard Bounds, Rate Caps No Throughput Gain 3 intervals: freeze Named Failure Modes and Their Handling Risk / gap Security / platform synchronous Twelve failure classes, each with a named handling. None of them is "retry until it works", and none of them removes capacity that is currently serving. v 1.0 · owner Data & AI Global Practice · date 2026-09

The rule

  • None of the twelve handlings is "retry until it works", and none of them removes capacity that is currently serving.
  • Split brain is prevented rather than detected: a desired-state write without a valid lease is rejected by the store.
  • Poison target — capacity added with no throughput improvement over 3 intervals — freezes and alerts rather than continuing to add.

Downstream saturation

  • Scaling out can amplify load on a shared dependency until it becomes the outage.
  • A per-target dependency-aware ceiling inhibits scale-out when the declared dependency signal is saturated. This is the failure class most autoscalers do not model at all.

Residual risk

  • A provider that reports success and does nothing is invisible until the next observe. The 3-consecutive-interval actuation alert bounds the exposure to roughly 45 s.
  • A workload that lies about health defeats verify-serving. The poison-target detector is the compensating control and it is deliberately slow.
◀  What It Does When It Cannot See All views
Solution Architecture v1.0 · Data & AI Global Practice · 2026-09 · 37 views · Azure and open source37-failure-modes