Edge Cache and CDN Platform  ·  View 29 of 29  ·  7 · Assurance

Failure Modes

The requirement's named failures: how each is detected, what contains it, what a user sees and how service returns to normal.

Editable source SVG draw.io All views
Detected by Contained by User sees Back to normal Cache node loss Katran health check Ring drops one owner Nothing 1/N misses Node rejoins warm persistent NVMe Whole-PoP loss BGP session loss Anycast to neighbours n−1 capacity +10–40 ms Reconcile, then warm Transit brown-out External probes sustained 3 min Guarded withdrawal One reconnect Re-announce after 15 min clean Origin down Shield: errors, not latency Freshness class Stale or declared error Ordered endpoint failover Flash crowd · 8× Rate per key Collapse at edge + shield Nothing No action Shield site down Parent health Declared secondary direct path capped Slower misses Primary refills Poison object Owner · Cache-Status Tag purge · short 5xx TTL Bad object ≤ 5 s after purge Refetch Purge storm Stream lag Per-property budgets coalesced tags Nothing serving unaffected Backlog drains Poison configuration Validator · canary gate Halt at first ring Canary PoP ≤ 10 min Revert ≤ 60 s Control plane down Agent heartbeat Last-known-good signed only Nothing for 7+ days Queued purges drain Cache key defect Vary mismatch · probe Rule set to bypass Possible disclosure Security incident property purge Failure Modes — Detection, Containment, What Users See Certificate expiry and storage corruption are on views 28 and 09: renewal at day 5 of 14, and a checksum failure is always a miss. v 1.0 · owner SRE · date 2026-09

The one that is different

  • A cache key defect is the only failure here that does not show up as errors, latency or lower hit ratio. It shows up as a correct-looking response to the wrong person. It is handled as a security incident: the rule is set to bypass, the property is purged, and access records are searched by key hash to find who received what.

Decisions

  • Transit brown-out is treated as PoP failure, but only after three minutes of external probe evidence. Re-announcing waits for fifteen clean minutes, so a flapping transit does not become a flapping PoP.
  • pop-guard will not withdraw a PoP if external probes cannot reach the anycast prefix through at least two other PoPs. A shared cause, such as a bad health rule, must not be able to withdraw every PoP one after another.

Covered elsewhere

  • Certificate expiry on view 28. Storage corruption on view 09: a checksum failure on read is a miss, the object is evicted and refetched, and it is never served.
◀  TLS Keys in Facilities We Do Not Own All views
Solution Architecture v1.0 · Edge Architecture · 2026-09 · 29 views · open source, fully on-premises29-failure-modes