Reliability & Resilience
General material on designing for failure.
4 to work through
-
intermediate
A team is launching a new service and asks what its SLO should be. How do you help them decide, and why is "99.99%" usually the wrong first answer?
2 min answer -
intermediate
The business asks for "100% uptime" for a new customer portal. Walk me through the conversation that ends in an agreed SLO.
3 min answer -
advanced
In July 2019 a single regex deployed globally took Cloudflare's network to 100% CPU within seconds. What does this say about how configuration should be released?
2 min answer -
advanced
Maersk rebuilt roughly 4,000 servers and 45,000 PCs in about ten days after NotPetya in 2017, and recovered its directory only because one data centre had been offline during the attack. What does this say about DR design?
2 min answer
15 terms in this topic
Availability Calculation
Deriving a system's availability from its components, remembering that dependencies in series multiply.
practiceBlameless Postmortem
An incident review that seeks the systemic conditions that made a failure possible, explicitly excluding individual fault.
practiceCapacity Planning
Deciding in advance how much capacity will be needed, given growth, seasonality and failure scenarios, and ensuring it can be there in time.
practiceChaos Engineering
Deliberately injecting failure into a system to discover, before an incident does, which of your resilience assumptions are false.
practiceDisaster Recovery
The plan and capability for restoring service after an event that takes out a whole site, region or system.
metricError Budget
The amount of unreliability an SLO permits, treated as a resource that feature velocity spends.
practiceGame Day
A scheduled exercise in which a failure is deliberately introduced and the team responds as though it were real, to test the system and the response …
practiceIncident Command
Assigning explicit roles during an incident — commander, operations lead, communications lead, scribe — so coordination does not compete with diagnosis.
conceptRedundancy
Having more instances of a component than the load requires, so that failures can be absorbed without loss of service.
metricRTO and RPO
How long recovery may take (RTO) and how much data may be lost (RPO), the two numbers that determine the cost of a resilience design.
conceptService Level Agreement
A contractual commitment about service level, with a defined remedy — usually a service credit — when it is missed.
metricService Level Indicator
The actual measurement of a service's behaviour that an objective is set against — a ratio of good events to valid events.
metricService Level Objective
An internal target for a service level indicator, set below the level at which users notice, and used to decide whether to ship or to stabilise.
practiceSeverity Levels
A predefined scale of incident impact that determines who is woken, how fast, and what process applies.
conceptStatic Stability
The property that a system keeps working on its existing state when its control plane or dependencies are unavailable, rather than needing them to ke…
Neighbouring topics
SLI, SLO & SLA
The measurement, the internal target, and the contractual commitment.
No content yetError Budgets
Unreliability as a resource that feature velocity spends.
Availability Mathematics
Series dependencies multiplying, and redundancy that is not independent.
No content yetRTO & RPO
How long recovery may take and how much data may be lost.
No content yetRedundancy
N+1, N+2 and 2N, correlated failure, and headroom for the failure itself.
No content yetFailover
Detection latency, promotion, fencing and the cost of failing over wrongly.
No content yetGraceful Degradation
Reduced but useful function when a dependency is gone.
No content yetFault Isolation
Cells, zones, tenants and the partitions that bound a failure.
No content yetChaos Engineering
Hypothesis-driven failure injection with a bounded blast radius.
No content yetGame Days
Testing the response — runbooks, access, comms — not only the system.
No content yetIncident Management
Command roles, severity levels and mitigation before diagnosis.
No content yetPostmortems
Finding the systemic conditions, and completing the actions afterwards.
No content yetOn-Call
Sustainable rotations, actionable pages and handover discipline.
No content yetCapacity Planning
What does not autoscale, and the lead-time items that need a date.
No content yetDR Testing
Restore drills, timed against the stated RTO, into a clean environment.
No content yetResilience Testing
Exercising retries, breakers and fallbacks that are otherwise never run.
No content yetStatic Stability
Serving from existing state when the control plane is unavailable.
No content yetDegradation Modes
Deciding in advance what is shed first and what is protected.
No content yetReliability Culture
Blamelessness, error budget policy and reliability as a funded property.
No content yet