Digs

Searching X architecture returns material written to be found by that query, which means tutorials and vendor pages. The material worth having was written by an engineer explaining a real system to their peers, and it uses different words. A dig is what comes back from going after that layer on one topic: the architecture reconstructed across real systems, the decisions and the conditions that flip them, what actually broke, and numbers you can plan against. Every claim is graded, linked and dated.

88 field guides 2376 graded sources 412 postmortems 8 categories

SLOs, observability, incident response, capacity, degradation and failover.

Reliability & Operations 11 Oct 2026

Evacuating the region: who can actually leave, and what it costs

Reconstructs regional evacuation from GitLab's public DR blueprints, runbooks and gameday timing ledger, Meta's Maelstrom and Taiji papers, Netflix's active-active record, Datadog's failover-safety work, and t…

What surprised meGitLab's regional-recovery change template pre-prints a 96-hour downtime component, and the biggest failover disaster in the record (GitHub 2018) cam…

26 sources · 15 organisations · 6 postmortems 35 min read
disaster-recoveryfailovermulti-regiongamedaysrto-rpo
Reliability & Operations 07 Oct 2026

Coming back from cold: the restart is a harder load case than the crash

Reconstructs, from the Kubernetes project's own design records, the fixes HashiCorp shipped after October 2021, the Chubby and Physalia papers, Netflix and Meta operational docs, and GitLab's public incident t…

What surprised meThe industry's warm-up machinery states in its own documentation that it assumes something warm is still standing: Envoy's slow start calls itself 'n…

20 sources · 10 organisations · 4 postmortems 26 min read
cold-startthundering-herdbootstrapbackoffcapacity
Reliability & Operations 05 Oct 2026

The way back is also a deploy

Reconstructs rollback as it actually works in production: a forward deploy of an old artifact, underwritten by a compatibility contract fixed weeks before the incident. Six postmortems (Knight, CrowdStrike, Cl…

What surprised meThe most expensive deployment failure on record, Knight Capital, was a rollback that executed cleanly: it restored old code under new configuration a…

28 sources · 25 organisations · 6 postmortems 28 min read
deploymentrollbackcanaryfeature-flagsincident-response
Reliability & Operations 29 Sep 2026

Where resilience policy lives: Netflix, 2016 to 2026

Between 2016 and 2026 almost every Netflix library that decided something about the network was retired, while the libraries that decide something about data were not. This guide reconstructs that decade from …

What surprised meThe four deprecated Netflix libraries all show a December 2025 commit, because one engineer pushed the same CI change into Hystrix, Ribbon, Servo and…

22 sources · 4 organisations · 4 postmortems 28 min read
netflixresilienceload-sheddingservice-discoverydeprecation
Reliability & Operations 25 Sep 2026

Deciding who gets told no

A field guide to request-level admission control, built from GitLab's public incident tracker and six years of its rate-limiting change record, the Envoy ratelimit service and filter contract, Kubernetes API P…

What surprised meIn every incident in the corpus the counting algorithm was innocent: what failed was the classifier that decides who a request is, the exception plan…

26 sources · 13 organisations · 4 postmortems 29 min read
rate-limitingadmission-controlredisenvoykubernetes
Reliability & Operations 17 Sep 2026

Adding capacity under fire

Reconstructs, from postmortems at Slack, AWS, Datadog, Robinhood and Coinbase plus the Kubernetes project's own rejected pull requests, why the mechanism bought to handle demand spikes so often fails during th…

What surprised meNetflix published the full failure catalogue of reactive autoscaling, including the scale-down-into-the-retry-storm trap that later hit Slack, in Nov…

28 sources · 18 organisations · 6 postmortems 31 min read
autoscalingcapacityincident-responsekubernetescloud
Reliability & Operations 16 Sep 2026

The router finds out last: taking a server out of service without dropping requests

Reconstructs the five-step removal sequence that Google, Kubernetes, AWS, Envoy and gRPC all converge on, and shows that implementations differ on exactly one axis: whether the drain closes the loop by subscri…

What surprised meAfter a decade of deploy-time 502s, upstream Kubernetes codified the sleep itself as the fix (KEP-3960, GA v1.34) while the proposal to sequence SIGT…

30 sources · 22 organisations · 3 postmortems 32 min read
graceful-shutdownload-balancingkubernetesdeploysconnection-draining
Reliability & Operations 11 Sep 2026

A backup is a claim, a restore is the proof

GitLab lost six hours of production data in 2017 because five backup mechanisms produced zero working restores; the same failure class returned in July 2026 as a severity-4 non-event because by then a CI pipel…

What surprised meThe version-drift failure that destroyed GitLab's backups in 2017 (pg_dump 9.2 against PostgreSQL 9.6) recurred in July 2026 (restore pipeline on pg1…

25 sources · 8 organisations · 6 postmortems 28 min read
backupsdisaster-recoveryrestore-testingrto-rpopostgresql
Reliability & Operations 10 Sep 2026

When the queue becomes the outage

The buffer that absorbs your bursts is also the mechanism that sustains your worst outages. This guide reconstructs, from postmortems at GitHub, Honeycomb and incident.io, source code and pull requests at Meta…

What surprised meSlack's 2016 job-queue outage wedged because dequeueing a job from full Redis itself required free memory: the full buffer jammed its own exit, and t…

27 sources · 24 organisations · 3 postmortems 31 min read
queuesbackpressureload-sheddingbacklog-recoveryasync-processing