Digs

Searching X architecture returns material written to be found by that query, which means tutorials and vendor pages. The material worth having was written by an engineer explaining a real system to their peers, and it uses different words. A dig is what comes back from going after that layer on one topic: the architecture reconstructed across real systems, the decisions and the conditions that flip them, what actually broke, and numbers you can plan against. Every claim is graded, linked and dated.

91 field guides 2462 graded sources 422 postmortems 8 categories
Reliability & Operations 31 Aug 2026

Deciding a server is dead

Reconstructs the failure-detection plane behind health checks from six published incidents, five production codebases, four papers and the Kubernetes and gRPC design records. Shows that mature systems independ…

What surprised meAt least five teams independently invented the same guardrail under five names (self-preservation, panic threshold, panic mode, fail open, velocity c…

30 sources · 24 organisations · 6 postmortems 30 min read
health-checksfailure-detectionkubernetesenvoyfail-open
Cost & Efficiency 30 Aug 2026

When owning hardware wins, and when it owns you

Dropbox banked an SEC-audited $74.6M by leaving S3; 37signals cut its bill from $3.2M to $1.3M on the way to deleting its AWS account; and in the same summer of 2025, Stack Overflow unracked its last server an…

What surprised meThe two most famous cloud skeptics of the 2010s swapped sides within weeks of each other in mid-2025, 37signals deleting its AWS account as Stack Ove…

29 sources · 25 organisations · 3 postmortems 32 min read
cloud-exitrepatriationtcounit-economicscapacity-planning
Cost & Efficiency 30 Aug 2026

Zone-local traffic and the cross-zone bill

Spreading across three availability zones is billed twice: in standing idle capacity, and at a cent per gigabyte for every crossing. This guide reconstructs, from Slack, DoorDash, Grab, Grafana Labs, Uber, Pol…

What surprised meMicrosoft deleted the charge entirely on 21 May 2024, so a whole body of engineering, a product category and three competing Kafka redesigns exist to…

34 sources · 21 organisations · 4 postmortems 44 min read
availability-zoneskuberneteskafkaenvoycloud-cost
Platform & Infrastructure 29 Aug 2026

Keeping one failure from reaching everyone

Slack, Shopify, Salesforce, DoorDash, Amazon EBS and AWS all cut their systems into independent replicas so that one fault hurts some customers rather than all of them, and all of them publish how. This guide …

What surprised meThe 2020 Physalia paper had already identified that cells do not contain deployments and had built a second, orthogonal partition for it, called colo…

28 sources · 22 organisations · 5 postmortems 42 min read
blast-radiuscell-architecturedeploymentavailabilitymulti-tenancy
Data Platforms 29 Aug 2026

The cutover is the easy part

A field guide to online resharding, reconstructed from thirteen production accounts (Notion, Figma, Slack, GitHub, Shopify, Discord, Etsy, Pinterest, Stripe, Adyen, Mercari, Monzo, AWS) and six published incid…

What surprised meNo published postmortem in this corpus blames the cutover, the step every team rehearses and fears. The incidents come from the shard key chosen nine…

34 sources · 23 organisations · 5 postmortems 41 min read
shardingpostgresmysqlvitessmigration
Security & Identity 29 Aug 2026

The renewal failed a month before anyone noticed

A field guide to the machine credential lifecycle, built from ten published incidents at Bazel, DigiCert, Microsoft, Let's Encrypt and Ericsson, the ballots and RFCs cutting public certificate lifetimes to 47 …

What surprised meCertificate lifetime is not the number that governs this failure. The number that governs it is the slack, the count of consecutive silent renewal fa…

22 sources · 17 organisations · 4 postmortems 44 min read
pkicertificatesacmerotationkubernetes
Distributed Systems 29 Aug 2026

The retry is the outage

A field guide to metastable failure: the class of outage where the system stays down after the thing that broke it has been repaired, because the retries, caches, health checks and autoscaling installed to mak…

What surprised meThe remediation from one incident has repeatedly produced the next one: after AWS SimpleDB failed in 2014 engineers made storage servers retry the lo…

30 sources · 19 organisations · 7 postmortems 48 min read
retriesload-sheddingoverloadenvoygrpc
Integration & APIs 29 Aug 2026

You cannot delete the dual write, only move it

A service commits a change and must announce it, and no mechanism makes both happen or neither. This guide reconstructs where eighteen organisations put that non-atomic seam, from Facebook's Wormhole in 2015 t…

What surprised meEvery remedy for the dual write is itself implemented as a dual write, and Kafka's own KIP-618 says so in its motivation: the connector produces reco…

28 sources · 24 organisations · 3 postmortems 41 min read
outboxchange-data-capturekafkapostgreseventing
Reliability & Operations 28 Aug 2026

The rule was already written down

A field guide to the machinery between an outage and the next design decision, reconstructed from six first-party incident reports (Cloudflare 2019 and 2025, AWS October 2025, Google Cloud June 2025, CrowdStri…

What surprised meGoogle Cloud's June 2025 remediation, 'enforce all changes to critical binaries to be feature flag protected and disabled by default', is the rule En…

34 sources · 24 organisations · 6 postmortems 40 min read
postmortemincident-reviewsreerror-budgetsrelease-safety