Portfolio ↗

Digs

Searching X architecture returns material written to be found by that query, which means tutorials and vendor pages. The material worth having was written by an engineer explaining a real system to their peers, and it uses different words. A dig is what comes back from going after that layer on one topic: the architecture reconstructed across real systems, the decisions and the conditions that flip them, what actually broke, and numbers you can plan against. Every claim is graded, linked and dated.

91 field guides 2462 graded sources 422 postmortems 8 categories
All AI & LLM Systems8 Data Platforms12 Distributed Systems10 Platform & Infrastructure24 Integration & APIs11 Security & Identity8 Reliability & Operations11 Cost & Efficiency7
Data Platforms 21 Sep 2026

The lock is brief. The queue is the outage.

Reconstructs live schema change from the places fourteen organisations wrote their lessons down: GitLab's production incident tracker, gh-ost's design docs and bug reports, the CockroachDB and TiDB implementat…

What surprised meWhere lock-timeout-and-retry is fully institutionalised the failure mode changes address: all four GitLab 2026 migration incidents are severity-3 dep…
24 sources · 14 organisations · 4 postmortems 28 min read
schema-migrationpostgresmysqlddlgh-ost
Platform & Infrastructure 21 Sep 2026

When the internal fork wins: ten years of Netflix retiring its published platform libraries

Netflix published the reference implementation of the fat client library, then dismantled it in public over ten years, one dated commit at a time. This guide gives an architect the six-stage lifecycle every on…

What surprised meThe remediation published for a Critical server-side request forgery advisory was to move the vulnerable component to a different GitHub organisation…
22 sources · 6 organisations · 7 postmortems 32 min read
open-sourcedependenciesdeprecationmicroservicesgovernance
Data Platforms 20 Sep 2026

When the database cannot be split: ten years of GitLab.com, read from its own incidents and design documents

A single-company archaeology of the problem every growing platform eventually hits: two components that cannot be sharded without rewriting the application above them. A reader leaves with a decision tree for …

What surprised meThe decade's most consequential number is a table size, not a latency: index builds on a 1.5 TB table ran 1.5 to 6 hours and blocked deployment, so G…
30 sources · 1 organisations · 9 postmortems 35 min read
postgresqlshardinggitlabcapacity-planningincident-review
Security & Identity 20 Sep 2026

When the secret leaks: the hour after exposure

A field guide to the race that starts when a token goes public: who is allowed to kill a credential automatically, how fast the machinery actually works, and why every incident in the corpus was reopened not b…

What surprised meRevocation, once aimed, worked in hours everywhere in the corpus; what reopened every incident was enumeration, the credential nobody listed, like th…
26 sources · 18 organisations · 7 postmortems 28 min read
secretscredential-rotationsupply-chainci-cdrevocation
Data Platforms 19 Sep 2026

Swapping the engine under the client: ten years of Pinterest, read from its own repositories

A decade of one company's data platform read from artefacts rather than announcements, built to answer a single question: what actually decides whether replacing a storage or messaging engine is a migration or…

What surprised meThe organisation's public record inverts itself over the decade: the systems Pinterest built and published are archived or unmaintained, the ten-year…
30 sources · 2 organisations · 5 postmortems 31 min read
pubsubobject-storagekafkamigrationpinterest
Distributed Systems 19 Sep 2026

When two hold the lock: what the repositories admit about distributed mutual exclusion

Every mainstream distributed lock ships with a written admission that it cannot guarantee mutual exclusion, and this guide reads those admissions where they live: client-go's package comment, etcd's post-Jepse…

What surprised meThe industry's real safety mechanism is not the lock but deliberate process death: a rejected Kubernetes pull request records the design intent in on…
22 sources · 13 organisations · 3 postmortems 26 min read
distributed-locksleasesfencingleader-electionsplit-brain
AI & LLM Systems 18 Sep 2026

Keep the format dumb: ten years of Hugging Face, measured from its own releases

Reads a decade of one company's decisions as a single argument about where a capability belongs, in the file format every reader must parse or in the loader only you control. A reader leaves able to defend tha…

What surprised meThe v5 cleanup removed 113 TensorFlow and Flax model files and 17.7% of the library's lines while adding 29 model directories, but left occurrences o…
20 sources · 9 organisations · 5 postmortems 31 min read
hugging-facemodel-serializationsupply-chaindeduplicationdeprecation
Security & Identity 18 Sep 2026

The leak goes around the tenant filter, not through it

A field guide to tenant isolation built from seven published cross-tenant incidents (Steam 2015, Cloudbleed 2017, GitHub 2021, ChaosDB 2021, AutoWarp 2022, Atlassian 2022, ChatGPT 2023) and the placement desig…

What surprised meIn every published cross-tenant incident in this corpus the tenant-scoping layer worked exactly as designed; the data crossed tenants in a layer with…
27 sources · 25 organisations · 5 postmortems 31 min read
multi-tenancytenant-isolationrow-level-securitycachingblast-radius
Reliability & Operations 17 Sep 2026

Adding capacity under fire

Reconstructs, from postmortems at Slack, AWS, Datadog, Robinhood and Coinbase plus the Kubernetes project's own rejected pull requests, why the mechanism bought to handle demand spikes so often fails during th…

What surprised meNetflix published the full failure catalogue of reactive autoscaling, including the scale-down-into-the-retry-storm trap that later hit Slack, in Nov…
28 sources · 18 organisations · 6 postmortems 31 min read
autoscalingcapacityincident-responsekubernetescloud