Portfolio ↗

Digs

Searching X architecture returns material written to be found by that query, which means tutorials and vendor pages. The material worth having was written by an engineer explaining a real system to their peers, and it uses different words. A dig is what comes back from going after that layer on one topic: the architecture reconstructed across real systems, the decisions and the conditions that flip them, what actually broke, and numbers you can plan against. Every claim is graded, linked and dated.

91 field guides 2462 graded sources 422 postmortems 8 categories
All AI & LLM Systems8 Data Platforms12 Distributed Systems10 Platform & Infrastructure24 Integration & APIs11 Security & Identity8 Reliability & Operations11 Cost & Efficiency7
AI & LLM Systems 26 Sep 2026

Keeping the training run alive

The only two complete public training logbooks (Meta's OPT-175B, BigScience's BLOOM) record what actually stops a big run: roughly two machine deaths a day, stalls that no timeout catches, one slow GPU taxing …

What surprised meThe most expensive failures in both logbooks were failures of the recovery machinery itself, including the cloud provider's support team deleting the…
23 sources · 12 organisations · 8 postmortems 28 min read
trainingfault-tolerancecheckpointinggpugoodput
Reliability & Operations 25 Sep 2026

Deciding who gets told no

A field guide to request-level admission control, built from GitLab's public incident tracker and six years of its rate-limiting change record, the Envoy ratelimit service and filter contract, Kubernetes API P…

What surprised meIn every incident in the corpus the counting algorithm was innocent: what failed was the classifier that decides who a request is, the exception plan…
26 sources · 13 organisations · 4 postmortems 29 min read
rate-limitingadmission-controlredisenvoykubernetes
Platform & Infrastructure 25 Sep 2026

Your singletons choose where you fail: ten years of GitLab's architecture

A decade of one company's platform evolution, reconstructed entirely from primary artefacts it publishes in Git repositories: 207 design documents, two rejected routing proposals, a closed merge request that d…

What surprised meThe horizontal-scalability programme rewrote its own goal: Cells opened in 2022 as a platform re-architecture and its current iteration describes its…
28 sources · 2 organisations · 5 postmortems 29 min read
multi-tenancyshardingmonolithpostgresqlgit-storage
AI & LLM Systems 24 Sep 2026

Everything that demanded the whole program got archived: ten years of Meta's ML platform

A decade of one company's answer to a problem every deployment path has: how much of a dynamic program you capture ahead of time, and what you do with the part you cannot. Read from defaults at release tags ra…

What surprised meIn PyTorch 2.8 the signature of torch.export.export changed from strict=True to strict=False, so the replacement for TorchScript repeated TorchScript…
24 sources · 11 organisations · 2 postmortems 30 min read
pytorchcompilersdeploymentdeprecationsupply-chain
Distributed Systems 24 Sep 2026

Your write hasn't happened here yet

Reconstructs the four-part machine behind read-your-writes (mint a position token at commit, carry it with the client, route to a caught-up replica, keep a bounded escape hatch) from the implementations at Git…

What surprised meEvery layer of the stack independently converged on the same four-part token machine, while the database underneath took three landing attempts acros…
24 sources · 16 organisations · 4 postmortems 28 min read
replicationconsistencyread-replicaspostgresqlmysql
Integration & APIs 23 Sep 2026

Putting the services back together: ten years of Airbnb, read from its own artefacts

Reconstructs a decade of one company's architecture from artefacts rather than announcements: release timestamps on five package registries, archive notices, in-repository design documents and two security adv…

What surprised meThe remedy decayed the same way the disease did: four years after Viaduct was started to fix microservice sprawl, Airbnb rebuilt it from the ground u…
28 sources · 4 organisations · 3 postmortems 29 min read
airbnbgraphqlmicroservicesmodularityarchaeology
Cost & Efficiency 23 Sep 2026

The meter runs on cardinality

Reconstructs the telemetry cost-containment funnel from the incident record at GitLab and Datadog, the pricing sheets of three clouds, Prometheus and OpenTelemetry source debates including a rejected pull requ…

What surprised meThe organisations that take observability most seriously throw almost everything away on purpose: Google's Dapper kept 1 trace in 1,024 and then samp…
37 sources · 20 organisations · 8 postmortems 35 min read
observabilitycardinalitytelemetrycostsampling
Distributed Systems 22 Sep 2026

How long to wait: timeouts and deadlines in distributed systems

A field guide to timeouts and deadlines: the difference between a local per-call timeout and a propagated deadline, the stack of independent timers every mature system exposes, and the three failure classes th…

What surprised meThe maintainers of the most-used HTTP libraries in the world still cannot agree on whether a default timeout should exist: requests parked the questi…
22 sources · 13 organisations · 3 postmortems 28 min read
timeoutsdeadlinesgrpcresiliencerpc
Integration & APIs 22 Sep 2026

Making the client carry it: ten years of Discord's gateway contract

A fanout platform pays for connected consumers multiplied by events, and it cannot deploy a fix to consumers it does not own. This guide reconstructs the six controls Discord added to its gateway contract over…

What surprised meEvery capability Discord took back had a pull-shaped substitute shipped first: interactions landed in December 2020, twenty months before message con…
28 sources · 7 organisations · 4 postmortems 31 min read
websocketfanoutapi-versioningrate-limitingdeprecation