Portfolio ↗

Digs

Searching X architecture returns material written to be found by that query, which means tutorials and vendor pages. The material worth having was written by an engineer explaining a real system to their peers, and it uses different words. A dig is what comes back from going after that layer on one topic: the architecture reconstructed across real systems, the decisions and the conditions that flip them, what actually broke, and numbers you can plan against. Every claim is graded, linked and dated.

91 field guides 2462 graded sources 422 postmortems 8 categories
All AI & LLM Systems8 Data Platforms12 Distributed Systems10 Platform & Infrastructure24 Integration & APIs11 Security & Identity8 Reliability & Operations11 Cost & Efficiency7
Platform & Infrastructure 12 Sep 2026

The update that expires: ten years of Datadog on machines it does not own

A decade of one observability company read through the only part of it that runs on other people's computers, reconstructed from commit history, systemd unit templates, CI budget files, closed pull requests an…

What surprised meThe commit rate on the abandoned Agent 5 repository went back up after 2022, and the work is certificate-authority rotation runbooks written for bash…
28 sources · 4 organisations · 3 postmortems 32 min read
fleet-managementrelease-engineeringagentssystemdremote-configuration
Reliability & Operations 11 Sep 2026

A backup is a claim, a restore is the proof

GitLab lost six hours of production data in 2017 because five backup mechanisms produced zero working restores; the same failure class returned in July 2026 as a severity-4 non-event because by then a CI pipel…

What surprised meThe version-drift failure that destroyed GitLab's backups in 2017 (pg_dump 9.2 against PostgreSQL 9.6) recurred in July 2026 (restore pipeline on pg1…
25 sources · 8 organisations · 6 postmortems 28 min read
backupsdisaster-recoveryrestore-testingrto-rpopostgresql
Platform & Infrastructure 11 Sep 2026

Where Shopify refused to rewrite, and where it rewrote three times

One company running two opposite policies in the same decade. On the substrate it can change, Shopify has not migrated since 2014: it funded a JIT inside CRuby, adopted a database driver and upstreamed the ada…

What surprised meShopify could not fit its own language into its own extension platform: Ruvy was archived in January 2026 because its WebAssembly modules exceed the …
32 sources · 9 organisations · 4 postmortems 34 min read
rubymonolithruntimewebassemblyresiliency
Distributed Systems 10 Sep 2026

When the answer is wrong, not late: ten years of CockroachDB read from its own record

A decade of one distributed database company read from artefacts nobody edits for effect: technical advisories with affected version ranges, design records with a status field including the rejected ones, and …

What surprised meThe public design-record process was retired between the February and May 2024 releases and moved to an internal wiki, while a competing vendor's pub…
26 sources · 3 organisations · 8 postmortems 34 min read
distributed-sqlcorrectnesspostmortemsstorage-enginemultitenancy
Reliability & Operations 10 Sep 2026

When the queue becomes the outage

The buffer that absorbs your bursts is also the mechanism that sustains your worst outages. This guide reconstructs, from postmortems at GitHub, Honeycomb and incident.io, source code and pull requests at Meta…

What surprised meSlack's 2016 job-queue outage wedged because dequeueing a job from full Redis itself required free memory: the full buffer jammed its own exit, and t…
27 sources · 24 organisations · 3 postmortems 31 min read
queuesbackpressureload-sheddingbacklog-recoveryasync-processing
Integration & APIs 09 Sep 2026

The reference architecture Netflix retired

Between 2013 and 2016 the industry copied one company's answer to service-to-service communication: discovery, load balancing, circuit breaking and configuration shipped as client libraries compiled into every…

What surprised meRibbon's own status notice marks four of its nine published modules as not used internally and says the production ones were wrapped in an internal c…
26 sources · 7 organisations · 4 postmortems 32 min read
service-discoveryresilienceapi-gatewaycontrol-planeopen-source-risk
Security & Identity 09 Sep 2026

Trusting the build pipeline

A CI job holds every credential you own and, on a public repo, runs code proposed by strangers. This guide reconstructs seven incidents (Codecov, CircleCI, Nx, tj-actions, Salesloft, Travis, PyTorch) into thre…

What surprised meNone of the seven incidents required breaking cryptography or a platform zero-day; each was a credential that should not have been reachable or code …
22 sources · 20 organisations · 7 postmortems 27 min read
ci-cdsupply-chainoidcsecretsgithub-actions
Platform & Infrastructure 08 Sep 2026

Changing a platform without stopping it: a decade of Zalando's cluster fleet

Zalando's production cluster configuration has been a public repository since 2017, and it took a change on the day this guide was written. The guide reads a decade out of it: the branch-per-risk-appetite prom…

What surprised meThe decision records argue for three channels and the running pipeline has four, promoted through two standing pull requests that nobody wrote down; …
20 sources · 2 organisations · 2 postmortems 28 min read
kubernetesplatform-engineeringfleet-upgradeadrzalando
AI & LLM Systems 08 Sep 2026

When the model quietly gets worse

A field guide to the production incident where answer quality drops while availability, latency and error dashboards stay green, because the change was in a router, a kernel, a precision mode, a prompt or a th…

What surprised meIn every documented incident the detector was a user, never an automated gate, because request-level monitoring is structurally blind to a quality re…
23 sources · 17 organisations · 4 postmortems 27 min read
llm-servingevaluationobservabilityinferencepostmortem