Portfolio ↗

Digs

Searching X architecture returns material written to be found by that query, which means tutorials and vendor pages. The material worth having was written by an engineer explaining a real system to their peers, and it uses different words. A dig is what comes back from going after that layer on one topic: the architecture reconstructed across real systems, the decisions and the conditions that flip them, what actually broke, and numbers you can plan against. Every claim is graded, linked and dated.

90 field guides 2433 graded sources 419 postmortems 8 categories
All AI & LLM Systems8 Data Platforms12 Distributed Systems10 Platform & Infrastructure24 Integration & APIs11 Security & Identity8 Reliability & Operations11 Cost & Efficiency6

Storage, ingestion, lakehouse, streaming pipelines, query engines and data governance.

Data Platforms 12 Oct 2026

The data survives the upgrade. The database around it does not.

Reconstructs major-version database upgrades from GitLab's public change and incident tracker, Zalando's Spilo automation, Supabase's platform docs and PostgreSQL's own commit history. A reader leaves with the…

What surprised meThe engine itself committed and then reverted the fixes for two of the four dropped-state traps (sequence replication in 2022, per-index collation tr…
22 sources · 8 organisations · 5 postmortems 31 min read
postgresqlpg-upgradelogical-replicationcutovercollation
Data Platforms 06 Oct 2026

Correctness, then cost, then neighbours: ten years of TiDB, read from its own design record

PingCAP has published a dated design document or RFC for nearly every substantial change to TiDB and TiKV since 2018, plus a release note for every version since 2017. Read in order, that record shows the bind…

What surprised meThe storage redesign that was meant to take clusters from terabytes to petabytes never appears in a release note after October 2023; what users actua…
26 sources · 5 organisations · 4 postmortems 38 min read
distributed-sqlraftmulti-tenancyschema-changestorage-engines
Data Platforms 30 Sep 2026

The shim outlives the migration: ten years of Sentry, read from its own deletions

Every team that outgrows its first database plans the new one; almost nobody plans the period when both are running. This guide measures that period across a decade of Sentry, reconstructed from git: five node…

What surprised meThe dual-backend shims outlived what they were built to retire: 4 months for one, 66 for another, and 91 and counting for the third, whose dead Redis…
31 sources · 3 organisations · 6 postmortems 34 min read
sentryclickhousekafkastorage-migrationtechnical-debt
Data Platforms 27 Sep 2026

There is no neutral identifier

Reads the repositories of GitLab, Rails, PostgreSQL, Twitter, Mastodon, Nextcloud, CockroachDB and the IETF's UUID revision to reconstruct how identifiers are chosen, sized, ordered and exposed. A reader leave…

What surprised meThe two halves of the database industry give opposite advice about the same property and both are right: the PostgreSQL world spent a decade making k…
24 sources · 15 organisations · 4 postmortems 29 min read
identifiersprimary-keysuuidsnowflakepostgres
Data Platforms 26 Sep 2026

Every backend you add is a six-year promise: ten years of Grafana Labs in git

A decade of Grafana Labs reconstructed entirely from its own git history across ten repositories: the convergence of Mimir, Loki, Tempo and Pyroscope on object storage plus a Kafka write path, the fork decisio…

What surprised meThe rewrites all landed roughly on schedule; the removals took four to six years and arrived in one wave, and one of them was reverted eight days aft…
26 sources · 4 organisations · 4 postmortems 35 min read
observabilityobject-storagedeprecationkafkaforks
Data Platforms 21 Sep 2026

The lock is brief. The queue is the outage.

Reconstructs live schema change from the places fourteen organisations wrote their lessons down: GitLab's production incident tracker, gh-ost's design docs and bug reports, the CockroachDB and TiDB implementat…

What surprised meWhere lock-timeout-and-retry is fully institutionalised the failure mode changes address: all four GitLab 2026 migration incidents are severity-3 dep…
24 sources · 14 organisations · 4 postmortems 28 min read
schema-migrationpostgresmysqlddlgh-ost
Data Platforms 20 Sep 2026

When the database cannot be split: ten years of GitLab.com, read from its own incidents and design documents

A single-company archaeology of the problem every growing platform eventually hits: two components that cannot be sharded without rewriting the application above them. A reader leaves with a decision tree for …

What surprised meThe decade's most consequential number is a table size, not a latency: index builds on a 1.5 TB table ran 1.5 to 6 hours and blocked deployment, so G…
30 sources · 1 organisations · 9 postmortems 35 min read
postgresqlshardinggitlabcapacity-planningincident-review
Data Platforms 19 Sep 2026

Swapping the engine under the client: ten years of Pinterest, read from its own repositories

A decade of one company's data platform read from artefacts rather than announcements, built to answer a single question: what actually decides whether replacing a storage or messaging engine is a migration or…

What surprised meThe organisation's public record inverts itself over the decade: the systems Pinterest built and published are archived or unmaintained, the ten-year…
30 sources · 2 organisations · 5 postmortems 31 min read
pubsubobject-storagekafkamigrationpinterest
Data Platforms 15 Sep 2026

Own the layer above the engine: ten years of LinkedIn's data platform

Reconstructs a decade of data-platform decisions at LinkedIn from artefacts rather than announcements: archive notices, dated release listings on Maven Central, PyPI, npm, Docker Hub and the Go proxy, publishe…

What surprised meThe registry record inverts the production record: Voldemort, retired from production in 2018, is fully downloadable, while Venice, the system Linked…
22 sources · 6 organisations · 5 postmortems 31 min read
linkedinderived-dataservingcatalogopen-source