Notion runs its product on sharded Postgres and originally loaded analytics into a managed warehouse through off-the-shelf connectors. In 2022 it built its own data lake on change data capture into Kafka, Hudi tables and S3 instead. What forced that, and where would copying it be a mistake?

The situation they were in

Notion's core data is blocks: every paragraph, page and database row a customer edits. Notion sharded Postgres into 480 logical shards in 2021 to keep the operational side alive, which means the analytical side has to pull from a large number of shards rather than one database.

The workload has an awkward property. A block is edited far more often than it is created, so the change stream is dominated by updates, not inserts. Analytical systems are built for appends. An update-heavy stream into an append-oriented target means either constant merge work or a full re-snapshot, and a re-snapshot of the whole block table is not something you run often.

What they chose

An in-house lake, described on their engineering blog: Debezium change data capture from Postgres into Kafka, written into Apache Hudi tables on S3 by Spark, with the transformed and denormalised output served onward to analytics, search and product features.

Hudi was chosen specifically for update-heavy workloads and its native handling of Debezium change messages — the merge-on-read behaviour that makes a stream of upserts cheap to apply is exactly the property their workload needed.

The reported outcome: data freshness improved from days to minutes or hours, and the change saved over $1 million against the managed ingestion and warehouse path.

Why it fit their constraints

• The bill was dominated by ingestion of updates, and that is the one cost an in-house merge-on-read layer attacks directly. A managed connector re-reads and re-writes rows because it cannot see which of the 480 shards actually changed.
• Notion already ran Kafka and Spark, so the new operational surface was smaller than it looks.
• The lake had to feed more than dashboards — search indexes, permission-denormalisation and later AI features. A warehouse optimised for BI is an awkward place to hang those.

What it cost them

An in-house ingestion path is now theirs to run: Debezium connectors per shard group, Kafka retention, compaction and clustering of Hudi tables, and a bootstrap process for new tables that a managed connector used to hide. Every one of those is a pager entry that did not exist before, and a connector that fails silently at 03:00 now produces stale analytics rather than a vendor support ticket.

When this is the wrong move to copy

If your ingestion is insert-heavy, or your managed connector bill is a few thousand a month, this trade is negative. Choose the managed path unless update volume dominates the bill. The saving scales with update volume and shard count; the operational cost is close to fixed and lands on a small team. A company with one Postgres primary and a nightly incremental load should keep the managed path and spend the engineers on modelling.

The transferable lesson is not "build your own lake". It is that the shape of your change stream, not its size, decides whether a general-purpose ingestion product fits you.