A publisher keeps every piece of published content in one ordered Kafka log and rebuilds every downstream store by replaying it, as The New York Times described in 2017. What does that buy, what does the ordering requirement cost, and where would copying it be a mistake?

The situation they were in

Content produced since the paper's founding lived in many systems with many APIs, and every new consumer of published content had to integrate with all of them. The 2017 design replaced that with a single log, which they call the Monolog, as the source of truth for published content: every producing system appends an asset to it, and every consuming system builds its own store by reading the log from the beginning.

What the design actually requires

Two properties carry the whole architecture, and both are stated explicitly in their write-up.

• The log retains everything forever. A consumer that must be able to rebuild from scratch cannot depend on a retention window. This is the opposite of the usual Kafka default, where the log is a buffer of recent events and the database holds the truth.
• Consumption is ordered. Rebuilding a store by replay only produces the same answer as the live stream if events are applied in the same order, which pushes the design towards a single partition for the normalised log and therefore towards a throughput ceiling set by one consumer's ability to keep up.

They publish assets normalised: an image is its own message, referenced by the articles that use it, so one image update does not rewrite every article. Messages are Protobuf against a defined schema, which is what makes a seven-year-old message still readable by today's consumer.

The second and third logs, and why they exist

A normalised log is cheap to write and awkward to consume: every consumer must resolve references itself, which means every consumer reimplements the same joins. So the pipeline also publishes a denormalised log, where each top-level asset appears with its dependencies already resolved, on many partitions, and a skinny log carrying only notifications that a piece of content was processed, which is what caches and SLO tracking consume.

That three-log shape is the transferable lesson. The normalised log is the truth, the denormalised log is a materialised view maintained once instead of by every consumer, and the notification log exists because most consumers only need to know that something changed.

What it costs

Replay is a capability you pay for continuously: schema discipline forever (a consumer replaying from 2017 must handle 2017's messages), storage that only grows, and a rebuild time that grows with history. A full rebuild that takes hours is fine for a publishing system and unacceptable for a system that must recover in minutes, so the log-as-truth design assumes the live stores are still serving while a rebuild runs.

When not to copy this

A newsroom's publishing volume is orders of magnitude below a clickstream or a telemetry feed. Ordered single-partition consumption is affordable precisely because the volume is small. Apply the same design to a clickstream, an IoT fleet or a payments ledger and the ordering requirement collides with throughput immediately: you need partitioning by key, which gives you per-key ordering only, and the "rebuild anything from the log" property weakens to "rebuild per-key state from the log".

Choose this when the corpus is small, long-lived, and read by many independent consumers. It flips when volume forces partitioning, when history is legally required to be deletable (an immutable forever-log and a right-to-erasure obligation are in direct conflict), or when there is exactly one consumer, in which case the log is a queue and the database is the truth.