The New York Times described (2017) moving to a Kafka log as the source of truth for all published content, with downstream services building their own views by consuming it. For an organisation whose audience can rise many times over on an election night while the content itself is small, what forced that change, what did it cost, and where would copying it be a mistake?

The situation they were in

A publishing estate accumulated over decades: multiple content management systems, several APIs, and every downstream consumer integrating with each producing system directly. Two consequences follow from that shape, and both are capacity problems in disguise.

Adding a consumer cost an integration per producer, so the cost of a new reader-facing surface grew with the number of producing systems. And bootstrapping a consumer's history meant calling the producers' APIs, so a backfill put load on the systems serving live readers — exactly when a new service is being launched.

What they chose

One append-only log as the source of truth, with every published asset written to it and every downstream service consuming the log to build whatever view it needs.

Why it fit their constraints

The content is small and the audience is large. An article is kilobytes; the whole published corpus is modest by modern standards. That is the property that makes a log-as-truth design affordable here — replaying everything is feasible — and it is the property most organisations copying it do not have.

For capacity, the consequences are direct:

• The read path becomes precomputed and cacheable. Consumers build their own views, which can be flat, denormalised and fronted by a CDN. An election-night spike then lands on a CDN rather than on a database, and the cache hit ratio is high because every reader wants the same few pages.
• A new consumer costs nothing from the producers. It reads the log from the beginning, so the backfill load falls on the log rather than on systems serving readers.
• Recovery is replay. A corrupted or lost view is rebuilt from the log instead of restored from a backup, which changes the recovery-point conversation entirely.

What it cost them

Every consumer owns a view and its correctness. Consumers can diverge, and a projection bug produces wrong content that looks healthy. The organisation needs per-consumer monitoring of lag and a way to detect divergence.

Schema evolution across a log that is never deleted. Content written years ago must remain readable by today's consumers, which means versioned payloads and upcasting maintained indefinitely.

Replay time is a planning number. A full rebuild is bounded by log volume and consumer throughput, and that number decides whether a projection bug is a 20-minute fix or an overnight one. It must be measured before it is needed.

Eventual consistency is now visible to editors. A correction published at 20:00 appears in different surfaces at slightly different times, which is an editorial and product problem rather than a technical one.

When not to copy this

When the data is large. A log as the source of truth for petabytes of user-generated media is not replayable, and the design's main benefit evaporates while its costs remain. The honest threshold is whether a full replay fits in a window the business can tolerate.

When writes dominate, or reads are per-user rather than shared. The election-night property is that millions of readers want the same handful of pages, so precomputation and caching are nearly free. A personalised feed, where every reader's view differs, gets none of that.

When the organisation cannot own many consumers. This design moves correctness into every consumer, which needs enough engineering capacity to monitor and operate them. A small team is better served by one database, read replicas and a CDN in front of the read path, which handles a large spike for a fraction of the complexity.

And the capacity lesson generalises further than the architecture does: for a predictable, shared-content spike, the right answers are precomputation, a CDN and static fallbacks. The log is how this particular organisation made precomputation cheap and recovery possible; it is not what made the spike survivable.