In 2017 The New York Times described replacing a web of API and feed integrations with a single ordered Kafka log - the Monolog - holding every published asset as the source of truth, with back-end systems consuming from it. What forced that change, what did it cost, and where would copying it be a mistake?

The situation they were in

A publisher accumulates consumers of content faster than it accumulates content: the website, the mobile apps, search indexing, personalisation, archives, syndication, alerting. In a request-oriented integration estate, each new consumer is a new integration with the producing system, and each one needs its own back-fill story when it launches or when its data model changes.

That is the cost that grows quadratically. Not the traffic, the coupling: N consumers each holding an opinion about how the publishing system's API behaves, so the publishing system cannot change.

What they chose

An immutable, totally ordered log as the interchange. The New York Times published every asset to a single-partition Kafka topic and had back-end systems consume it, with the log - not any one database - as the source of truth for published content.

Three properties do the work:

• Order is a property of the log, not of each consumer. Every consumer sees the same sequence, so reconciliation questions ("did these two systems apply the updates in the same order?") disappear rather than being answered per integration.
• A new consumer back-fills by replaying from offset zero. No bespoke migration, no export job, no coordination with the publishing team. This is the property that changes how fast the organisation can add systems, and it is why "log" beats "queue" here: a queue deletes what it delivers.
• Producers are decoupled from consumers in time. A consumer that is down or slow is a consumer with lag, not a failed integration.

Why it fitted their constraints

The content corpus is large in history and modest in throughput - a newsroom does not publish at social-network rates. A single partition is the strongest ordering guarantee available and it is only affordable when volume is low enough for one partition to keep up. That constraint is what makes the elegant version possible.

Their data also has the right shape for a log: published assets are facts, immutable once emitted, meaningful forever. Ordering is a genuine business requirement, because a correction must not be overtaken by the article it corrects.

What it cost

Everything that reads the log must handle the whole history. Schema evolution becomes permanent: a consumer replaying from zero in 2026 must still parse a 2017 message. That forces disciplined, additive schema management and a normalised internal representation - work that a point-to-point integration defers forever. Retention becomes a storage bill rather than an operational convenience, and consumer lag becomes an operational concept every team must understand.

Where copying it would be a mistake

• High-volume mutable data. A single ordered partition does not scale, and once you shard for throughput you lose global ordering, which was the main reason to do this.
• Data governed by erasure obligations. An immutable log of personal data collides directly with deletion rights; crypto-shredding or tokenisation becomes mandatory rather than optional.
• Two or three consumers. The quadratic coupling problem is not real yet, and a Kafka cluster with its own operational team is a large fixed cost to solve a problem you do not have.
• Transactional state. Published content is append-only by nature. An order, a balance or an inventory count is not, and modelling it as a log means rebuilding a database badly.

Common weak answers

• "They used Kafka for scale." They used it for ordering and replay. The volume was never the hard part.
• "Event-driven is better than request-response." This is one estate with one data shape. The answer flips on volume, mutability and legal obligations.