Denormalised Log

A normalised log is the cheap side of a bad bargain. Publishing each entity as its own message keeps writes small and updates surgical: change an image once and no article is rewritten. Then the first consumer discovers that an article message is mostly identifiers, and to render anything it must look up the image, the author, the section and the tags. So it builds those lookups. So does the second consumer, and the third, each with its own cache, its own staleness, and its own version of the bug where a reference resolves to nothing.

The denormalised log is the decision to do that resolution once, in the pipeline, and publish the result as a second topic. The normalised log remains the source of truth; the denormalised log is a materialised view of it, maintained centrally, with the same durability and replay properties as the truth it derives from.

Why it matters

The alternative is not "consumers do slightly more work". It is N teams independently implementing the same join and diverging. One caches author records for an hour, another for a day, a third forgets tags exist. When a reference changes, each consumer discovers it at a different time, so two downstream systems show different versions of the same content and nobody can say which is correct.

There is a second effect that matters more at scale: the resolution work moves off the read path. A consumer that must resolve four references per message does four lookups per message forever, against services that then need capacity for that traffic. Doing it once at publish time converts N × M lookups into M writes.

Implementation patterns

• Keep the normalised topic as the write target and the denormalised topic as derived. Producers never write to the denormalised log directly, or it stops being derivable and becomes a second source of truth that can disagree with the first.
• Partition them differently. The normalised log may need strict global ordering and therefore few partitions; the denormalised log is keyed per top-level asset and can be widely partitioned, because ordering is only required per asset.
• Republish on dependency change, not only on asset change. When an image is updated, every asset referencing it produces a new denormalised message. This fan-out is the pattern's main cost and must be bounded: a dependency referenced by a million assets produces a million messages.
• Carry a version or source offset on every denormalised message, so a consumer can tell which normalised state it reflects and discard an older message that arrives out of order.
• Add a notification-only topic where many consumers just need to know something changed: caches to invalidate, SLO timers to stop. The New York Times calls this the skinny log and keeps it alongside both others.

Industry example

The New York Times' publishing pipeline, described in 2017, is the reference implementation. Content is published to the Monolog in normalised form as Protobuf messages, retained indefinitely so any store can be rebuilt by replay. A denormalised log republishes each top-level asset with its dependencies resolved across many partitions, and a skinny log carries processed-content notifications. Consumers choose the shape that fits them rather than all consuming the same one.

Failure scenarios

• Fan-out storm. A change to a widely referenced entity republishes every asset that references it. Without rate limiting, one edit to a shared tag saturates the topic and every downstream consumer for hours.
• Silent divergence. A bug in the resolution step produces denormalised messages that no longer match the normalised truth. Because consumers read only the derived topic, nothing detects it until a customer reports a stale byline.
• Replay asymmetry. A consumer rebuilding from the denormalised log gets the dependency values as they were at publish time, which is usually what you want, and is not what a consumer rebuilding from the normalised log gets. Two rebuild paths, two answers.
• Unbounded message growth. Resolving deeply nested references produces messages of megabytes, which hits broker message-size limits at the worst moment.

Trade-offs

When not to use it

With one or two consumers, this is overhead. The join belongs in the consumer, where it can be shaped to exactly what that consumer needs, and the second topic is storage and a pipeline stage bought for nothing. It is also the wrong pattern when dependencies change far more often than assets: republishing thousands of assets per dependency edit costs more than the lookups it saves. The rule: denormalise when reads of the references vastly outnumber writes to them, and when the consumer count is high enough that the duplicated join is a real organisational cost.

Interview question

Q: Your denormalised topic republishes an asset whenever any of its dependencies change. A content tag used by 4 million articles is renamed. Walk me through what happens and what you would have built beforehand.

What a strong answer covers: 4 million republished messages against normal daily volume, probably orders of magnitude above it · the queue this creates for every consumer, and that lag is now measured in hours · rate-limiting the republication and treating it as a backfill with its own topic or priority, not as ordinary traffic · questioning whether the tag needs to be denormalised at all, or whether it is the one field consumers should resolve themselves · the detection signal, which is publish-rate anomaly, not consumer lag.

Quick check

Quiz: Why must producers never write directly to the denormalised log? Because it would stop being derivable from the source of truth, and the two logs could then disagree with no way to say which is correct.

Flashcard: What does a denormalised log cost that a normalised one does not? — Write amplification on dependency change: one edit to a shared entity republishes every asset that references it.