Integration Log of Record

An estate with one producing system and twelve consumers of its data has twelve integrations, twelve opinions about the producer's API, and twelve back-fill conversations whenever a consumer launches or changes its data model. The producer can no longer change, because twelve teams have to agree first.

The log of record moves the contract. The producer appends facts to an ordered, retained log; every consumer reads from it and owns its own projection. The producer no longer knows who its consumers are.

Why it matters

Replay is the property that changes what an organisation can do, and a queue cannot provide it. A queue deletes what it delivers, so a new consumer needs an export, a migration script and time from the producing team. A log retains, so the thirteenth consumer starts at offset zero, rebuilds its whole view unassisted, and costs the producing team nothing.

The second property is ordering. When order is a property of the log rather than an agreement between consumers, a whole class of reconciliation question disappears - "did these two systems apply the update and the correction in the same sequence?" is answered by construction rather than per integration.

The third is temporal decoupling. A consumer that is down becomes a consumer with lag, which is a metric with a threshold, rather than a failed integration that needs a person.

Implementation patterns

• Append facts, not commands. "Asset 41 published at version 7", not "update the search index". A command binds the producer to a consumer; a fact does not.
• One ordered topic where global order is a requirement. A single partition gives the strongest ordering guarantee and caps throughput at what one partition sustains - an acceptable trade for publishing, editorial and reference data, and not for telemetry.
• Additive, versioned schemas, forever. A consumer replaying from zero in 2026 must parse a 2017 message. Never remove or repurpose a field; add and deprecate.
• Normalise the payload before it enters the log. The log's schema becomes the estate's interchange contract, so a producer-shaped payload becomes everyone's problem.
• Publish the compacted view alongside the full history, so consumers that need only current state do not pay for the whole archive.
• Make lag the primary health metric per consumer group, with thresholds and alerts, because it is the only signal that a projection is stale. A practical starting threshold is 60 seconds for interactive projections and 15 minutes for analytical ones.

Industry example

The New York Times described this design in 2017: every published asset written as an event to a single ordered Kafka topic, the Monolog, holding the full history of published content and acting as the source of truth. Back-end systems - the website, apps, search indexing, personalisation, archives - consume the log and build their own views, rather than calling a publishing API.

The reason it works there is specific and worth carrying: the corpus is enormous in history and modest in write rate, so one partition can keep up, and published content is immutable once emitted. Both facts are preconditions, not details.

Failure scenarios

• Schema break on replay. A field removed in 2024 makes every historical message unreadable, and the failure surfaces only when someone replays - often years later, under pressure.
• Ordering lost to sharding. Throughput grows, the topic is partitioned, and the global order that justified the design quietly disappears while everything still appears to work.
• Consumer drift. Twelve consumers each interpret the same event slightly differently, so the estate has twelve versions of the truth derived from one log. The log guarantees delivery and order, not agreement.
• Erasure obligations collide with immutability. A subject-access deletion request against an append-only log of personal data has no clean answer without crypto-shredding designed in from the start.
• Retention cut for cost, which silently removes replay - the capability the whole design was for.
• The log becomes a database. Consumers query it directly for point lookups it was never built to serve.

Trade-offs

When not to use it

With two or three consumers. The coupling problem is not real yet, and a broker plus its operational surface is a large fixed cost against a problem you can solve with an API call.

Also when the data is high-volume and mutable - a single ordered partition will not hold, and a partitioned log loses the ordering guarantee - and when the data is transactional state rather than facts. An order, a balance or an inventory count is not append-only by nature, and modelling it as a log means rebuilding a database badly. The pattern fits facts that are immutable once emitted and meaningful forever.

Interview question

Q: Your company has one system of record for customer data and nine downstream consumers, each integrating through a nightly extract. A tenth is proposed. Argue for and against moving to a log of record, and tell me what you would need to know about the data before committing.

What a strong answer covers: that the pain is back-fill and coupling rather than traffic, so the justification is replay and producer independence · the questions that decide it - write rate, whether global ordering is genuinely required, mutability, and whether personal data brings erasure obligations · the permanent cost of additive schema discipline and of a broker's operations · that the migration is incremental (dual-run the log beside the extracts, move consumers one at a time, retire extracts last) · and the honest counter-argument that nine nightly extracts which nobody complains about may not be worth a platform.

Quick check

Quiz: Why does a queue not substitute for a log in this pattern? Because a queue deletes what it delivers, so a new consumer cannot rebuild its view by replaying - it needs a bespoke back-fill from the producing team, which is the coupling the pattern exists to remove.

Flashcard: What must be true about the data before a single ordered topic is viable? — Write rate low enough for one partition to keep up, and records that are immutable facts rather than mutable transactional state.