Partition Immutability Promise

A finance team reconciles a data product's daily revenue against the billing ledger and signs off on 2026-03-12. On 2026-03-19 the producing team finds a defect, rewrites the week in place, and the signed-off day no longer matches anything. Nothing failed and no pipeline errored. The producer exercised a right it never told anyone it was holding.

The promise is one sentence in a data contract: a partition, once closed, is immutable, and corrections arrive as new rows carrying their own effective date. That sentence is what lets a consumer store a checkpoint, trust an exported snapshot, and reconcile once rather than continuously.

Why it matters

The question a serious consumer asks is not "is this data correct" but "can yesterday's answer change without me knowing". Without the promise the answer is always yes, so every consumer either reloads full history on every run or accepts silent divergence.

Both responses cost. A nightly full reload of a 4-billion-row table scans tens of GB for a change affecting a handful of rows, and accepting divergence means the reconciliation regulated consumers are obliged to perform cannot be performed at all. The promise also moves the cost of late data from every consumer to one producer, which is the right place for it, because the producer knows the late-arrival distribution and the consumers do not.

Implementation patterns

• Close on a stated lag, not on arrival. Day D closes at D+2 06:00, chosen from the measured late-arrival tail. Publish the lag; it is the number consumers plan around.
• Corrections as signed delta rows, each carrying the business date it corrects and the date it was issued, so a correction is additive rather than destructive, and flagged on a separate feed so consumers that do not care can ignore it.
• An as_of or issue-date column on every row, so a consumer can reproduce exactly what the product said on any past date. This is the column that makes a dispute resolvable.
• A named escape hatch. Some defects require a rewrite — a privacy erasure, a legally incorrect value. State the conditions, the notice period and the notification list in advance.

Industry example

Append-only correction is settled practice in financial and regulatory reporting, where a restated figure is published as a dated amendment rather than by editing the original filing. It reappeared in data platforms from about 2017 onward, as lakehouse table formats made in-place rewrites of historical partitions easy, and therefore easy to do by accident. Teams adopting event-log architectures over the same period arrived at it independently, because a log you can rewrite is not a log.

Failure scenarios

• Silent in-place rewrite. A backfill fixes a defect, and every downstream snapshot, export and signed reconciliation is now wrong with no error anywhere.
• The promise without the correction path. Immutability is declared, a defect appears, the team has no mechanism to publish the fix, so it rewrites anyway.
• Closing too early. Day D closes at D+1 while 0.4% of events arrive later, so the correction feed carries routine traffic and consumers learn to ignore it.
• Corrections without a business date, so a consumer aggregates by issue date and moves revenue between months.

Trade-offs

When not to use it

When no consumer reconciles, do not build it. A dashboard answering "roughly how did last week go" is better served by a table that is simply correct as of now, and the correction mechanism is pure cost: delta rows, an extra column on every query, a second feed to operate.

It is also the wrong promise for a dataset whose grain is an entity rather than a period. A current customer-profile table is supposed to change, and promising immutability there means a new row per attribute edit, which is a slowly-changing-dimension design and a different decision. A defensible middle for an internal product: promise immutability past 7 days, allow rewrites inside the window, publish the window.

Interview question

Q: Your data product's daily partitions are rewritten whenever a defect is found. Finance wants to reconcile monthly and sign off. What do you change, and what do you refuse to promise?

What a strong answer covers: closing partitions on a lag derived from the measured late-arrival tail; corrections as dated delta rows rather than edits; an as_of column so any past answer is reproducible; a written escape hatch for erasure; and the refusal — you do not promise the data will never need correction, only that corrections will be visible and additive.

Quick check

Quiz: A consumer exported your closed partition and signed off on it. You then fix a defect by rewriting that partition. What has the consumer lost? Their reconciliation and any ability to reproduce the signed figure. Publish a dated correction row and leave the original partition alone.

Flashcard: What single column makes a restatement dispute resolvable? — An issue or as_of date on every row, so you can reproduce exactly what the product said on any past day.