Retraction Stream

A stream operator that must answer before its input is complete has only one honest option: answer now and correct later. An unwindowed left outer join sees an order with no shipment and emits the order padded with nulls. Two minutes later the shipment arrives and the earlier answer is wrong, so the operator withdraws it and emits the joined row.

Three physical records therefore cross the wire for one logical fact. Whether that is correct or catastrophic depends entirely on what the destination does with the second one. A destination that can delete converges. A destination that only inserts accumulates every wrong answer the pipeline ever gave.

Why it matters

The arithmetic is quiet and exact. If 3% of orders arrive before their match, each contributes two extra rows and the table runs about 6% over the ledger. The gap tracks shipment latency rather than volume, so it moves with upstream behaviour nobody is watching, and it survives every test that uses a fixture where both sides are already present.

Nothing in the pipeline is broken. The join is correct, the topic is correct, and the loader is correct for an append-only stream that this is not. The defect lives in the contract between two components that each pass their own tests, which is why it is normally found by a finance reconciliation months later rather than by monitoring.

Implementation patterns

• Declare the changelog mode at every boundary. Stream engines classify output as append-only, upsert, or full changelog, and will refuse to plan a job whose sink cannot absorb its output. The failure here is the declaration being lost when the output passes through a topic, where the format carries the operation flag but the contract does not.
• Key the destination and upsert. The retraction becomes a delete of that key and the following insert replaces it. Smallest change, and it keeps the early provisional row visible to anyone who wants it.
• Or remove the retraction at the source. An interval outer join bounded by a window emits one append-only row per left record, null-padded only when the window expires. The output is append-only by construction; the price is latency equal to the window on every record, paid even by the 97% that matched immediately.
• Carry the operation flag as a first-class column when the changelog must cross a topic, and make downstream consumers fail loudly on an unknown value rather than ignoring it.
• Reconcile on row count per key, not on totals. Totals hide a doubling that is spread thinly across millions of keys.

Industry example

Flink's SQL runtime makes the distinction explicit: regular joins and updating aggregates produce a changelog, windowed and interval operators produce append-only output, and connectors declare which changelog modes they support, so an incompatible plan is rejected rather than shipped. Teams hit the problem in production when they bridge the two worlds by hand, writing a changelog into a Kafka topic in a debezium-style format and then loading it with a copier that was written for events. The copier does exactly what it was built to do, and the declaration that would have stopped it never crossed the topic.

Failure scenarios

• Silent inflation of any count or sum downstream, proportional to the share of records unmatched at arrival.
• Resurrected deletions: a team deduplicates the destination on the key to hide the inflation, and a genuinely cancelled record never disappears because its retraction was already discarded.
• Divergent replicas when one consumer honours the flag and another does not, so two systems built from one topic disagree and neither is obviously wrong.
• Backfill amplification: a replay re-emits every historical retraction, so a correction run doubles the damage it was meant to repair.

Trade-offs

When not to use it

When fewer than about one in a thousand records is unmatched at arrival, the provisional row is noise. Use an inner join and raise a separate alert for records still unmatched after a deadline. That removes the changelog machinery entirely and leaves one simple contract. The same applies when no human or system acts on the early answer: if the provisional row is never read, paying for the ability to correct it buys nothing.

Interview question

Q: Your streaming job's output is correct and the analytics table built from it is 6% high. The job emits no errors and the loader logs no failures. Walk me through how you would find this, and what you would change.

What a strong answer covers: that the pipeline is a changelog and the loader is treating it as events; the arithmetic linking the inflation to the unmatched-at-arrival share; the two structural fixes (upsert on key, or an interval join) with the latency each costs; declaring the changelog mode at the topic boundary so the next consumer fails at deploy rather than at year end; and resisting the deduplication shortcut because it breaks genuine deletes.

Quick check

Quiz: A left outer join emits three records for one order. What are they, and what does an insert-only loader produce? An insert of the null-padded row, a retraction of it, and an insert of the joined row - three table rows where one was intended.

Flashcard: What must a destination support before it can consume a changelog correctly? A primary key and a delete, so a retraction converges rather than accumulating.