A team is told that a table and a stream are two views of the same thing. They have a Postgres table holding 12 million account balances and are asked to produce the stream of changes that produced it. Why does the equivalence only run one way in practice, and what has to exist before it runs the other way?
Show the full answer Hide the answer
What is being tested
Whether you can state the direction of the duality and name the artefact that the reverse direction depends on. The phrase "a table is a stream and a stream is a table" is true about information, not about storage, and the gap between those two claims is where a project loses a month.
The asymmetry
A stream of keyed changes can always be folded into a table: read the log from the beginning, apply each record to a map, and the result is the current state. Nothing is lost, because every intermediate value was written down.
A table cannot be unfolded, because a relational update overwrites in place. When the balance goes 100 → 40 → 100, the row ends at 100 and the two earlier values were never stored anywhere. The table holds the result of the history, not the history, so the stream is not recoverable from it. You can produce a stream of snapshots by reading the table repeatedly, which is a different and weaker thing: it misses every change that was reversed between reads, and it carries no ordering guarantee per key.
This is the same asymmetry a compacted Kafka topic has. Compaction keeps the latest value per key and discards the rest, so a compacted topic is a table in log clothing and a consumer replaying it from offset zero learns the current state, not how it got there.
What has to exist before the reverse direction works
- A write-ahead-log reader (CDC). Debezium-style capture turns the database's own redo record into a change stream. It works because the log is the one place where intermediate values survive, and it costs you a replication slot that the database must keep until the consumer confirms — a stalled consumer turns into a disk-full incident on the primary.
- An outbox table written in the same transaction as the row. This gives you business events rather than row mutations, and it costs an insert per change plus a publisher.
- A history or audit table, if the requirement is only "what did this look like last Tuesday". Cheapest of the three and the usual right answer for 12 million rows.
The decision rule: if anyone will ever ask for the sequence of changes, record the changes when they happen. Retrofitting history is not possible; you can only start.
When the simpler answer wins
If the question is "what changed since yesterday" and nothing needs per-change ordering,
a nightly snapshot diff over 12 million rows is minutes of work and answers it. Adding CDC,
a broker and a stream processor to answer that is a large operational surface for a WHERE
updated_at > ? query. Snapshot diffing only fails when a value flips and returns between
snapshots, or when the business needs the reason for each change — at which point you need real
events, and you need them emitted by the code that makes the change.
Common weak answers
- "Just query the table with a timestamp column." That gives you rows that changed, not the changes. Two updates in one window collapse into one, and a delete disappears entirely.
- "Turn on binary logging later when we need it." The log starts when you turn it on. It cannot reconstruct last quarter.
- "The duality means we can pick either representation freely." It means the information is equivalent when the changelog is retained. Retention is the whole cost.