A product search index is fed from a Postgres change-data-capture stream. Price and title edits appear within two seconds. For about 3% of products the indexed description is a placeholder string nobody recognises, and every affected product has a description over roughly 2 KB. Replication lag is zero and the connector reports no errors. Where is the data going?
Show the full answer Hide the answer
The first three things to look at
- Compare the row in Postgres with the change event in the topic, not with the index. One look localises the loss to producer or consumer. Here the event itself has no description value, which rules out the index writer as the origin.
- Check the size boundary. Every affected row is over about 2 KB, which is the threshold at which Postgres moves or compresses attributes out of line into the companion TOAST table (roughly a quarter of the 8 KB page). A defect that correlates with value size is a storage mechanism, not business logic.
- Check the table's
REPLICA IDENTITY. Under the default, the before-image carries only the primary key, so the consumer cannot diff old against new to notice anything is absent.
The diagnosis
Logical decoding emits the new tuple as the write-ahead log recorded it. For an UPDATE that did
not change an out-of-line attribute, the WAL does not repeat that value - it records a marker
meaning "unchanged TOAST datum". Debezium turns that into a literal placeholder,
__debezium_unavailable_value by default and configurable through toasted.value.placeholder;
this behaviour is described in Debezium's 2019 post on handling unchanged Postgres TOAST values.
So every price-only update to a product with a long description produces an event whose description field is a sentinel. The consumer replaces the whole document with the event's fields and writes the sentinel into the index. The 3% are the large-description products that have had any subsequent edit to some other column.
The misleading signal
Replication lag of zero. Lag measures the WAL position a consumer has reached; it says nothing about payload completeness. A placeholder is a successfully delivered event, so freshness monitoring, connector error counts and topic throughput all look perfect. This class of defect is invisible to every signal except a comparison of content.
The fix in order
- Make the consumer placeholder-aware. Never write a field whose value equals the configured sentinel; issue a partial update that omits it. This requires a sink that supports partial updates - a full-document replace cannot be made safe this way, which is the real design error.
- Or re-read the value at the source. Debezium's reselect post-processor queries the primary for columns that arrived unavailable. The cost is a point query per affected event on the production database, which must be budgeted - practitioners report it becoming significant load when a large share of events need it.
- Or
ALTER TABLE ... REPLICA IDENTITY FULL. It makes the before-image complete and it is expensive: every update logs the whole old row, so WAL volume rises for every consumer and for the replication slot. It is also not a guaranteed cure - Debezium issue DBZ-7193, filed against 2.3.4, reports unchanged TOASTed array columns still arriving as placeholders under FULL. - The alert that would have caught it on day one: a sink-side counter of fields equal to the configured placeholder, plus a nightly sampled row-by-row diff between source and index.
When this is the wrong answer
If the affected column is small, this mechanism is not your problem - look at the consumer's merge
logic or an event ordering issue instead. And do not reach for REPLICA IDENTITY FULL on a
high-write table to repair one column: the WAL amplification hits the whole pipeline and the
replication slot, and a placeholder-aware consumer costs nothing.