Unchanged TOAST Datum

A search index fed by change data capture shows correct prices and correct titles. For about 3% of products the description field holds a placeholder string nobody recognises, and every affected product has a description over roughly 2000 bytes. Replication lag is zero. The connector has logged no errors.

Postgres stores an attribute out of line, in a companion TOAST table, once a row would exceed about a quarter of the 8 KB page. When an UPDATE does not change such an attribute, the write-ahead log does not repeat its value - it records a marker meaning the TOASTed datum is unchanged. Logical decoding emits what the WAL holds, so the change event has a hole where the large value would be, and Debezium fills that hole with a literal placeholder (__debezium_unavailable_value by default, settable through toasted.value.placeholder).

Nothing has failed. The event was produced, delivered and consumed successfully. Only a consumer that compares content with the source can see the defect, which is why this one survives in production for months.

Why it matters

It breaks the assumption every full-document sink makes: that a change event describes the row's current state. It does not. It describes the change, plus whatever else the WAL happened to carry.

The blast radius is biased towards the data people care about. Large text and JSON columns are descriptions, documents, settings blobs and rendered content - exactly the fields a search index or a cache serves. And the affected rows are the ones that were edited after the big value was last written, so the defect accumulates with ordinary activity rather than appearing at once.

It also defeats the usual monitoring. Freshness, lag, throughput and error counts are all healthy, because a placeholder is a successful delivery.

Implementation patterns

• Make the consumer placeholder-aware. Compare each field against the configured sentinel and omit it from the write. This requires a sink that supports partial updates; a full-document replace cannot be made safe, which is the real design choice.
• Re-read at the source. Debezium's reselect post-processor queries the primary for columns that arrived unavailable. Budget the load: it is one point query per affected event against the production database, and practitioners report it becoming significant when a large share of events need it.
• REPLICA IDENTITY FULL on the specific table. It makes the old row image complete. The cost is WAL volume: a 2000-byte row changed by 40 bytes can write roughly 50x the WAL it used to, which every consumer and the replication slot then carry.
• Move the large value out of the replicated table into a table whose updates are rare, so unchanged-TOAST events stop occurring on the hot path.
• Alert on it. A sink-side counter of fields equal to the configured placeholder, plus a nightly sampled row-level diff between source and sink. Nothing else detects it.

Industry example

Debezium documented the behaviour in a 2019 engineering blog post on handling unchanged Postgres TOAST values, which introduced the configurable placeholder. The limits of the obvious fix are documented too: issue DBZ-7193, filed against Debezium 2.3.4, reports unchanged TOASTed array columns still arriving as placeholders even under REPLICA IDENTITY FULL. A separate report describes the placeholder reaching a JDBC sink in upsert mode and failing at the database because the sentinel string was written into a typed column.

Failure scenarios

• Sentinel in the index. The placeholder is written as the value, so the field is searchable and visibly wrong to users.
• Silent blanking. The consumer treats the sentinel as empty and writes an empty description, which looks like a content problem and gets escalated to the wrong team.
• Permanent divergence. The source value never changes again, so no later event ever carries the correct value. The sink cannot heal without a reseed.
• The WAL cure that causes an outage. REPLICA IDENTITY FULL applied to a high-write table inflates WAL for every consumer and grows the replication slot, which is how a data-quality fix becomes a disk-space incident on the primary.

Trade-offs

Placeholder-aware partial updates are the cheapest correct answer and they constrain the sink. Reselect is the most accurate and puts read load on the primary that scales with the defect rate. REPLICA IDENTITY FULL is the broadest and the most expensive, and is not a complete cure for every column type. Choose per table, never per cluster.

When not to use it

If the affected column is small, this mechanism is not the cause - the row never went to TOAST, so look at the consumer's merge logic or event ordering instead. And if the table's consumers only need changed columns, as an audit trail does, the placeholder is harmless information and the right move is to leave everything alone and document that whole-row reconstruction is not available from this stream.

Interview question

Q: A CDC-fed cache serves stale content for a small subset of rows and the pipeline reports perfect health. Walk me through your diagnosis and then tell me which fix you would ship first and why.

What a strong answer covers: comparing the source row with the raw change event rather than the sink, to split producer from consumer in one step; noticing that the defect correlates with value size and naming TOAST and the unchanged-datum marker; explaining why lag cannot see it; shipping the placeholder-aware partial update first because it is contained to one consumer; treating REPLICA IDENTITY FULL as a per-table decision with a WAL bill; and adding a placeholder counter plus a sampled diff so the next instance is caught in a day.

Quick check

Quiz: Why does a price-only UPDATE produce a change event with no description? The description is stored out of line in TOAST and was not changed, so the WAL records an unchanged-datum marker rather than the value, and the connector emits a placeholder.

Flashcard: Which fix is cheapest and which is most expensive? - Cheapest: a placeholder-aware consumer doing partial updates. Most expensive: REPLICA IDENTITY FULL, which rewrites the whole old row into the WAL on every update.