A CDC pipeline feeding your warehouse falls three hours behind during a source system's batch job, and the source's transaction log retention is 24 hours. What is the risk and what do you change?

The immediate risk

Lag consumes the retention window. At three hours behind against a 24-hour retention, you have 21 hours of margin. If the consumer stops entirely — a deployment failure, a schema change it cannot handle, a full disk — the clock runs, and once lag exceeds retention the log positions you needed are gone.

Recovery then means a full re-snapshot of the source: heavy load on an operational database, a long outage of freshness downstream, and in a large table a multi-day exercise.

So the number to watch is not lag in isolation but lag as a fraction of retention, with the derivative. Constant lag at three hours is stable; lag growing by an hour each day reaches the cliff on a predictable date and is the alert that matters.

Why it happened

Almost certainly the source's batch job generated far more change volume than the steady state, and the consumer is throughput-bound. Confirm rather than assume: per-partition lag will show whether it is total throughput or one hot partition caused by a key that the batch touches disproportionately.

What to change

Raise retention on the source log to cover the worst plausible consumer outage plus a margin. This is usually the cheapest single mitigation and it is frequently left at a default nobody chose.

Add headroom for the batch window — more consumer parallelism, or a partition count that permits it. Note that repartitioning a live topic with committed offsets is itself a migration, so this is a decision better made early.

Alert on lag as a fraction of retention, with the rate of change, not on an absolute message count.

Test the recovery path. If the answer to "what happens if we exceed retention" is a re-snapshot, that procedure should have been rehearsed and timed, so the incident response is a known-duration operation rather than an investigation.

The design point underneath

Retention is not a storage setting; it is the size of the recovery window. Every consumer's maximum tolerable outage is bounded by it, and that relationship should be written down where someone changing the retention will see it.