advanced 2 min answer

Users report seeing stale data intermittently. Replication lag is normally under a second but spikes to minutes twice a day. How do you handle it?

replicationlagroutingconsistency
Show the full answer Hide the answer

First: find the cause of the spikes

Twice a day is a schedule, so look for one. The usual candidates, each with a different fix:

A batch job or bulk write on the primary generating log volume faster than the replica can apply it — often single-threaded apply on the replica being the bottleneck even when the primary is parallel.

A long-running query on the replica blocking apply, because the replica must not remove rows the query is reading. PostgreSQL exposes this directly through conflict handling and hot_standby_feedback.

A schema change or index build generating enormous log volume.

Backup or snapshot activity competing for I/O on the replica.

Under-provisioning — replicas are frequently smaller than primaries, which is fine until apply becomes the constraint.

Second: stop serving stale reads regardless of the cause

This is the part that must be done whether or not the cause is fixed today.

Make lag visible to the routing layer. A replica exceeding a threshold is removed from the read pool automatically. This is the single most valuable change: it converts a correctness problem into a capacity problem, which is far easier to reason about and degrades honestly.

Route reads that need freshness to the primary. Not all reads — the ones that gate a decision, and the ones in a session that has just written.

Provide read-your-writes for the writing session, via sticky routing for a window after a write, or by returning the written entity in the response and rendering from it.

Third: decide the policy explicitly

Which reads tolerate what staleness is a product decision, and it should be written down per read path rather than discovered during an incident. "Balance shown on the dashboard: up to 5 seconds stale, acceptable. Balance used to authorise a payment: primary, always."

What a strong answer adds

Noting that an SLO on replication lag is the right framing — it makes the tolerance explicit, gives a threshold for the routing layer, and turns "the replica was behind" into a measurable, alertable property rather than an after-the-fact explanation. And that lag should be measured in time, not just log position, because time is what maps to what the user experienced.