Your metrics pipeline normally makes data queryable about 20 seconds after it is emitted. During a large incident the ingestion tier falls behind and the lag grows to four minutes. Nothing is lost. Walk through what happens to the on-call engineer's decisions.

Minute by minute

t+0. The engineer starts a rollback. The dashboard is showing data emitted four minutes ago, which is entirely from the bad version.

t+2. The rollback has completed and the good version is serving. The dashboard still shows the bad version's error rate, because the good version's first data points have not arrived. The screen and reality now disagree, and nothing on the screen says so.

t+3. The engineer concludes the rollback did not help and starts a second mitigation — failing a region out, scaling the fleet, disabling a feature flag.

t+4. The rollback's effect finally appears. Errors fall. The recovery is now attributable to two changes made three minutes apart, and the postmortem will never resolve which one worked, so the team keeps both and carries a permanent superstition.

Where it amplifies

Burn-rate alerts are computed over windows. A five-minute short window evaluated on data that is four minutes late either fires four minutes late or, worse, evaluates over a window that is only partly populated. A partially filled numerator produces an artificially low burn rate — an incident can auto-resolve its own page while it is still happening.

Auto-remediation reads the same lagging data. Anything that scales, sheds or fails over on a metric threshold is now acting on stale state, and the classic result is over-correction: it keeps scaling because the effect of the last scale has not arrived yet. This is a control loop whose feedback delay exceeded its actuation interval, and it oscillates for the same reason any such loop does.

What the user sees

Nothing directly. The user sees an outage that lasts longer than it needed to, because every decision in the response loop was made against a four-minute-old world.

What stops it

1. Publish pipeline lag as a metric and put it on the incident dashboard, next to the golden signals. It is the one number that tells the engineer how much to trust everything else on the screen.
2. Make alert rules refuse to evaluate on stale data. "No data" and "healthy" must be distinct states, and a rule whose window is shorter than current lag should go to an unknown state rather than to OK.
3. Keep one low-latency signal off the main pipeline. Edge error rate straight from the load balancer, or a synthetic probe, arriving in seconds. A rollback verdict should never depend on the pipeline that is currently degraded.
4. Shed rather than buffer. An ingestion tier that buffers trades lag for completeness, and during an incident lag is the thing you cannot afford. Dropping low-priority telemetry — debug traces, verbose logs — to keep the golden signals current is the correct priority order, and it has to be configured in advance because nobody chooses it under pressure.

What would have to be true for it to self-heal

The pipeline would need priority classes with independent capacity, so that the metrics driving alerts cannot be queued behind a flood of debug telemetry produced by the same incident. Without that separation, the incident generates the telemetry surge that delays the signals used to fix the incident, and the loop closes on itself.

Common weak answers

• "Over-provision ingestion so it never lags." Ingestion surges during incidents are correlated with the incident, so the headroom you would need is set by your worst event, not your peak traffic. Priority classes are cheaper than buying for the tail.
• "The engineer should have waited four minutes." Correct in hindsight and useless in practice: nobody knows the lag unless it is on the screen, and waiting is exactly the wrong instinct when an outage is live. The fix is to display the lag, not to ask for patience.
• "Alert on ingestion queue depth." Queue depth is a cause, not the effect that matters. Alert and display on age of the newest queryable data point, which is what every decision actually depends on.