Telemetry Pipeline Lag

Every incident decision is made against a picture of the system that is already old. Under normal conditions the gap between a request completing and its data appearing in a query is seconds, and nobody thinks about it. During a large incident the same pipeline is carrying a surge of telemetry generated by the incident itself, and the gap grows exactly when decisions are being made fastest.

A four-minute lag does not merely delay the dashboard. It means a rollback completed at 14:02 is invisible until 14:06, so the engineer who started it concludes at 14:04 that it did not work and starts a second mitigation. Two changes, three minutes apart, and a postmortem that can never say which one recovered the service.

Pipeline lag is a measurable quantity, and the useful form of it is age of the newest queryable data point, not queue depth. Queue depth is a cause; age is the thing every decision depends on.

Why it matters

Lag sets the minimum period of any control loop that reads telemetry. Autoscalers, load shedders, automated rollbacks and burn-rate alerts are all feedback loops, and a loop whose feedback is delayed beyond its actuation interval oscillates. The autoscaler that keeps adding capacity because the effect of the last addition has not arrived yet is not badly tuned; it is correctly tuned for a lag it does not know about.

It also breaks alerting arithmetic. A five-minute burn-rate window evaluated on data that is four minutes late computes its ratio from a window that is only one-fifth populated. The numerator is small, the burn rate looks low, and a live incident can silently resolve its own page.

Implementation patterns

• Emit and display it. Compute now - max(timestamp) per pipeline and put it on the incident dashboard beside the golden signals. It tells the reader how much to trust everything else on the screen.
• Make "stale" a distinct alert state. A rule whose evaluation window is shorter than current lag should go to unknown, not to OK. Treating missing data as healthy is the specific mistake that hides outages.
• Keep one signal off the main pipeline. Edge error rate read directly from the load balancer, or a synthetic probe, arriving within seconds. A rollback verdict must never depend on the pipeline that is currently degraded.
• Shed rather than buffer. Buffering trades lag for completeness. During an incident, completeness of debug telemetry is worth less than freshness of the golden signals, so drop low-priority classes first — and configure that in advance, because nobody chooses it under pressure.
• Give alert-driving telemetry its own capacity, so it cannot queue behind a flood of debug output produced by the same incident.

Industry example

The pattern is visible in every large incident where the observability stack shares infrastructure with the workload, and it is the mechanism behind the well-documented cases where diagnosis, not repair, dominated time-to-restore — Roblox's 2021 outage being the most-studied example, where degraded shared infrastructure progressively removed the engineers' ability to see what was degrading. Time-to-restore is dominated by time-to-diagnose, and lag is a direct tax on diagnosis.

Failure scenarios

• Confounded mitigations, as above: two changes attributed to one recovery, and a superstition the team carries for years.
• A page that auto-resolves because a short burn-rate window was evaluated on a partially filled bucket.
• An autoscaler oscillating between over- and under-provisioning for the duration of an incident.
• A canary analysis passing because the canary's bad data had not arrived when the gate evaluated.
• A dashboard reading zero that is indistinguishable from a dashboard reading zero because nothing has arrived yet.

Trade-offs

Low lag is also bought with cost: smaller batches, more frequent flushes and more ingestion headroom all raise the bill, and the headroom needed is set by your worst incident rather than your peak traffic.

When not to use it

Not every pipeline needs to be fast. Billing extracts, long-horizon capacity analytics and compliance archives are legitimately hours behind, and optimising their lag is spending money on a number nobody reads. The discipline applies only to telemetry that drives a decision under time pressure. Publishing lag for a batch pipeline as though it were an operational signal is how dashboards fill up with numbers that never mean anything.

Interview question

Q: Your metrics pipeline normally makes data queryable in 20 seconds. During an incident it drifts to four minutes without losing anything. What breaks, and what would you change first?

What a strong answer covers: that nothing is lost and everything is still wrong; the confounded-mitigation sequence minute by minute; the effect on windowed burn-rate alerts and on any automated remediation; the difference between alerting on queue depth and alerting on age of newest data; and the design answer — publish lag, make staleness a state, keep one low-latency signal off the pipeline, and shed by priority rather than buffer.

Quick check

Quiz: Why is "age of the newest queryable data point" a better signal than "ingestion queue depth"? Because queue depth is one possible cause among several, while age is the quantity every decision made from the dashboard actually depends on.

Flashcard: Your dashboard lags four minutes during an incident. Name the two failures. Mitigations get confounded because effects arrive after the next decision, and windowed alerts evaluate on partially filled buckets, so a live incident can resolve its own page.