The first ten minutes of each of the last two incidents went on arguing about whether the graphs could be trusted. Design the single dashboard that answers "is the telemetry lying to us", with the signals and thresholds that belong on it, and say what you deliberately keep off.

The requirement that drives the structure

Telemetry in production fails in three ways, and a responder needs to tell them apart in under a minute: data is late, data is missing, or data is wrong. "The graphs look odd" is not one of those, so every panel on this page names a class and carries a threshold. One screen, no scrolling, no panel that needs a mental model of the pipeline to read.

The design

1. Ingest-to-queryable lag, per signal. Computed from event timestamp minus the time it became queryable, not from the pipeline's own progress counters. Threshold: 2x the scrape interval, so 60 s on a 30 s scrape. It is first because every number below it is stale by this amount, and decisions made on 4-minute-old data during a rollback are the expensive kind.
2. Collection health. Count of targets with up == 0, plus scrape_samples_scraped per job against yesterday. The failure mode worth printing on the panel: a target that exceeds its configured sample_limit fails the entire scrape, so one team adding an unbounded label removes every metric for that instance, not just the new one.
3. Series churn. scrape_series_added and active series by job, which catches a cardinality event while it is still a graph rather than an invoice.
4. Pipeline stages in order. For the collector fleet: otelcol_receiver_refused_spans, then otelcol_processor_incoming_items against otelcol_processor_outgoing_items, then otelcol_exporter_queue_size against otelcol_exporter_queue_capacity, then otelcol_exporter_send_failed_spans and otelcol_exporter_enqueue_failed_spans. Left to right is upstream to downstream, so the first non-zero counter is where data is being lost, which is the whole diagnostic value of the row.
5. Agent drop counters for logs, so a dropped-record burst is a number rather than an inference from missing lines.
6. A heartbeat series per cluster, emitted by a job that does nothing else. Without it, an empty panel and a genuine zero are indistinguishable, and that ambiguity costs more incident minutes than any other single gap.

The hard part

Panels 1 and 6 are the ones teams skip, and they are the ones that answer the question in the stem. Lag has to be measured end to end, which means a synthetic record with a known emit time, because every internal counter in the pipeline reports progress from the pipeline's own point of view. A stage that has stopped reading reports no backlog at all.

What stays off, and why

Collector CPU and memory, cost per team, per-service volume tables, retention settings. They are real and they belong in a weekly review. The rule: if a panel cannot change a decision in the next five minutes, it is not on this page. Resource graphs for the collector in particular invite the response of scaling the collector, which is rarely the live problem.

How it earns its place

Link it from the first line of every runbook and make the first question of every incident call "is telemetry healthy", with this page as the answer. Measure the dashboard by whether the argument in the stem happens again.

When not to build this

With fewer than about 20 services on a managed vendor, prefer three panels: a lag panel, the vendor's status page and an agent-up count. Building self-health machinery for a pipeline you do not operate adds a system you do operate, which is the wrong direction for a small team and costs a maintainer you do not have. Choose to build the full page only once you own the collectors and the metrics tier yourself, which in practice means a few hundred services and 2 or more teams depending on it.