Design the one dashboard your team opens first during an incident. What is on it and what is deliberately not?
Show the full answer Hide the answer
What is being tested
Editorial judgement under stress. The failure mode of dashboards is showing everything, and the skill is deciding what to leave out.
What is on it
One screen. No scrolling. If someone must scroll during an incident, it has failed.
Top left — the business outcome. Orders per minute, or the equivalent, against the same time last week. This answers the only question that matters first: is the product working?
Top right — the SLO and its burn rate. Are we breaching, and how fast is the budget going?
Middle — RED for the critical path. Request rate, error rate and latency percentiles for the two or three most important endpoints. Not every endpoint.
Below — dependency health. One row per critical dependency: error rate and latency. This is where the cause usually is, and the row that is red narrows the search immediately.
Bottom — saturation. Thread pools, connection pools, queue depth (as age), database connections. The resources that get exhausted.
Across every panel — deployment and configuration change annotations on the time axis. The single most valuable feature of any incident dashboard, because "what changed" is usually answered visually in the first ten seconds.
What is deliberately not on it
- Per-instance detail. Aggregate first; drill down elsewhere.
- Every endpoint. The long tail belongs on a service-specific dashboard.
- Infrastructure metrics — CPU, memory per host. These are causes, and you look at them after a symptom points you there.
- Anything requiring expertise to interpret. If only its author can read a panel, it is a single point of failure with a name.
- Anything without a comparison baseline. A number with no "same time last week" is uninterpretable — is 4,000 requests per second high?
The design rules
- Most important top left, because people read in that order, especially when panicking.
- Percentiles, never averages, on every latency panel.
- Link to the next step — the trace query, the log query, the runbook — so the dashboard is the start of an investigation rather than the end.
The other two dashboards that should exist separately
Failure-specific incident dashboards, one per known failure mode, linked from the runbook of the alert that fires for it — "database saturation" showing connections, slow queries, replication lag and affected endpoints.
Deep service dashboards for investigation after the triage screen has pointed somewhere.
Mixing all three produces forty panels nobody reads under pressure.
What a strong answer adds
That unused dashboards should be deleted. A directory of 400 dashboards means nobody can find the right one during an incident, so everyone builds another — and the proliferation is itself the problem.