Observability
General material on understanding a system from its outputs.
3 to work through
-
intermediate
You are designing a new service. What must be in place before it goes live so that whoever is paged at 3 AM can diagnose it without you?
2 min answer -
advanced
Your observability bill is now 40% of your compute bill. Leadership wants it cut in half without going blind. What do you cut?
2 min answer -
advanced
p99 latency on checkout tripled overnight. Dashboards look normal, no deployment went out, and every service reports healthy. How do you find it?
3 min answer
13 terms in this topic
Alert Fatigue
The desensitisation that follows from alerts that are frequent, non-actionable, or not tied to user impact — after which real alerts are missed too.
toolApplication Performance Monitoring
Instrumentation inside the application that attributes latency and errors to specific code paths, queries and dependencies.
conceptCardinality
The number of distinct time series produced by a metric, which is the product of the distinct values of all its labels — and the main driver of monit…
practiceCorrelation ID
A single identifier attached to one logical operation and included in every log line it produces, anywhere in the system.
toolDistributed Tracing
Following one logical request across every service it touches by propagating a shared trace identifier and recording timed spans.
metricGolden Signals
The four measurements that cover most of what matters for a request-driven service: latency, traffic, errors and saturation.
practiceHealth Check
An endpoint the platform polls to decide whether an instance should be restarted or should receive traffic — two different questions needing two diff…
conceptObservability
The property of being able to answer new questions about a system's internal state from its external outputs, without shipping new code.
practiceRED Method
A minimal per-service dashboard: Rate, Errors, Duration — the request-centric view of whether users are being served.
practiceRunbook
A short, actionable document telling an on-call engineer what an alert means, what to check, and what the safe mitigations are.
practiceStructured Logging
Emitting log entries as machine-parseable key-value records rather than as formatted prose.
practiceTelemetry Sampling
Keeping a subset of traces or events to bound observability cost, chosen so the ones that matter survive.
practiceUSE Method
For every resource, track Utilisation, Saturation and Errors — the resource-centric complement to request-centric monitoring.
Neighbouring topics
Logging
What to log, at what level, and what must never appear in a log.
No content yetStructured Logging
Machine-parseable events with stable names and consistent fields.
No content yetMetrics
Counters, gauges and histograms, and percentiles rather than means.
No content yetCardinality
The label that multiplies series count and the bill with it.
No content yetDistributed Tracing
Reconstructing one request's path across every service it touched.
No content yetCorrelation IDs
One identifier propagated through every hop and every log line.
No content yetSampling
Head-based versus tail-based, and keeping the traces that matter.
No content yetHealth Checks
Liveness versus readiness, and the check that causes the outage.
No content yetAlerting
Symptom-based, actionable, user-impacting — and linked to a runbook.
No content yetAlert Fatigue
How noise makes the real page invisible, and the structural fix.
No content yetDashboards
Answering 'is it us' in under a minute, for someone who was asleep.
No content yetApplication Performance Monitoring
Attributing latency to code paths, queries and dependencies.
No content yetProfiling
Continuous CPU and memory attribution in production.
No content yetBusiness Metrics
Orders per minute alongside error rate, because healthy is not enough.
No content yetSLO Monitoring
Burn-rate alerting that fires on user impact rather than on thresholds.
No content yetLog Management
Aggregation, retention tiering, search and the cost of keeping everything.
No content yetTelemetry Cost
Observability bills that rival compute, and where to cut without going blind.
No content yetDebugging Distributed Systems
Localising a regression when every service reports healthy.
No content yetOpenTelemetry
Instrumenting once against an open standard rather than a vendor agent.
No content yet