Observability in Practice

Definition

The property of being able to determine a system's internal state from its external outputs — specifically, being able to answer questions about behaviour that nobody anticipated when the instrumentation was written.

That last clause is the distinction from monitoring. Monitoring answers known questions; observability supports novel ones.

Why it matters

Because in a distributed system, the failure you are debugging at 3 AM is usually one nobody predicted, and the ability to ask a new question of existing telemetry is what determines whether the incident lasts twenty minutes or six hours.

Implementation patterns

The three signals have distinct jobs and are frequently used for the wrong one:

Metrics — aggregate, low-cardinality, cheap to store and query. How many, how fast, what error rate. Their labels must have small closed value sets.

Traces — the path of one request across services, with timing per span. This is what turns "the request was slow" into "it spent 1.8 seconds waiting on this call".

Logs — high-cardinality detail about specific events. Which user, what parameters, what exactly happened.

Correlation identifiers propagated from the edge through every service are what make the three navigable as one dataset rather than three islands.

Exemplars — a trace id attached to a metric data point — close the last gap: clicking a latency spike opens an actual trace of a request in that bucket.

Failure scenarios

Cardinality explosion. A user id, request id, email address or full URL used as a metric label creates one time series per distinct value. Monitoring platforms degrade badly, queries slow, ingestion falls behind — and it happens during incidents, when unusual label values proliferate. The rule: metrics answer aggregate questions with bounded labels; traces and logs answer per-entity ones.

Alerting on causes rather than symptoms. CPU, disk and queue depth page someone at 3 AM about a healthy system, while a novel failure that breaks checkout for 30% of users trips nothing. Page on error rate, latency and journey success.

Sampling that discards the interesting traces. Head-based sampling decides before knowing the outcome, so the error you want is probably gone. Tail-based sampling decides after, keeping every error and every slow trace.

Cost growing past the value. Observability spend can approach the cost of the infrastructure being observed, and the reflexive response — turning things off — removes exactly the visibility needed next time.

Industry example

Google's four golden signals — latency, traffic, errors, saturation — remain the most useful starting point precisely because they are symptom-oriented and apply to any service. The related SRE practice of burn-rate alerting against an error budget replaces a large number of static thresholds with a single mechanism that scales urgency to consequence.

Trade-offs

More telemetry gives better answers and costs more in ingestion, storage and query. Retention tiering, cardinality discipline and tail-based sampling are the levers that preserve the answers while controlling the bill.

The framing that funds it properly: express the value as time to detect and time to restore, which makes the trade a decision about incident duration rather than an argument about a tooling invoice.

Interview question

A new service goes live next week. What observability must exist on day one?

Expect: the four signals per endpoint, structured logs with a propagated correlation id, tracing at the boundaries, health checks that exercise real dependencies, one dashboard, symptom-based alerts, and a runbook. Strong answers note this belongs in the service template so it is inherited rather than built per service, and apply the test — "when this fails at 3 AM, how will the team know what happened?"