You are designing a new service. What must be in place before it goes live so that whoever is paged at 3 AM can diagnose it without you?

What the interviewer is testing

Whether observability is designed in or bolted on, and whether you think about the person on call rather than about the tooling.

The minimum viable set

Structured logs with a correlation ID. JSON, not free text, so they are queryable. A trace or correlation ID on every line, propagated across every hop. Log levels used consistently. No secrets, no PII, no full request bodies.

The four golden signals per endpoint. Latency at p50/p95/p99, request rate, error rate, saturation of the most constrained resource. Split latency between successes and failures.

Distributed tracing, if this service calls others. Without it, "the request was slow" cannot be localised to a hop.

A health endpoint that means something. Liveness should test the process only — if it tests dependencies, the platform will restart healthy instances during a downstream outage and turn a degradation into an outage. Readiness may test dependencies, because removing an instance from rotation is the correct response.

Dependency metrics. Latency and error rate per downstream call, plus connection pool utilisation. Pool exhaustion is one of the most common causes of an outage and it is invisible in application-level metrics.

What actually helps at 3 AM

The metrics are necessary but not sufficient. What the person on call needs:

A runbook per alert, linked from the alert itself, naming what the alert means, what to check first, what the safe mitigations are, and who to escalate to. An alert without a runbook is a puzzle.

A dashboard that answers "is it us" in under a minute — this service's golden signals next to its dependencies', so the first triage step does not require constructing a query.

Alerts that are worth waking for. Every page must be urgent, actionable and user-affecting. If it is not all three, it is a ticket. Alert fatigue means the real page is also ignored.

A deploy marker on every graph. The first question in any incident is "what changed", and the answer is a deployment far more often than anything else.

A tested rollback. The mitigation for most 3 AM incidents is "put the previous version back", and it should be one command that does not require judgement.

What a strong answer adds

Naming cardinality and cost as a design constraint — observability bills can rival compute bills, and sampling strategy is an architectural decision, not a vendor setting. Tail-based sampling keeps the slow and failed traces, which are the only ones anybody looks at.

Also: an SLO with burn-rate alerting, so pages are driven by user impact rather than by component thresholds.