A platform of 40 services has logs only, and incidents take hours to diagnose. Design the observability strategy and its rollout order.

Why logs alone fail at this size

Logs answer "what happened in this service". They cannot answer "where did this request spend its time across twelve services", and reconstructing that by searching each service's logs by timestamp is exactly the hours being lost.

Rollout order, by return on effort

1. Correlation IDs everywhere. The cheapest change with the largest immediate effect. Generate at the edge, propagate on every call and message, attach to every log line automatically via middleware, and return it in error responses so a support conversation becomes one query.

2. Structured logging with a consistent schema. Prose can only be grep'd; structured records can be queried and joined. Standardise the baseline fields and the names and types of common identifiers — adopt OpenTelemetry semantic conventions rather than negotiating a house standard. Enforce with a shared library, because documentation alone does not achieve consistency, and retrofitting it later is expensive.

3. RED metrics per service — rate, errors, duration — with histograms for latency, since averages hide the behaviour that generates complaints. Bounded labels only.

4. Distributed tracing, adopted at platform level. One service failing to propagate breaks traces into fragments, so the value is highly non-linear in coverage. Use auto-instrumentation for breadth first.

5. SLOs and burn-rate alerting, replacing threshold alerts with something proportionate to impact.

6. Continuous profiling, which closes the last gap: metrics say the service is busy, traces say the span was slow, neither says which function.

Standardise on OpenTelemetry

Instrument once against a vendor-neutral API and change backends by changing collector configuration. The collector is where the operational value concentrates: batching, retry, tail-based sampling, redaction, cardinality limits, fan-out — all changeable without redeploying 40 services.

Know the maturity: tracing and metrics are stable; logs are least mature, so keeping the existing logging pipeline initially is reasonable.

Control the cost before it controls you

Observability commonly reaches 10–30% of infrastructure spend and grows silently.

Tail-based sampling — keep all errors, all slow traces, a small baseline of successes. Head-based sampling discards 99% of exactly the traces anyone would look at.

Cardinality allow-lists at the collector with an alert on series growth. One commit adding user_id as a label turns 150 series into 150 million, and that is the most common way a monitoring system is taken down.

Tiered log retention and aggressive sampling of successful-request logs.

Per-team cost visibility, or nobody manages it.

What "good" looks like when finished

An engineer can go metric → trace → log → profile along one correlated request without leaving the context. That chain, not the number of tools, is what turns hours into minutes.

What a strong answer adds

Naming the two classes this still does not cover: business metrics, which catch failures where every technical signal is green and orders are down 30%; and client-side telemetry, which is the only place the user's actual experience — network, rendering, and requests that never arrived — is visible.