A post-mortem finds the telemetry system depended on the same infrastructure that failed, leaving engineers blind. What must be isolated, and what is the minimum set of break-glass signals?

Why this happens to good teams

Running observability on the production platform is efficient, standard, and recommended. It reuses the service mesh, the identity system, the shared storage, the deployment tooling and the service registry, and it means the monitoring stack benefits from every platform improvement.

The coupling is invisible because it only matters during the failures that take out shared infrastructure, which are rare and are precisely the failures where the coupling is most damaging. The tooling is reliable in every situation except the ones where it is needed most.

What must be isolated

• Metrics storage and the query path, so historical data remains readable when the primary datastore is the problem.
• Alert routing, which must not traverse the failing message bus or depend on the failing identity provider. Being unable to page anyone converts an incident into a delayed incident.
• Log collection, at minimum with buffering at the source so telemetry survives a shipping outage and the post-mortem has the data even if the response did not.
• The status page, on entirely separate hosting in a different provider or region. A status page inside the blast radius produces silence during the incident, which converts a technical event into a trust event.
• Incident communications — the channel must not be the product that is down, which is a live concern for any company whose product is communication.
• Access to hosts: a documented path to reach machines that does not depend on the platform's own access system, tested rather than assumed.

The minimum break-glass set

Full duplication of the observability stack is expensive and usually unjustifiable. A deliberately minimal independent signal path covers the catastrophic case at a small fraction of the cost:

1. External black-box probes from outside the infrastructure — a handful of synthetic transactions against the critical user journeys, run from a third-party service, reporting to a third-party destination. This single item answers "is the product working?" when everything internal is unavailable, and it is cheap.
2. A separately-hosted status page with a manual update path.
3. An out-of-band alert channel — a different paging provider, or at minimum a different delivery path.
4. Direct host access with documented credentials stored outside the platform.
5. The ability to read logs on a machine directly, without the aggregation layer.
6. A short list of critical metrics written to an independent destination.

The test that makes it real

In a game day, disable the shared component and confirm the dashboards still render, the alerts still fire, and the status page can still be updated. This is the only way the hidden coupling is discovered before it matters, and it is almost never done — because the dependency is invisible until it is exercised.

Every incident-response plan silently assumes the dashboards work. Naming that assumption and testing it is the entire exercise.

The design principle

Observability must fail independently of the systems it observes. Where full independence is uneconomic, the fallback position is explicit: the rich stack shares the platform, and a minimal independent path does not — and the organisation knows precisely which signals survive a total platform failure, because it has checked.