intermediate 2 min answer

Review this monitoring setup for a consumer web service. An uptime check calls /health every 30 seconds from a monitoring host in the same cloud region as the service. It is green all quarter. Yet users on two mobile carriers in one country could not sign in for 3 hours last month, and nobody was paged. What would you remove, change and keep?

synthetic monitoringuptime checksvantage pointsloginblack-box monitoring
Show the full answer Hide the answer

What is actually required

The business promise is "a user can sign in from where they are". The existing check verifies something narrower: that one endpoint answers, to a client sitting next to it, over a path no user takes. The three hours of failure lived in the path the check skips: carrier DNS resolvers, a CDN edge, a TLS certificate chain a few clients reject, or a routing problem between one network and the origin. None of those exist from inside the region.

What I would change

  1. Probe from outside, on several vantage points. Run the check from at least three networks and two countries, including one consumer-ISP or mobile vantage. Page when 2 of 3 fail for 3 consecutive runs, so one flaky probe does not wake anyone and a regional failure does.
  2. Probe a journey, not a URL. Perform the sign-in with a dedicated test account, then fetch one authenticated page. The step that fails names the layer: DNS, TLS, login, session.
  3. Raise the frequency to 1 minute, the cheapest lever on time-to-detect. At a 30 s cadence with a 3-run rule, detection takes about 90 s; the number to watch is that product.

What I would remove

The idea that this check is the availability signal. Keep it as a liveness canary for the origin, and stop quoting its green history as uptime in reports.

What I would leave

The in-region check. It is the fastest way to distinguish "origin is down" from "path is down" when the external probes disagree. Keep it, in a different role.

The signal synthetic probes cannot give

Probes sample a few paths. They do not see the two carriers' users unless one probe sits on those networks. Pair them with a real-user signal: sign-in success rate by network, country and client version, from client beacons or server logs. When it drops against a baseline for the same hour last week, that is the page, and the probes then explain where.

When this is the wrong answer

Internal-only tools, with all users in the corporate network, are well served by an in-network check. And synthetic journeys that move money or send messages need sandboxes or a clearly tagged test tenant; otherwise they pollute revenue metrics and trigger real notifications.

Common weak answers

  • "Add more internal health checks." Depth inside the region does not add a vantage point.
  • "Buy a RUM tool and drop synthetics." Real users tell you only after they fail, and at 3 am there are few of them.

How I would argue it in review

Show the incident timeline: the failing path, the number of users affected, and the minutes the new design would have saved at a 1-minute cadence with a 2-of-3 rule.