Synthetic Journey Probe
also called Black-Box Journey Check, Synthetic Transaction, Outside-In Probe
A scripted user journey, such as sign-in then add to basket, run on a schedule from several networks outside the system, so an outage on the path real users take is detected even when every internal health check is green.
An uptime check calls /health from a monitoring host in the same cloud region as the service, every 30 seconds, and has been green for a year. Then users on two mobile carriers cannot sign in for three hours, because a DNS resolver on those networks cached a bad answer, and nothing pages. The check verified that the origin answers a neighbour; it never verified that a user can do the thing the business exists to offer. A synthetic journey probe plays the user from outside and fails when the journey fails.
The practice has three parts: a journey (several steps with assertions, not one URL), a set of vantage points (distinct networks and countries, not distinct servers in one cloud), and a quorum rule that decides when a failing probe becomes a page.
Why it matters
Internal signals describe the system from inside. Many real failures sit on the path between: DNS, a CDN edge, a certificate chain some clients reject, a routing problem between one provider and the origin, a third-party script that blocks the login button. Real-user metrics catch these only after users fail, and at 3 am there are few users, so a failure can run for an hour on a rate that looks like noise. A probe that runs every minute produces a signal at any hour, at a known cost, with a step name that localises the layer.
Detection time is a product: the probe interval multiplied by the number of consecutive failures required. One-minute probes with a three-failure rule detect a full outage in about 3 minutes; 5-minute probes with the same rule take 15.
Implementation patterns
- Journey, not URL. Sign in with a dedicated test account, load an authenticated page, and for commerce add an item to the basket and stop short of payment. Assert on content (an element or a JSON field), not only on a 200.
- At least three vantage points, including one consumer-network or mobile location, across two or more countries. Page when 2 of 3 fail for 3 consecutive runs, so one flaky probe host cannot wake anyone and a regional failure can.
- Step-level timing and reason codes, so the alert says "TLS handshake failed from vantage B" rather than "journey failed".
- Tag the traffic. Send a header the edge converts into a field on logs and a label on metrics, and exclude the label from business dashboards by default. Make the test tenant's side effects inert: no shipment, no email, a payment sandbox.
- Keep the in-region check as a second signal. When outside probes fail and the inside one passes, the fault is on the path, not the origin.
- Complement with a real-user rate (sign-in success by network and country compared with the same hour last week) so the probe is the early warning and the real-user rate sizes the impact.
Industry example
A consumer streaming platform in the mould of Hotstar or JioCinema, expecting tens of millions of concurrent viewers for a live match, cannot rely on real viewers to reveal a carrier-specific playback failure in the first minutes of an event. An archetypal design runs synthetic players on a handful of carrier networks that start a stream every 30 seconds and report the stage reached (manifest, first segment, first frame). This is described as an archetype, not as any company's documented system.
Failure scenarios
- Single vantage point: the probe host's own network blips and pages the team, which learns to ignore it, or the host sits in the same region as the origin and misses path failures.
- Shared test account: the account is locked after repeated logins or its password expires, producing a permanent false alarm. Monitor the account's own health.
- Polluted metrics: unlabelled probe orders inflate conversion for months before anyone notices.
- Drift: the journey script breaks when the page changes, and the alert is silenced instead of fixed. Treat a failing probe as a defect in either the system or the script, and own it.
- Bot protection: the probe is blocked by the same defences that guard users. Allow-list by a secret header, not by IP alone.
Trade-offs
| Choose | Gains | Pays |
|---|---|---|
| Shorter interval | Faster detection | More probe cost and more test-data churn |
| More vantage points | Fewer false pages and broader coverage | Vendor or hosting cost, more scripts to maintain |
| Longer journeys | Covers more of the product | Slower, flakier, harder to localise |
| Real purchase | Exercises payment and fulfilment | Fraud-rule noise, cost, side effects |
When not to use it
Internal tools used only from the corporate network are well served by an in-network check. A product with a handful of users an hour gives real-user data too sparse to be a rate, so the probe is useful there, but a journey that spends real money or sends real messages needs a sandbox first. Do not use probes as the availability figure in contracts unless the journey represents what the contract promises.
Interview question
Q: All dashboards are green and customers on one carrier cannot log in. What monitoring was missing, and how would you design it?
What a strong answer covers: the difference between internal and outside-in observation; multi-vantage-point journeys with a quorum rule; step-level assertions; tagging synthetic traffic and sandboxing side effects; pairing with a segmented real-user success rate; the arithmetic of detection time; and the maintenance cost of scripts.
Quick check
Quiz: Why page on 2 of 3 vantage points failing instead of any one? One probe host can fail on its own; two agreeing indicates a path or origin fault.
Flashcard: What is detection time for a synthetic probe? — Interval multiplied by the number of consecutive failures required.