advanced 1 min answer

A team's synthetic monitoring is green and users report the site is slow. What is the monitoring missing?

rumsyntheticpercentilessegmentationdatadog
Show the full answer Hide the answer

What synthetic monitoring cannot see

  • Real device capability. Tests run on consistent infrastructure; users have devices spanning an order of magnitude in CPU performance, and JavaScript-heavy pages are CPU-bound.
  • Real network conditions, including congested mobile networks, high latency and packet loss, which synthetic tests approximate poorly.
  • Real cache states. Synthetic tests usually run cold or warm consistently; real users are distributed across both.
  • Geographic distribution. Tests run from a few locations, users are everywhere.
  • Third-party variability. A slow analytics or advertising script affects users and may not affect the synthetic run, particularly if it is blocked in the test environment.
  • The specific paths users actually take, rather than the handful of routes someone configured.

What to look at instead

Field data at the percentiles that matter, segmented. The averages hide the problem entirely — a good median with a terrible 95th percentile is a large number of users having a bad experience, and it is the shape almost every complaint corresponds to.

Segment by device class, connection type, geography, route and cache state, because the aggregate metric is a mixture of populations with very different experiences and the mixture moves when traffic composition changes.

The complementary roles

Synthetic detects regressions consistently — same conditions each run, so a change is attributable. Field data tells you what users experience but is noisy and confounded by traffic mix.

Use synthetic as a change detector in the pipeline and field data as the source of truth about experience. The mistake is using either alone: synthetic alone misses the population, and field data alone cannot attribute a regression to a release.