A latency-injection experiment adds 5 seconds to the recommendation dependency. Page p99 is unchanged, exactly as the fallback design intended, and the team records a pass. A month later a real recommendation slowdown takes the page p99 from 300 ms to 4 s. What did the experiment actually prove, and what was missing?

The first three things I would look at, and why in that order

1. Was the fallback path actually executed during the experiment? This is the question the experiment never asked. A healthy outcome is consistent with the fallback working and with the fallback never being needed, and only one of those is what was being tested.

2. How the injection was applied. Latency injected at a proxy, a sidecar, a service mesh or in the client library each intercepts a different path. If the injection point sits in front of a cache that the page reads first, the slow dependency is never called at all.

3. The cache hit rate during the experiment window compared with normal. If it was near 100% during the test and sits at 70% in production, the experiment ran under conditions that do not occur.

The diagnosis

The injection bypassed the actual call path. The usual mechanism: the page reads a local cache, the experiment's latency was injected on the network hop to the dependency, and during the experiment window the cache was fully warm — often because the experiment itself was preceded by a warm-up, or because it ran at a quiet hour with a stable working set.

So the result "p99 unchanged" is true and means the cache served everything, not that the fallback works. In the real incident, the dependency was slow and the cache had normal misses, so the misses queued behind a 4-second call, caller threads filled, and the page tail followed the dependency.

The misleading signal is the clean pass itself. A passing experiment feels like evidence and is only evidence if the mechanism under test was exercised, which is the thing nobody checks.

What was missing: a path assertion

The experiment should have asserted the path, not only the outcome:

• Fallback invocation count greater than zero, from a counter incremented in the fallback branch. If it is zero, the experiment is invalid regardless of how good the latency graph looks.
• Observed calls to the injected dependency greater than zero, and their observed latency at the injected value. This proves the injection reached the call.
• Cache hit rate held at production-normal during the window, or the cache deliberately cleared as part of the experiment. The second is a different and harder experiment, and it is the one that matters.
• A negative control. Run the same experiment with the fallback disabled and confirm the page does degrade. If nothing you do makes the page worse, your injection is not working.

The fix for the real failure

Beyond the test design: a timeout on the recommendation call well below the page's own budget — at a 300 ms page target, something like 80 to 120 ms — so a slow dependency becomes a fast failure that triggers the fallback. Slow is not an error, and a fallback triggered only by errors is not triggered by the most common failure mode. Then a concurrency limit on that call, so a slow dependency cannot occupy every caller thread.

When not to apply this level of rigour

For a dependency that is either up or hard-down. If the failure mode is connection-refused, an error-triggered fallback is sufficient and the latency experiment is testing a case that does not arise. The path assertion still costs almost nothing, so keep it; the cache-clearing variant is the part to skip.

The alert that would have caught it earlier

Fallback invocation rate, exported as a first-class metric. A fallback that has never fired in production is untested code on the critical path, and its invocation count is the cheapest available evidence about whether resilience machinery is real. Nobody was looking, because the dashboard showed the page was healthy.