Path Assertion

An experiment injects 5 seconds of latency into a recommendation dependency. Page p99 does not move, exactly as the fallback design intended, and the team records a pass. A month later a real recommendation slowdown takes the page from 300 ms to 4 seconds.

The experiment proved nothing, because an unchanged p99 has two possible causes: the fallback worked, or the dependency was never called. In this case a warm local cache served every request during the test window, so the injected latency was never on the path.

A path assertion closes that gap. It checks that the mechanism ran, not only that the outcome looked acceptable.

Why it matters

A passing resilience experiment is treated as evidence, and it is only evidence if the mechanism under test was exercised. The false pass is more dangerous than no test at all, because it converts an untested fallback into a documented, believed-in control.

The failure is systematic rather than occasional. Experiments are usually run in quiet windows, often after a warm-up, which is precisely when caches are full, connection pools are established and working sets are stable. The conditions that make an experiment convenient are the conditions that make it bypass the path.

And fallbacks are the least-exercised code in a system by construction. A fallback that has never fired in production is untested code on the critical path, which is why the invocation counter is worth having whether or not an experiment is running.

Implementation patterns

• Assert fallback invocation count greater than zero, from a counter incremented inside the fallback branch. If it is zero, the experiment is invalid regardless of the latency graph.
• Assert observed calls to the injected dependency, at the injected latency. This proves the injection reached the call rather than a layer in front of it.
• Control the cache state explicitly. Either hold hit rate at production-normal and record it, or clear the cache as part of the experiment. The second is a different and much harder experiment, and it is the one that matters.
• Run a negative control. Repeat with the fallback disabled and confirm the outcome degrades. If nothing you do makes the system worse, the injection is not working.
• Check the injection point against the real call path. A proxy, a sidecar, a mesh and a client library each intercept different things, and a cache in front of the injection point invalidates everything downstream of it.
• Export fallback invocation rate as a first-class production metric, not only a test artefact.

Industry example

Netflix's published chaos-engineering practice makes the same point structurally: experiments are built around a stated hypothesis about steady-state behaviour, with the experiment designed to disprove it, and the control group exists so that "nothing changed" can be distinguished from "nothing was tested". The pre-registered hypothesis is what forces the question "what should have happened differently", which is exactly what a path assertion answers mechanically.

The same discipline appears in load-testing practice, where a test that reports healthy latency while the application returned cached error pages is a well-known trap. The remedy in both fields is the same: assert on the mechanism, not on the summary metric.

Failure scenarios

• A warm cache absorbing the whole experiment, so the dependency is never called.
• An injection point in front of the real call, such as mesh-level latency on a path the service reaches through a different client.
• A fallback whose counter does not exist, so invocation cannot be asserted and nobody notices the gap.
• An error-triggered fallback tested with latency, which never triggers because slow is not an error — the specific defect behind most of these false passes.
• A negative control never run, so a broken injection harness looks like a resilient system for months.
• An experiment at a quiet hour whose conditions never occur during real traffic.

Trade-offs

Path assertions cost a counter per mechanism, a cache-state check, and the discipline of running a negative control — perhaps a day of work for an existing experiment suite. The return is that the suite's results become interpretable.

The expensive part is not the assertion but what it forces: once you must prove the fallback ran, you often have to clear the cache or run during real traffic, which turns a safe experiment into a risky one. That is the real cost, and it is the cost of the experiment being meaningful.

When not to use it

It is cheap enough that there is no case for omitting the invocation counter. The parts to scope are the cache-clearing variant and the negative control.

Skip the cache-clearing variant for a dependency that is either up or hard-down — connection refused, DNS failure — where an error-triggered fallback is sufficient and a latency experiment tests a case that does not arise. And skip the negative control on a mechanism you have already seen fire in production, since production is the better evidence.

Interview question

Q: Your team's resilience suite has 30 experiments and all of them pass. The last two incidents were both caused by a dependency becoming slow. What do you look at, and what do you change?

What a strong answer covers: suspecting false passes and checking, for each experiment, whether the mechanism under test was exercised rather than whether the outcome was healthy · adding fallback invocation counters and asserting them, plus observed calls at the injected latency · examining cache hit rate during the experiment windows against production-normal · running negative controls to prove the harness can produce a failure at all · the design fix for the incidents themselves, which is a timeout well below the caller's own budget so slow becomes a fast failure, plus a concurrency limit so one slow dependency cannot occupy every caller thread · and the production metric worth exporting permanently, which is fallback invocation rate.

Quick check

Quiz: Why is "p99 unchanged during latency injection" not evidence that a fallback works? — Because it is equally consistent with the dependency never having been called; without a path assertion the two cannot be distinguished.

Flashcard: What is a negative control in a resilience experiment, and what does its absence hide? — Running the same injection with the fallback disabled to confirm the system does degrade. Its absence hides a broken injection harness, which looks exactly like resilience.