Three teams followed the published reference architecture exactly. All three produced the same defect - unbounded retry behaviour against a shared dependency - and two of them caused incidents. The reference architecture is not being ignored. Diagnose.

The first three things to look at

1. What the reference actually specifies about the failing behaviour. Almost certainly: nothing, or a sentence of prose. A diagram showing a service calling a dependency says nothing about retry counts, budgets, jitter or deadlines.
2. Whether the reference ships as a document or as code. A diagram costs nothing to publish and carries none of the behaviour; If teams copied a diagram, each team invented the retry logic separately and they converged on the same wrong default because it is the library's default.
3. The library defaults themselves. Three teams producing an identical defect is not three mistakes; it is one mistake, made once, somewhere upstream of all of them - and it is now running in production in three places.

The diagnosis

The reference architecture described structure and the failure was in behaviour. Structure is what diagrams carry well: which components exist, who calls whom, where data lives. Behaviour under failure - timeouts, retry budgets, backoff, deadline propagation, what happens when the dependency is slow rather than down - is where incidents come from, and it is the part a box-and-arrow reference silently omits.

The teams complied. Compliance with an underspecified standard produces uniform defects, which is worse than variety, because now the failure is correlated across the estate and will arrive everywhere at once.

The misleading signal

"Three teams, three incidents, so the teams need better training." It reads as a competence problem because three groups independently got it wrong. The identical shape of the defect is the clue that it is not independent - three independent errors would look different.

The fix

• Ship the reference as a working service, not a document. A repository that builds, deploys and passes its own tests, with the resilience behaviour already correct. Teams start from a clone. The behaviour is inherited rather than re-derived.
• Move the retry policy into a shared client or the mesh, so it is configured in one place and cannot be re-invented per service.
• Add the missing specification to the reference explicitly: a retry budget capped at 10% of the first-attempt rate, full jitter required, deadline propagated, no retries on non-idempotent operations.
• Fix the default, then find every other consumer of it. There are more than three.

The alert that would have caught it earlier

Ratio of retried to first-attempt requests against the shared dependency, per caller, with a threshold. Retry storms are visible in that ratio long before they are visible in error rates, because the early phase looks like a healthy increase in traffic.

When to prefer a document anyway

For genuinely structural guidance - how domains are split, where a boundary sits, which systems own which data - a document is the right artefact and executable form adds nothing. Prefer a document for structure and code for behaviour: anything specifying behaviour under failure should be executable, because prose about timeouts is read once and implemented from memory. The cost of getting that backwards is an estate-wide correlated defect rather than a local one.

Common weak answers

• "The teams need training." Three identical defects are one upstream defect. Training three teams leaves the next team to make it a fourth time.
• "Add retry guidance to the wiki." The wiki is where the current reference already is.