Behavioural Specification Gap

Three teams follow the published reference architecture exactly. All three end up with unbounded retries against a shared dependency, and two of them cause incidents. Nobody ignored the standard. The standard did not contain the thing that failed.

Three identical defects are not three mistakes. They are one mistake, made once, upstream of all of them - usually a library default that each team inherited because the reference said nothing about it.

Why it matters

Compliance with an underspecified standard produces correlated defects, which is strictly worse than variety. Independent teams making independent mistakes fail at different times in different ways; teams inheriting the same gap fail together, under the same conditions, across the estate.

The diagnosis is also routinely wrong. Three teams, three incidents reads as a competence problem and produces a training response. The identical shape of the defect is the evidence that it is not independent - three genuinely independent errors would look different - and the training response leaves the fourth team to make it again.

Structure is what diagrams carry well: which components exist, who calls whom, where data lives. Behaviour under failure - timeouts, retry budgets, backoff, jitter, deadline propagation, what happens when a dependency is slow rather than down - is where incidents come from and is exactly what a box-and-arrow reference omits.

Implementation patterns

• Ship the reference as a working service, not a document. A repository that builds, deploys and passes its own tests, with resilience behaviour already correct. Teams clone it, so the behaviour is inherited rather than re-derived.
• Move the policy out of the template and into a shared client or the service mesh, so it is configured once and cannot be re-invented per service.
• Specify the numbers, not the intention. "Retry budget capped at 10% of the first-attempt rate · full jitter · deadline propagated on every hop · no retries on non-idempotent operations" beats "retry appropriately".
• State the failure semantics for each dependency in the reference: what happens when it is slow, when it is down, and what the caller returns in each case.
• Test the reference against the failure, in its own repository, so the specification cannot drift from the behaviour.
• When a defect is found, fix the default and then find every consumer of it. There are always more than the ones that had incidents.

Industry example

The pattern is visible in published incident reviews across the industry whenever an outage is traced to retry amplification against a shared dependency: the individual services were built to the house pattern, and the house pattern specified structure. The generic form has been documented since Google's SRE material of the 2010s popularised retry budgets and deadline propagation precisely because per-service retry logic, written independently against a shared standard, converges on unsafe defaults.

The operational signature is consistent: a shared dependency degrades slightly, caller retry volume triples, and the dependency is pushed from degraded to down by its own clients.

Failure scenarios

• Retry amplification. Each of N callers retries three times, so a 20% degradation becomes a 3x load increase on a system already struggling.
• Timeouts longer than the caller's own deadline, so a slow dependency holds every caller thread and saturation propagates upward as latency rather than as errors.
• No jitter, so retries synchronise into waves and the dependency is hit by a periodic thundering herd.
• Retries on non-idempotent operations, producing duplicate side effects that surface as a reconciliation mismatch days later.
• Correlated deploys of the same defect, so the fix must be rolled out to every service rather than one.
• The reference updated and nobody re-derives, because the teams copied a diagram and there is no link back to the source.

Trade-offs

Executable references cost real maintenance: a repository that builds and deploys needs dependency updates, CI, and an owner, where a diagram costs nothing after publication. They are also more opinionated, which some teams will resist, and they can encode a wrong default estate-wide just as efficiently as a right one - centralisation makes the blast radius of the standard itself larger.

The counterweight: a document that specifies behaviour is read once and implemented from memory, and the memory is the library default.

When not to use it

For genuinely structural guidance, a document is the right artefact and executable form adds nothing: how domains are split, where a boundary sits, which systems own which data, what the integration contract is between two organisations. None of that has runtime behaviour to get wrong.

Also when there is exactly one consumer - a reference for a single team is just that team's code - and in estates too heterogeneous for a shared template to run, where the better investment is the shared client library alone.

Interview question

Q: You are writing the reference architecture for how services in your estate call a new shared rate-limited API. What goes in it, and how do you make sure the twentieth team gets it right without reading your document carefully?

What a strong answer covers: the behavioural specification with numbers - timeout below the caller's remaining budget, retry budget as a fraction of first-attempt rate, full jitter, deadline propagation, no retries on non-idempotent calls, and what the caller returns when the limit is hit · shipping it as a client library or mesh policy so the twentieth team inherits rather than implements · a runnable example service · the observability that proves it - retried over first-attempt ratio per caller - and an alert on it · and the recognition that the document is the weakest of these mechanisms and should carry only what code cannot.

Quick check

Quiz: Three teams produced an identical defect while following the reference. Why is training the wrong response? Because three identical defects are one upstream defect - a shared default or a gap in the reference. Training three teams leaves the fourth to make it a fourth time; fixing the default fixes it everywhere.

Flashcard: Which alert catches retry amplification before error rates move? — The ratio of retried to first-attempt requests per caller against the dependency. The early phase of a retry storm looks like a healthy traffic increase in every other metric.