A platform team owns one CI template that 120 repositories reference by branch rather than by version so every build pulls the template's current HEAD. At 09:05 the platform merges a change to the template's test stage. What happens over the next hour and what mechanism stops it?

What happens, in order

09:05. The merge lands. Nothing deploys, because there is nothing to deploy - the next build in each of the 120 repositories picks up the new template on its own.

09:06 to 09:30. Builds start failing in whatever repositories happen to run first. At a few hundred builds a day across the estate, the failure spreads at the rate teams happen to push, so it arrives as a slow drip rather than as an event. Each team sees its own pipeline red on its own change and starts debugging its own change.

09:30. Nobody can merge anything. The template is in the required checks, so the failure is a global merge freeze - including for the fix. If the platform's own repository uses the template, the revert cannot pass its own pipeline, which is the loop that turns a ten-minute incident into an hour.

Through the morning. In-flight releases stall mid-pipeline. A build that already failed cannot be rolled back - there is no previous version of a build to go back to - so every stalled change has to be re-run after the fix, and the queue absorbs an hour of accumulated pushes at once.

Why detection lags

The signal is spread across 120 repositories and the tickets arrive attributed to the wrong cause. The platform team's own dashboard looks fine: their repository is green, their deploy succeeded, and their change was not a deploy as far as any tool was concerned.

A template referenced at HEAD is a production dependency with no version, no rollout and no owner on call. That is the structural fact; the test-stage edit is just what exposed it.

What stops it

1. Immutable references. Repositories pin a tag or a commit digest, never a branch. This is the single change that converts a fleet-wide push into a per-repository upgrade.
2. A cohort rollout, because a template change is a deploy. Three to five volunteer repositories on the new version for a day, then roughly 10% of the estate, then the rest. The platform's own repository goes last, so the revert path always works.
3. A test matrix for the template itself, exercising the real shapes in the estate - a monorepo, a library that publishes, a service with a database - rather than one example repository.
4. An estate-level signal. Alert on build failure rate across all repositories and on the share of repositories on each template version. Per-repository alerting cannot see this class of failure at all.
5. A stated rollback: previous version still referenceable, and a one-line instruction teams can apply themselves without waiting for the platform team.

What it costs

Pinning reintroduces the problem it solves. Within a year the estate holds a long tail of old template versions, and the security fix you need everywhere reaches nothing - the drift failure the library's template-propagation practice exists to handle. Pay for it with automated bump pull requests and a supported-version window, and track the version distribution so the tail is visible before it matters.

The honest cost of the whole arrangement is that the platform team now runs a release process for a build system: versioning, changelogs, deprecations and a cohort. That is weeks of work and permanent overhead.

When this is the wrong answer

At six repositories owned by one team, referencing HEAD is correct. Everyone notices within minutes, the fix is one commit, and the versioning machinery would cost more than the failures. The threshold is not a repository count so much as a question: can one person fix every broken build before the next team is blocked? When the answer becomes no - usually somewhere between 15 and 30 repositories, or the first time two teams are affected who do not share a chat channel - pin and roll out in cohorts.