Silent Control Failure

Every control has two properties an audit tests separately. Design effectiveness asks whether the control, running as described, would prevent or detect what it exists for. Operating effectiveness asks whether it actually ran, across the whole period, and produced a result.

The dangerous gap is a control that is well designed and not operating, because the organisation behaves exactly as if it were protected. A backup job's success signal is that it exited zero, which tests the job and not the backup; a scanner that silently stopped receiving new repositories still reports 100% of what it can see.

GitLab's January 2017 incident is the canonical illustration: five protection mechanisms, none of them producing a usable recent restore when it mattered, and roughly 6 hours of data lost.

Why it matters

The cost is not the control's absence. It is the decisions taken on the strength of it: a retention policy signed off because deletion is "automated", a risky migration approved because "we can restore", a vendor accepted because "the scanner covers it".

Silent failure also defeats the assurance cycle. An annual sample tests what the control produced; if the control produced nothing, there is often nothing to sample, and an absence of exceptions reads as a clean result.

Implementation patterns

• Verify the property, not the process. "Backup job succeeded" is a process signal. "A restore reached 99.9% of production row count in 45 minutes" is a property signal.
• Give every control a consumer. A control whose output nobody reads will stop working unnoticed. Route it to a dashboard someone is accountable for, or to a gate that blocks.
• Alert on absence, not only on failure. The most common silent failure is a job that stopped running, so alert when an expected record does not arrive within its window.
• Monitor coverage as a first-class metric: repositories scanned over repositories that exist, accounts reviewed over accounts that exist. Coverage drift is how a control becomes partial.
• Treat warnings as failures or delete them. A warning nobody reads is a failure with extra steps, which is precisely what the 2017 version mismatch was.

Industry example

GitLab published its incident in near-real time and then in a detailed postmortem. Several backup and replication mechanisms existed; one produced near-empty output because of a version mismatch that printed only a warning, and others were not configured where people believed. Recovery depended on a snapshot taken hours earlier for an unrelated reason. What made this a silent failure rather than a gap is that nobody thought they were unprotected, and the organisation had been operating for months on that belief.

Failure scenarios

• A backup that has been failing for 90 days with a green job status.
• A control whose scope silently narrowed: new accounts created outside the reviewed directory, new repositories outside the scanner's organisation.
• An alert routed to a channel nobody reads, which is functionally identical to no alert.
• A gate left in warn mode after a rollout, so it reports violations and stops nothing.
• A quarterly review performed by approving everything, which operates but does not control.

Trade-offs

Verification costs real work: restores consume compute and a scratch environment, coverage metrics need an inventory to compare against, and alerting on absence generates pages during legitimate quiet periods. The exchange is a predictable running cost against an unbounded, undetected exposure, and it is one of the few governance investments that pays in incidents avoided rather than in documents produced.

When not to use it

Not every control deserves continuous verification. For a low-consequence control on a small population — a naming convention, an internal tagging rule — an annual look is proportionate and the machinery costs more than the risk. Reserve the verification effort for controls whose failure would be discovered during an incident: backups, access revocation, deletion, encryption, deployment gates.

Interview question

Q: You join a team whose compliance dashboard is entirely green. What would you check first to find out whether that is true, and what would convince you either way?

What a strong answer covers: asking each control what it would look like if it had stopped, and whether anyone would know; comparing coverage against an independent inventory rather than against the control's own view; picking the two or three controls whose failure would be catastrophic and testing them end to end, restore included; looking for warn-mode gates and muted alert channels; and treating an absence of exceptions as a question rather than an answer.

Quick check

Quiz: Why is "the backup job succeeded" not evidence that backups work? — It tests that the job ran, not that its output can be restored; only a restore tests the property.

Flashcard: What is the single cheapest defence against silent control failure? — Alert on absence: if the expected record has not arrived within its window, page someone.