Stop Authority

In most incidents the expensive minutes are spent not on repair but on agreement. The responders hold different hypotheses, each is plausible, and the system keeps doing whatever it is doing while they work. Where damage accumulates at a rate, the absence of a pre-agreed right to stop is itself a decision, made by default, at a price nobody chose.

Stop authority separates two decisions that groups habitually fuse. "What is wrong" is a diagnostic question that takes as long as it takes. "Should this keep running" is a risk question answerable with no diagnosis at all, and it belongs to one person who does not need anybody's agreement to answer it.

Why it matters

The manufacturing analogue is the andon cord, where any worker may halt the line. The reason it works there is not the cord, it is that management declared in advance that stopping wrongly is acceptable. Without that declaration, the authority exists on paper and nobody uses it, because the person holding it is weighing a certain personal cost against an uncertain organisational one.

Software teams reproduce the failure precisely. Anyone may raise a concern; nobody may act alone; the group deliberates; the loss rate continues.

Implementation patterns

• Name the role, not the person, so it is always covered: the incident commander, the on-call lead, whoever holds the pager.
• Declare the acceptable cost of a wrong stop, in writing. "We would rather stop trading for an hour unnecessarily than continue for ten minutes wrongly." Without this sentence the authority is theoretical.
• Make the mechanism exist and test it. A documented kill switch, a feature flag, a traffic drain, a rate limit to zero. If stopping requires a deploy or a vendor call, the authority cannot be exercised inside the window that matters.
• Give every automated signal a response contract: expected action, owner, escalation on silence. A signal nobody must act on should be deleted, because its presence trains people to ignore the channel it arrives on.
• Review uses, not just non-uses. A stop that turned out to be unnecessary should be praised publicly the first time, or the second one will not happen.

Industry example

Knight Capital, 1 August 2012: a deployment left one server running repurposed dead code, and erroneous orders flowed for about 45 minutes, costing roughly \(440M, which is on the order of \)10M a minute. The SEC's 2013 settlement order records that the firm's systems had sent 97 automated emails flagging errors before the market opened, that these were not designated as system-critical notifications, and that Knight had no written procedures for when to shut the system down. Both halves of this term are in that record: a signal with no response contract, and no authority to stop while the cause was unknown.

Failure scenarios

• Deliberation at a loss rate. The group converges on a cause after the damage is done.
• Authority without mechanism. The person allowed to stop discovers stopping requires a release.
• Authority without cover. Nobody uses it because the cost of a wrong stop has never been declared acceptable, so it falls entirely on the individual.
• Alert channels with no severity, where the warning arrives in an inbox alongside routine mail and is read after the incident.
• Stopping the wrong thing, because the blast radius of the stop was never worked out; a global halt when a per-tenant one would have done.

Trade-offs

Stop authority buys bounded loss and pays in availability and in occasional unnecessary disruption. It is only affordable where the damage rate is high and the stop is cheap relative to it. It also concentrates a consequential decision in one person under pressure, which is why the mechanism should make the stop reversible: a drain you can undo beats a shutdown you must recover from.

When not to use it

When damage does not accumulate with time. A recommendation service returning poor results, a slow dashboard, a delayed batch: stopping buys nothing, and the ten minutes spent understanding costs nothing. Pre-authorised halting is for systems where money, data corruption or safety accrue per minute — trading, payments, bulk deletion, physical actuation, mass communications. Applying it everywhere produces a culture of pre-emptive shutdowns, which is its own availability problem and trains the organisation to distrust the mechanism.

Interview question

Q: During an incident your team is split between two theories and the system is corrupting about 40 records a minute. You are the on-call lead. What do you do in the first two minutes, and what should have been true beforehand?

What a strong answer covers: stopping the corrupting path immediately without resolving the disagreement, because the stop decision does not require the diagnosis · choosing the narrowest stop that bounds the damage · what should have been in place: a named authority, a tested mechanism, a declared acceptance of the cost of a wrong stop · the reconciliation question, since 40 records a minute for twelve minutes is a data repair job that begins the moment the bleeding stops · and how you would write this into the runbook so the next person does not have to be brave.

Quick check

Quiz: Why must the decision to stop be separated from the decision about the cause? Because the first needs no diagnosis and the second can take hours, and fusing them means the damage continues for as long as the disagreement does.

Flashcard: What makes stop authority real rather than theoretical? — A declared, written acceptance that a wrong stop is a cost the organisation will carry without blaming the person who made it.