An interviewer describes the Knight Capital failure from the SEC's 2013 settled order - new order-routing code reached seven of eight servers; the eighth still held a retired function that the repurposed flag activated - then asks "the deployment was peer-reviewed by nobody and monitored by everybody. What property of the deployment system, not of the review, would have made this harmless?" What is being tested and what does a strong answer contain?

What the interviewer is testing

Whether you can tell a preventive control from a detective one, and whether you price them. Almost every candidate answers "a second reviewer" or "better alerting". Both are detective, and the order's own numbers show why that is the wrong class of answer here.

The number that settles the class of control

The SEC order (Release 34-70694, October 2013) states that Knight accumulated an unintended multi-billion-dollar portfolio in approximately forty-five minutes on 1 August 2012 and lost more than \(460 million**. That is roughly **\)10 million a minute.

A detective control's residual loss is its detection-and-response time multiplied by the loss rate. A ten-minute human response - fast, for a novel symptom at 09:30 on a trading day - leaves a $100 million floor. No amount of alert tuning changes that floor, because the floor is set by how fast a person can decide. When loss accrues faster than a human can be convinced, the control has to be preventive. That single sentence is what the question is reaching for.

The property, named

A partially-deployed fleet must be unable to serve, not merely visible as partial. Four mechanisms give it:

1. Version in the routing identity. The order carries the build it expects; a node that cannot satisfy it refuses rather than interpreting. Partial deploy becomes an error rate, not a behaviour change.
2. A fleet-version assertion as a gate on the flag. The flag that repurposed the field cannot flip until every node in the target set reports the new build. The deploy and the semantic change stop being two independent events.
3. Dead code removed by the build, not by discipline. A function with no live caller fails the build. Retired behaviour that remains callable is an unexploded ordnance with a flag for a fuse.
4. A kill switch scoped to the capability, not the process. "Stop building positions" must be one action with a known effect, available to whoever notices first, and rehearsed - not "turn off the servers", which during the real event is a decision nobody wants to own.

Common weak answers

• "A second technician should have checked." The order does note there was no second review and no written procedure requiring one. But a second pair of eyes on a file copy is a control whose reliability is a human one, deployed against a failure mode whose cost per minute is $10 million. It reduces frequency and does nothing to the consequence.
• "Canary the deploy." Reasonable and it inverts here. The canary was the eighth server - the one node with different behaviour - and the system had no way to notice that one node was behaving differently because nothing compared node behaviour.
• "More monitoring." Detective again, and the failure was not invisible. It was visible and unstoppable.

What a strong answer adds

The sequencing rule: never let a flag's meaning differ by node version. Repurposing a field is a protocol change, and a protocol change requires that all speakers agree before any speaker uses it - expand, then migrate, then contract.

And the general lesson, stated as a test you can run on your own system this week: list the changes whose blast radius per minute exceeds what a human can authorise stopping. For each one, the control must be a gate, not a graph. Everything else can be detective.