Knight Capital: $440 Million in 45 Minutes
also called Knight Capital Incident
A deployment that reached seven of eight servers, combined with a reused feature flag, activated dormant test code and destroyed the company in three quarters of an hour.
What happened
On 1 August 2012, Knight Capital deployed new trading code to eight production servers. The deployment reached seven; one server was not updated, and nobody noticed.
The new code reused a flag that had previously controlled an old, dormant piece of functionality known internally as Power Peg — test code that had been unused for years but was still present in the codebase. On the seven updated servers, the flag activated the new behaviour. On the eighth, it activated Power Peg.
When trading opened, the eighth server began executing erroneous orders at high volume. Knight lost approximately $460 million in about 45 minutes. The firm did not survive as an independent company.
The compounding failures
No deployment verification that confirmed all eight servers were running the expected version.
Dead code retained in the production codebase for years after it stopped being used.
Flag reuse, where a semantically meaningful control was repurposed for a different behaviour rather than being retired and a new one introduced.
No kill switch and no rehearsed rollback, so the response was manual and slow while losses accumulated by the second.
Alerting that did not distinguish the catastrophic from the routine, so the signal was there and not acted on decisively.
The transferable lesson
This is the canonical argument for a set of practices that are individually unglamorous:
Deployment must verify what is actually running, not assume the deploy succeeded — deploy by digest, and reconcile the running fleet against the expected version. Remove dead code, because code that is not executed is not tested and is therefore a latent behaviour of unknown shape. Never reuse a flag name or value for a different purpose; retire it and introduce a new one. And have a tested kill switch for anything that transacts, because the time between "something is badly wrong" and "it has stopped" is the whole of the loss.
Every one of these is cheap. The incident is what they are cheap relative to.