A deployment reaches seven of eight servers. The eighth runs old code that a reused flag activates. Losses accumulate for 45 minutes. What are the findings?
Show the full answer Hide the answer
What the interviewer is testing
Whether you can enumerate the compounding failures rather than fixating on the partial deployment.
The findings
No deployment verification. The system did not confirm that all eight servers ran the expected version. Deployment is not complete when the command returns; it is complete when the running fleet is verified to match the intended version — by digest, reconciled continuously.
Flag reuse. A control that had previously governed different functionality was repurposed. On the un-updated server it activated dormant legacy behaviour. Never reuse a flag name or value for a different purpose; retire it and introduce a new one.
Dead code retained in the production codebase for years. Code that is not executed is not tested and is therefore a latent behaviour of unknown shape. Remove it.
No kill switch. The response was manual and slow while losses accumulated by the second. Anything that transacts needs a tested mechanism to stop it immediately, independent of deployment.
No automated rollback, and no canary — the change went to full production capacity at once with no staged exposure and no automated abort.
Alerting that did not distinguish catastrophic from routine, so the signal existed and was not acted on decisively.
This is the Knight Capital incident, which destroyed the firm as an independent company in 45 minutes.
The cheapest fixes
Every one of these controls is inexpensive relative to the outcome:
Deploy by immutable digest and reconcile the running fleet. Fail the build on flags older than a defined age, and forbid reuse. Delete dead code as routine hygiene. Build the kill switch before you need it. Stage the rollout. Practise the rollback.
What a strong answer adds
The general principle: the time between "something is badly wrong" and "it has stopped" is the whole of the loss. For any system that transacts, that interval is the single most valuable thing to minimise — more valuable than reducing the probability of the fault, because faults are inevitable and the interval is controllable.
Common weak answers
Better deployment tooling alone, which addresses one of six findings. Blaming the engineer who reused the flag.