At 09:00 a customer reports that a report they exported contains another customer's records. You have the on-call. Walk me through your first hour, and tell me what architectural question you would ask on day two.
Show the full answer Hide the answer
What the interviewer is testing
Whether you can hold two incompatible pressures at once. Containment destroys evidence, and investigation prolongs exposure. A candidate who only talks about fixing the bug has missed that this is a potential data breach with notification clocks attached, and one who only talks about legal process has left the leak running.
The first hour
Minutes 0 to 10. Establish scope crudely and stop the bleeding. Disable the export feature, for everyone, now. It is a single feature and the cost of having it off for a few hours is far below the cost of another cross-tenant export. Confirm from the reported artefact what actually leaked: which fields, how many records, whose.
Minutes 10 to 20. Preserve before you touch. Snapshot the logs, the generated file, the request trace and the relevant database state. A rollback or a redeploy can destroy what the investigation needs, and the question "how many customers were affected" will be asked in a form that requires evidence, not recollection.
Minutes 20 to 40. Determine blast radius from data, not from reading code. Query the export logs for the pattern that produced this one. The critical question is whether this is one request or a class of requests, and the answer usually comes from a correlation between exports and a shared cache key, a missing tenant predicate or a job that ran with the wrong context. Reading the code tells you what should happen; the logs tell you what did.
Minutes 40 to 60. Engage the people whose clocks have started. Legal, privacy and the incident commander. Regulatory notification windows are measured from awareness and can be as short as 72 hours, and awareness began at 09:00 whether or not engineering has finished. Tell the reporting customer something true and specific.
Common weak answers
- Rolling back immediately. It feels decisive and it destroys the state that would have told you the scope.
- "We fixed the query" as a conclusion. One fixed query does not tell you whether 3 exports or 3000 were affected, and that number is the entire regulatory question.
- Waiting to disable the feature until the cause is understood. Exposure continues while you investigate, and investigation is slower than you think.
- Treating it as a bug. The mechanism is a bug; the incident is a data breach until proven otherwise, and the two have different obligations.
The day-two question
Why was cross-tenant data reachable at all? Not "why did this query omit the filter", but what in the architecture made a missing filter sufficient to cause this. The answers that matter:
- Is the tenant predicate applied in the data access layer or written by hand in each query? Hand-written means this recurs.
- Is tenancy enforced by the database, through row-level security or separate schemas, so an application mistake yields nothing rather than yields somebody else's rows?
- Do background and export jobs run with a tenant context at all, or do they run as an administrative identity that can see everything? Export and reporting paths are the classic gap, because they are built later, by different people, against a different data store.
- Is there a test that asserts cross-tenant reads return empty, running on every build?
What a strong answer adds
A canary: synthetic records in each tenant that must never appear in another tenant's output, checked automatically on generated exports. It turns the next occurrence into an alert at minute one rather than a customer email at 09:00. And the honest cost note - the strongest fix, per-tenant isolation in the store, is expensive and often unwarranted, so prefer enforcement in one code path plus a detection that does not depend on a customer noticing, and revisit isolation only when a contract or a regulator requires it.