advanced 3 min answer

An interviewer asks how you decide how much to say publicly, and how often, during an incident in which you have lost customer data. GitLab live-streamed its January 2017 database recovery and kept a publicly visible running document. Argue both sides of that level of disclosure, and say what you would commit to.

gitlabincident communicationdisclosuretransparencytrust
Show the full answer Hide the answer

What the interviewer is testing

Whether you can separate three things that get fused under pressure: what is true, what is useful to say, and what you can commit to. The weak answer is a position ("be radically transparent" or "say nothing until legal clears it"). The strong answer is a decision rule with the conditions that move it.

The documented extreme

On 31 January 2017 an engineer troubleshooting replication removed a PostgreSQL data directory on the wrong host, and GitLab's recovery depended on a staging snapshot taken roughly six hours earlier because LVM snapshots were not enabled on the database servers. The published postmortem records that data modified between 17:20 and 00:00 UTC was lost, affecting roughly 5,000 projects, 5,000 comments and 700 new user accounts.

The communication decisions are the part worth studying. Progress was tracked in a publicly visible document, the recovery itself was live-streamed publicly and peaked at about 5,000 concurrent viewers, and social media carried updates for people not watching. The postmortem was then published on the company blog with the sequence of events and the failed recovery paths named.

The clarifying questions that change the answer

  • Is the lost data the customers' own work, or derived state you can recompute? Lost user content obliges disclosure of scope, because only the customer can tell what is missing.
  • Who is the audience? A developer tool's users are technical, tolerate uncertainty and reward detail. The same stream for a consumer bank would be reckless.
  • Is there a regulatory clock? A personal-data breach starts one, and a statement made in hour two constrains the notification in hour fifty.
  • Is the incident still live and adversarial? Live commentary during a security incident tells the attacker what you can see.
  • Does the exposure include credentials, keys or third-party data? That caps detail hard.

A strong answer's arc

State the rule: commit to a cadence and to scope, never to a cause or a restoration time. In practice that means an update at a fixed interval even when the content is "no change", because a published rhythm is what stops people asking; the scope of impact in the units customers care about ("projects created after 17:20 UTC"), published as soon as it is bounded rather than when it is exact; and a named mechanism by which a customer can check whether they are affected.

Then argue both sides honestly. What the live stream bought GitLab was the thing companies usually lose in an outage: the assumption of concealment. An audience watching the recovery cannot suspect a cover-up, and the postmortem's credibility was borrowed from the stream. What it cost was control of the narrative and the responders' attention — every observer is a potential commentator, mistakes are made in public, and a company whose product was not a developer tool would have paid a much higher price for the same candour.

Common weak answers

  • "Be fully transparent because trust matters." A slogan. It does not survive credentials in the blast radius, a live attacker, or a regulator's sequencing.
  • "Wait for the postmortem." Silence during the incident is read as concealment, and the postmortem then has to overcome that as well as explain the failure.
  • "Give an ETA so customers can plan." A restoration time leaves the room without its caveats and is repeated to customers by people who never heard them.

What a strong answer adds

The internal consequence. Public updates cost responder time, so the update path needs a person who is not fixing anything, working from a template agreed before the incident, with pre-agreed limits on what may be said without a second pair of eyes. Decide the disclosure posture in advance, because a policy written at 02:00 by tired people under pressure is the one thing in this whole scenario that is certain to be wrong.