advanced 3 min answer

Review this emergency data access design. A shared role called prod_break_glass grants read on every production table. Any team lead can approve a request in a chat channel. The grant has no expiry and is removed when someone remembers. Every use writes a row to an audit table in the same warehouse. It was used 140 times last year. What would you remove, what would you change, and what would you leave alone?

access-controlbreak-glassauditexpiryattribution
Show the full answer Hide the answer

What is actually required

One path to production data during an incident that is faster than the normal request workflow, leaves evidence the user cannot edit, and expires without anyone remembering. This design has none of the three cleanly. Note what is not required: approval by the data owner, or narrow scope. Both make the mechanism slower than the incident it exists for, so people borrow credentials instead.

What I would remove

  • The shared role. A grant to prod_break_glass tells you that someone in a group of 40 read the customer table. Attribution is the primary output of this control and a shared principal destroys it. Replace it with a per-person grant; the role becomes a template, not an identity.
  • "Removed when someone remembers." At 140 uses a year, roughly 3 a week, forgetting a handful produces permanent broad read access held by people unaware they have it. This is how standing access is created by a control designed to eliminate it.
  • The audit table in the same warehouse. A role broad enough to read everything is one configuration slip from writing to it. An audit record the subject can reach is not evidence.

The one change that matters

A hard time-to-live enforced by the issuing system — 4 hours, after which the grant is gone whether or not the incident is over. Everything else here is hygiene; this is the change that converts the mechanism from a way of acquiring access into a way of borrowing it. Expiry enforced by a scheduler the requester cannot influence, not a reminder and not a weekly sweep. Prefer a short TTL with renewal on request over a long grant, because each renewal is a second evidence record.

Three cheap additions alongside it: a recorded reason tied to an incident identifier, an append-only log in a separate account under a different identity, and a notification to the data owner at grant time, which costs nothing and is the only thing making unusual use visible while it happens. Review every use within 5 working days; a slower cadence is a logging exercise, which is what auditors have been reporting on break-glass controls since well before 2020.

What I would leave, even though it looks odd

  • Approval by any team lead. Reviewers hate this and it is correct: at 02:00 the question is whether a second human agrees this is an incident, and any lead can answer it. Requiring the data owner adds hours and the owner approves anyway.
  • Broad scope. Narrowing the grant mid-incident is guesswork — you do not yet know which table holds the answer, which is why you are in break-glass. Broad and brief beats narrow and negotiated.
  • 140 uses a year. The number that matters is the shape, not the count: if 60 of them are one team reading one table for a recurring task, that is a missing dashboard, and building it removes the uses. Usage trending down as tooling absorbs recurring cases is the sign the control is healthy.

How I would argue this in the review

Lead with the audit table, because an auditor will reach it independently and it is indefensible once stated. Then trade: accept the chat approval and the broad scope — what the security reviewer wants to tighten — for per-person grants and enforced expiry. The control doing the work is expiry plus attribution, not approval, and saying so stops the review ending with a stricter process nobody uses.

When this is the wrong answer

In an estate where three engineers already hold permanent production read, break-glass is theatre: there is nothing to remove, and an issuing service with TTLs and a separate audit account costs more than it protects. Build it once the population with standing access exceeds the number of people you would be comfortable listing to a customer.