intermediate 3 min answer

Every analyst gets a personal warehouse schema with create-table rights, no expiry, and permission to grant read on their objects to colleagues. The platform team considers this the self-service tier and governs only the certified tier. What happens over 18 months, and what stops it?

self-servicesandboxshadow pipelinesgovernancelineage
Show the full answer Hide the answer

What happens, in order

Month 1 to 3. The sandbox works exactly as intended. Analysts answer questions faster than the governed tier can, which is the point and is good.

Month 3 to 6. A useful sandbox table gets shared. Someone schedules a refresh of it, because the warehouse's scheduler is right there. It now has consumers and a cadence, which are the two properties of a production dataset, and none of the properties of a governed one.

Month 6 to 12. A certified dashboard reads it - usually indirectly, through a second sandbox table. The certified tier's lineage is now rooted in an object with no owner of record, no test, and no change notice. The governance programme's own certification is resting on the ungoverned tier and the certification process cannot see it, because it checks the model it certified and not what that model reads.

Month 12 to 18. The author changes team or leaves. The schema survives because deleting a schema that something reads is nobody's idea of a safe Tuesday. The SQL that built it lived in a notebook cell that is now gone. The table is reloaded by a job whose definition is a saved query nobody can explain.

Where it amplifies

The failure is not a broken table; it is a wrong number that reconciles against itself. The sandbox copy drifts from the governed definition after the governed one is corrected, and every consumer of the sandbox path agrees with every other, so the disagreement looks like one team being wrong rather than a structural duplicate.

What stops it

  • Grants expire, not just objects. A read grant out of a personal schema gets a 90-day life. Expiry on the object alone is the control people try first and it fails, because the reaction to a deleted table is to recreate it.
  • Refuse scheduling in personal schemas. Scheduling is the line between exploration and production. Crossing it should require a one-week promotion path - a repository, an owner, a freshness check - and that path has to be genuinely one week or nobody uses it.
  • Two detection queries, run monthly. Objects in personal schemas read in the last 30 days by a principal other than the owner. And any personal-schema object appearing upstream of a certified asset, which should be zero and is the only one worth paging on.
  • A quarterly orphan sweep on schemas whose owner has left, with a 30-day notice to readers.

What would have to be true for it to self-heal

Nothing in the system makes a sandbox less useful over time, so there is no restoring force. It self-heals only if the governed path is faster than rebuilding in the sandbox, which is a claim about lead time from request to certified metric - measure that number, because it is the single input that decides whether governance holds.

When this is the wrong answer

With a 12-person data team the sandbox is the platform, and expiry rules are hostile theatre. Run the cross-schema read query once a quarter and talk to the two people it names. The threshold is when sandbox objects start having consumers the author cannot name - typically somewhere past 30 analysts - not a headcount by itself.