A support notes column is labelled Internal. A scan of 10,000 sampled rows finds payment card digits in about 0.4% of them and health details in about 0.2% - agents paste what the customer sent them. The table has 50 million rows and feeds three downstream marts. Why does column-level classification fail here, and what are the two honest options?
Show the full answer Hide the answer
The mechanism
A sensitivity label is a statement about what a column is for, and a free-text column is not for anything. Every other column in the table has an intended domain - this one holds an email address, that one a currency code - so a label is a true and stable fact about it. A notes column holds whatever a human typed, which means its content is drawn from the whole space of things customers send. Labelling it Internal is not wrong today and right tomorrow; it was never a description of the column, only a hope about it.
The numbers make the exposure concrete. At 0.4% of 50 million rows, roughly 200,000 rows contain card digits, and they have already been copied into three marts, which inherited the Internal label and its wider access.
The two honest options
- Label it at the highest sensitivity any input could reach, and accept what that costs. The column becomes Restricted, the marts that carry it become Restricted, and self-service analytics loses the table. This is correct and it is usually how a free-text column leaves the analytical estate altogether, replaced by a flag or a category that the agent selects.
- Control what enters, so the label becomes true. Detect and redact on write - card and identifier patterns replaced at the point the note is saved, or the note stored in the operational system and never copied to analytics. After that, Internal is an accurate label because the column's domain is now enforced rather than assumed.
The choice between them is about who needs the text. If analysts read notes to find themes, pay for redaction. If nobody reads them and they are copied because the pipeline copies everything, stop copying.
What does not work
- Scan and quarantine after the fact. Finding the 200,000 rows is the easy part; by the time you find them the data is in three marts, a search index and whatever extracts people took. Detection without a write-time control is a permanent cleanup job.
- Telling agents not to paste. They are pasting what the customer sent while the customer waits. A policy against the shortest path to doing the job loses.
- Masking the column in the BI tool. The column is still there for anything that reads the table directly, which includes every pipeline.
When this is the wrong answer
A 20,000-row notes column that nothing copies downstream does not need a redaction pipeline. Mask it on read in the one tool people use, restrict the table, and write down why. The investment scales with how far the column has already travelled, not with how sensitive its worst row is - one copy in one place is a configuration problem, and three marts and an index is an engineering project. Check the lineage before choosing.