A privacy review finds that a customer's date of birth and partial bank details are visible in session-replay recordings for a subset of users, although the analytics vendor's configuration masks those fields. It affects roughly 3% of sessions, all on one flow. Where do you look, and what does the pattern tell you?

The first three things I would look at, and why in that order

1. What distinguishes the 3%. One flow and a small share points at a code path rather than a configuration: a variant behind a feature flag, a legacy page not migrated to the current component, or a browser where the masking mechanism behaves differently. Get the distinguishing attribute before touching the vendor's settings.
2. How the masking is specified. Most replay tools mask by selector or by attribute. Both are opt-in, which means any element that does not carry the marker is captured by default, so a new component rendered without the attribute is recorded and nothing warns anyone.
3. When the affected pages last changed. If the flow was refactored recently, the mask attributes were almost certainly dropped in the rewrite, which is the single most common cause of exactly this symptom.

The diagnosis

Nearly always: masking is a deny-list by construction, and the estate changed underneath it. A new field, a re-rendered component, a library upgrade that alters the DOM, or a dynamically inserted element arrives without the marker and is captured. The configuration is correct and incomplete at the same time, which is why the review found something the vendor's dashboard says cannot happen.

The privacy consequence is not proportional to the field count. Recordings are typically held by a third party, retained for months, accessible to a broad group of product and support staff, and outside the deletion paths built for the primary database — so data that was minimised everywhere else accumulates here.

The misleading signal

The vendor's configuration screen showing the fields as masked. It is describing the rule, not the outcome, and there is no feedback loop between them. Trusting a control that has no verification is the actual root cause, and it would have produced this result eventually even without the refactor.

The fix, in order

1. Stop recording on the affected flow now, before diagnosis is complete. This is a live disclosure and the cost of pausing analytics for a day is negligible.
2. Purge the affected recordings, with the vendor, and record what was retained and for how long. If the assessment is that personal data was disclosed to a processor outside the agreed scope, that is a decision for the privacy team on the record, not an engineering judgement.
3. Invert the default. Mask everything and allow-list the elements that may be recorded. It is more work and it fails safe, which is the only property that matters for a control on personal data.
4. Block replay on sensitive flows structurally — payment, identity verification, health questions — rather than relying on field-level rules inside them.
5. Automate the verification. A test that drives the flow, pulls the recording payload, and fails the build if any field matching a known pattern appears unmasked. This is the check whose absence made the incident possible.

The alert that would have caught it earlier

A scheduled scan of recorded payloads for patterns that look like dates of birth, card numbers, national identifiers and email addresses, alerting on any match. It is cheap, it needs no vendor cooperation beyond an export, and it turns a control that was assumed into one that is measured — which is the difference between privacy by design and privacy by intention.

When not to run replay at all

On any flow handling payment, identity verification, health or financial detail, the honest answer is that the analytics value does not justify the standing exposure, and no masking configuration is worth trusting there. Replay earns its place on browse, search and navigation flows, where the data is behavioural rather than personal and a configuration error is an inconvenience rather than a disclosure. Prefer the structural block to the field-level rule wherever the flow is sensitive enough that you would care about a mistake.