intermediate 2 min answer

A data-flow diagram in your compliance pack was drawn eight months ago and is signed off. A reviewer asks how you know it is still accurate. What would you change about how it is produced, and what would you keep doing by hand?

data flowlineageautomationcompliancedrift
Show the full answer Hide the answer

What is actually required

The pack needs to answer three questions on any given day: where personal data enters, where copies of it come to rest, and which of those crossings leave a trust boundary. The current artefact answers them as of a date, with no mechanism to detect that the answer changed. A signed-off diagram is a claim about the past presented as a statement about the present.

What to generate rather than draw

  • The inventory of stores holding personal data, from the infrastructure definitions and the catalogue, not from memory. Anything holding data that is not in the diagram is the finding.
  • The flows the pipelines actually run, from lineage metadata in the warehouse or the orchestration tool. This is where the diagram is usually wrong, because an analyst added a derived table in March.
  • Egress crossings, from network policy and outbound allowlists. A diagram cannot show a third-party call that a service added last month; a policy audit can.
  • A diff against the signed diagram, run on a schedule, with the diff as the review artefact. The thing worth reviewing quarterly is the change, not the picture.

What to keep doing by hand

The trust boundaries and the purpose labels. No tool knows that the marketing warehouse is a different legal purpose from the fraud model, or that the vendor in eu-west-1 is a processor under a specific contract. Automation can enumerate the flows; only a person can say which ones are allowed and why, and that judgement is the actual content of a compliance pack.

Also keep the deliberately simplified overview for humans. The generated picture of every table is unreadable and unreviewable, which is a different failure from being out of date.

The one change that matters

Attach a date and a provenance line to every diagram: what it was generated from, when, and what the next refresh date is. Teams resist this because it looks like admitting the diagram is stale. A diagram with a date is trusted appropriately; one without is trusted wrongly, and the reviewer's question exists because nothing on the page told them which they were holding.

How to argue this in the review

Not as "our diagram is wrong", which invites a finding. As a control: the current control is a point-in-time attestation, and the proposed control is continuous detection of change with a quarterly human review of the differences. Auditors generally prefer the second, because it is evidence-producing, and it is also less work after the first month.

When this is the wrong change to make

A system with three services and one database does not need a lineage pipeline. The cost of generation is justified when the number of data stores exceeds what one person can hold, or when changes are made by people who never see the compliance pack. Below that, a dated diagram and a calendar reminder are proportionate and honest.