advanced 2 min answer

The source data contains records that violate the target's constraints. How do you handle it and who decides?

data-migrationqualityprofilingbusiness-decisionsreconciliation
Show the full answer Hide the answer

What is being tested

Whether you profile early and route the decisions to the people who can make them.

The answer to "who decides"

The business, not the migration team. Whether a customer record with no valid address should be fixed, defaulted, rejected or quarantined is a business decision with commercial and regulatory consequences. Engineers making it silently is how migrations produce outcomes nobody agreed to.

The options per violation class

  • Fix, where the correct value is derivable.
  • Default, where a defensible default exists — recorded, so it is not mistaken for real data.
  • Reject and quarantine, migrating everything else and handling these separately. Usually the right answer for a small tail.
  • Relax the target constraint, if the source reflects a legitimate reality the target's model failed to anticipate. This possibility is frequently overlooked and is sometimes correct.
  • Migrate to a holding area for manual review.

Each needs a named owner and a recorded decision, because they will be questioned during reconciliation.

The timing that decides everything

Profile the source data early — at the start of the project, not during the cutover window. Discovering the quality problem in the window is the classic failure, because the window is then spent making decisions rather than executing.

Profiling means: null rates on mandatory fields, referential integrity checks, duplicate detection, value distributions against the target's permitted set, and format compliance.

The other hard parts

Semantic mapping. A status of 3 in the source means something the target expresses differently, and the mapping is frequently not one-to-one. Domain experts own these decisions.

History. How much to bring? Full history is expensive and often unnecessary; none makes reporting discontinuous. A common answer is full detail for a recent window and aggregates beyond — decided by the business.

Identifiers. New identifiers break external references and integrations; preserving old ones carries the source's constraints forward. A persistent mapping table is usually required.

Reconciliation

Counts by entity and segment, financial totals, and a field-by-field sample — signed off by the business owner, not by the migration team.