The source data contains records that violate the target's constraints. How do you handle it and who decides?
Show the full answer Hide the answer
What is being tested
Whether you profile early and route the decisions to the people who can make them.
The answer to "who decides"
The business, not the migration team. Whether a customer record with no valid address should be fixed, defaulted, rejected or quarantined is a business decision with commercial and regulatory consequences. Engineers making it silently is how migrations produce outcomes nobody agreed to.
The options per violation class
- Fix, where the correct value is derivable.
- Default, where a defensible default exists — recorded, so it is not mistaken for real data.
- Reject and quarantine, migrating everything else and handling these separately. Usually the right answer for a small tail.
- Relax the target constraint, if the source reflects a legitimate reality the target's model failed to anticipate. This possibility is frequently overlooked and is sometimes correct.
- Migrate to a holding area for manual review.
Each needs a named owner and a recorded decision, because they will be questioned during reconciliation.
The timing that decides everything
Profile the source data early — at the start of the project, not during the cutover window. Discovering the quality problem in the window is the classic failure, because the window is then spent making decisions rather than executing.
Profiling means: null rates on mandatory fields, referential integrity checks, duplicate detection, value distributions against the target's permitted set, and format compliance.
The other hard parts
Semantic mapping. A status of 3 in the source means something the target expresses differently, and the mapping is frequently not one-to-one. Domain experts own these decisions.
History. How much to bring? Full history is expensive and often unnecessary; none makes reporting discontinuous. A common answer is full detail for a recent window and aggregates beyond — decided by the business.
Identifiers. New identifiers break external references and integrations; preserving old ones carries the source's constraints forward. A persistent mapping table is usually required.
Reconciliation
Counts by entity and segment, financial totals, and a field-by-field sample — signed off by the business owner, not by the migration team.