Enterprise Metadata Management System  ·  View 10 of 22  ·  Runtime

Harvest and Conflict Resolution

One incremental run end to end, including the case where the source contradicts a steward.

Editable source SVG draw.io All views
Harvest Orchestrator
Harvest Orchestrator
Connector
Connector
Source System
Source System
Canonical Mapper
Canonical Mapper
Precedence Engine
Precedence Engine
Aspect Store
Aspect Store
Change Log
Change Log
1. run since watermark
1. run since watermark
2. read metadata only
2. read metadata only
3. schemas, stats, logs
3. schemas, stats, logs
4. 429 rate limited
4. 429 rate limited
5. backoff and resume
5. backoff and resume
6. raw payload
6. raw payload
7. map to URN + aspects
7. map to URN + aspects
8. change proposal
8. change proposal
9. read current version
9. read current version
10. curated description
10. curated description
11. curated outranks source
11. curated outranks source
12. write version + 1
12. write version + 1
13. conflict to steward
13. conflict to steward
14. change log event
14. change log event
15. advance watermark
15. advance watermark
Incremental Harvest and Conflict Resolution
Incremental Harvest and Conflict Resolution
A harvest never rewrites a curated field. When source and curation disagree the source value is retained as evidence and a task is raised.
A harvest never rewrites a curated field. When source and curation disagree the source value is retained as evidence and a task is raised.
v 1.0 · owner Platform Engineering · date 2026-08
v 1.0 · owner Platform Engineering · date 2026-08
Text is not SVG - cannot display

What this proves

  • The curated description survives the harvest. This is the single most important runtime behaviour in the system — every stewardship hour depends on it.
  • The watermark advances only after a successful commit, so a failed run re-reads rather than skips.
  • Rate limiting is handled by backoff inside the connector, not by failing the workflow; the orchestrator sees a slow run, not a broken one.

Numbers

  • Incremental harvest of a 50,000-table warehouse completes in under 20 minutes.
  • Peak sustained ingest 5,000 change proposals per second during a full-estate rebuild.

Assumption

  • Sources expose a reliable change marker — an information-schema timestamp, a system table or a log offset. Sources without one fall back to full harvest on the weekly schedule.