Customer 360 & Real-Time Risk Intelligence Platform · View 07 of 20

CDC and Historical Batch Ingestion

Log-based change capture from core banking systems, the initial historical snapshot, and how both land in bronze and merge into slowly changing silver dimensions.

Editable source SVG draw.io All views

Capture approach

Log-based CDC only — no query-based polling against production databases
Snapshot and stream share one topic, so history-to-CDC cut-over needs no manual switch
10 years of history loaded once, chunked and throttled to protect source systems

Correctness controls

Delta MERGE keyed on business key plus commit sequence makes reapplication idempotent
SCD Type 2 preserves point-in-time truth for risk and regulatory restatement
Row-count and control-total reconciliation against source before silver promotion

Late and out-of-order data

A watermark buffer holds late arrivals rather than discarding them
Restated records re-enter the merge and propagate to gold on the next build
Consumers of restated periods receive a notice through the catalog