Customer 360 & Real-Time Risk Intelligence Platform  ·  View 08 of 20

Source Onboarding Patterns

Four reusable templates — transactional events, database CDC, files and semi-structured payloads — so a new source is onboarded without touching existing pipelines.

Editable source SVG draw.io All views
Contract
Contract
Capture
Capture
Land in bronze
Land in bronze
Conform to silver
Conform to silver
Publish
Publish
Transactional events
Transactional events
Avro schema + SLA
Avro schema + SLA
Produce to Kafka
Self service topic
Produce to Kafka...
Streaming sink
Exactly once
Streaming sink...
Dedup + enrich
Dedup + enrich
Gold topic + mart
Gold topic + mart
Database CDC
Database CDC
Table & key registry
Table & key registry
Debezium connector
Log based
Debezium connector...
Change log table
Change log table
MERGE + SCD2
MERGE + SCD2
Dimension tables
Dimension tables
Files & partner feeds
Files & partner feeds
Layout + control totals
Layout + control totals
SFTP / API pull
Airflow sensor
SFTP / API pull...
Raw file snapshot
Raw file snapshot
Parse + standardise
Parse + standardise
Reference tables
Reference tables
Semi-structured
Semi-structured
JSON contract
Evolving fields
JSON contract...
Gateway ingest
Gateway ingest
Variant column
Schema on read
Variant column...
Flatten known paths
Flatten known paths
Interaction tables
Interaction tables
Source Onboarding Patterns by Data Class
Source Onboarding Patterns by Data Class
Interface / broker
Interface / broker
Queue / topic
Queue / topic
Application we own
Application we own
Data store
Data store
Each lane is an independent onboarding template; a new source reuses one without touching the others.
Each lane is an independent onboarding template; a new source reuses one without touching the others.
v 1.0 · owner Data Integration · date 2026-08
v 1.0 · owner Data Integration · date 2026-08Text is not SVG - cannot display

Self-service model

  • A team picks a lane, registers a contract, and runs their own onboarding
  • No central engineering ticket is required for a source that fits a template
  • Platform reviews the contract, not the pipeline implementation

Decoupling guarantee

  • Lanes share infrastructure but no code path; one source cannot break another
  • Each source has its own topic, its own bronze table and its own DAG
  • Failure of one source degrades its domain only, never the platform

Semi-structured handling

  • JSON lands in a variant column so unknown fields are never lost
  • Only the paths a consumer needs are flattened into typed silver columns
  • New fields appear in bronze immediately and in silver on demand
Solution Architecture v1.0 · Enterprise Data Architecture · 2026-0808-source-onboarding-patterns