Ingestion Pattern
The mechanism by which data leaves a source system, which determines freshness, source load and how faithfully change is captured.
Four mechanisms cover most of what exists, and the choice is usually forced by what the source permits rather than by preference.
Full extract copies everything each run. Simple, self-correcting, and untenable past a certain volume. Incremental extract by watermark selects rows changed since the last high-water mark, which is efficient and misses hard deletes and any update that does not touch the timestamp column — a very common silent data quality failure. Change data capture reads the transaction log and captures every insert, update and delete in order, with minimal load on the source; it is the most faithful option and the most operationally involved. Event push has the source publish changes deliberately, which is the cleanest contract and requires the source team to build it.
The question that exposes whether a pipeline is trustworthy: how are deletes handled? Watermark extraction typically ignores them entirely, so records deleted in the source live forever downstream, quietly inflating every count and violating retention commitments nobody has connected to the pipeline.