Silent Coercion

When a source changes a column's type, or emits a value outside the expected range, an ingestion pipeline has three options: fail the load, quarantine the record, or coerce the value.

Coercion is the worst of the three and is frequently the default. The load succeeds, the pipeline is green, and the data is wrong downstream with nothing indicating it — a number truncated, a timestamp reinterpreted, a string parsed into a different value.

Why it matters

The failure has no signal. A failed load produces an alert; a quarantined record produces a queue someone reviews; a coerced value produces a correct-looking number that is wrong, and it propagates into every downstream aggregate, dashboard and decision.

It is also discovered late and expensively — usually by someone noticing that a total does not match an external figure, weeks after the change.

Implementation patterns

• Quarantine rather than coerce or fail. The record goes to a review path, the sync continues, and the quarantine has an owner and an alert threshold. A single malformed record must not fail the whole run, which is what the naive implementation does — and it must not silently pass either.
• Fail the load on a schema change that cannot be handled additively, since a type change or a removal is a decision rather than a data event.
• Record schema history per source, so a downstream consumer can see when a column's meaning changed.
• Alert on quarantine rate, since a rising rate is the earliest signal that a source has changed.
• Propagate the quarantine downstream conceptually: an aggregate computed over data with quarantined records should be marked as incomplete rather than presented as complete.

Industry example

Ingestion platforms such as Airbyte and Fivetran face this across hundreds of heterogeneous sources that change without notice. The connectors' behaviour on unexpected data is the property that distinguishes a reliable integration from a plausible one, and it is rarely what a connector comparison examines.

The same failure appears in any pipeline reading from an external party — a carrier feed, a supplier catalogue, a partner's file — and the resolution is identical.

Failure scenarios

• Coercion by default, producing silent corruption.
• Fail-the-whole-run on one bad record, which makes the pipeline fragile and encourages coercion as the remedy.
• Quarantine with no owner or alert, which is silent dropping.
• No schema history, so a downstream anomaly cannot be traced to a source change.
• Aggregates presented as complete while records were quarantined.

Trade-offs

Quarantining means the data is incomplete until the records are resolved, and for a pipeline feeding a time-sensitive dashboard that is a visible cost. Coercion keeps the numbers flowing.

The choice is between visibly incomplete and invisibly wrong, and for anything a decision is made on, the first is strictly better. The judgement is only about the alert threshold — how many quarantined records constitute a problem — which is a per-source decision rather than a global one.

Interview question

"A source changes a numeric column to a string containing formatted values. Describe what your pipeline does, and what your downstream dashboard shows the next morning."