A quarterly board report shows a category down 40%. Investigation finds an upstream system stopped sending a field three months ago. Nothing alerted. What do you change?

Understand why nothing fired

The pipeline completed successfully every night. It read the source, applied its transformation, and wrote rows — all of which is what success means to an orchestrator. The field was null, the CASE statement had a fallthrough, and the records landed in "other".

Data pipelines fail silently far more often than they error, and monitoring built around task exit codes cannot see it.

Immediate: the four signals

Add monitoring on the data rather than on the jobs:

• Freshness — did each table update when expected?
• Volume — is the row count within its normal range for this day of week?
• Schema — did columns appear, disappear or change type?
• Distribution — did a column's null rate, cardinality or mean shift materially?

The last one would have caught this on the first night: a field going from five percent null to one hundred percent is unmissable.

Baselines must respect seasonality — Monday is not Sunday, month-end is not mid-month — or the alerting is noise and gets muted, which is how the second incident happens.

Structural: stop defaulting unknown values

The CASE fallthrough turned a detectable error into a silent misclassification. Assert on the known set and fail on an unrecognised or missing value. A loud failure at 02:00 is cheaper than a wrong board report at quarter end.

The same applies to reference data generally: a new status value arriving from a source is an interface change, and it should stop the pipeline rather than be quietly bucketed.

Structural: a contract with the producer

The underlying failure is that an upstream team changed behaviour with no obligation to anyone. A data contract binds them: schema stability within a compatibility mode, stated semantics, completeness guarantee, notice before a breaking change — with schema and enum checks in their build, so it blocks their merge rather than reporting after the fact.

That is a negotiation, and the incident is the moment it is winnable.

Structural: reconciliation on the numbers that matter

For figures that reach a board, a periodic reconciliation against an independent source — the operational system's own count — catches the class of error that survives everything above. Monthly is usually enough, and it is what turns a three-month silent failure into a one-month one.

What to report

Say plainly that the pipeline could not have detected this, name the three controls being added, and give the detection latency each provides. Avoid attributing it to the upstream team as a failing — they had no contract, so they broke nothing. The contract is the fix, and framing it that way is what gets their cooperation.