Data Observability

Data systems fail differently from applications, and that difference is the entire justification for a separate discipline. An application failure is loud: errors, alerts, a page. A data failure is usually silent and successful — the pipeline ran, reported success, and produced wrong numbers, because an upstream system changed a unit from dollars to cents, or a join started dropping rows, or a source began sending nulls in a column that used to be populated.

The five signals that catch most of it: freshness (did the data arrive when expected), volume (is the row count within the expected range, catching both partial loads and accidental duplication), schema (did the structure change), distribution (have the values shifted in a way that suggests a semantic change), and lineage (what is downstream of this, so an incident can be scoped in minutes rather than by asking around).

Distribution monitoring is the one that finds the genuinely dangerous failures and the one that generates the most false alarms, since business reality legitimately shifts. It needs seasonality awareness and tuning, or it gets muted within a fortnight.

The value only materialises with ownership and routing. An anomaly detected and sent to a shared inbox is not detection. It needs to reach the team that owns the dataset, with the lineage attached so they can see who is affected.