A marketplace's seller metrics look wrong but every pipeline succeeded and every test passed. How do you diagnose it?

Why green pipelines produce wrong numbers

Pipeline success measures whether the code ran, not whether the data is right. The dimensions that fail silently are:

• Completeness. A source delivered 60% of its records and the pipeline processed all of what arrived, successfully. Volume trending against expectation catches this; row-count-greater-than-zero does not.
• Timeliness. A time-based downstream dependency ran on partial upstream data, computing correctly from incomplete input.
• Accuracy. Values are well-formed and wrong — a unit change, a currency assumption, a sensor drift.
• Consistency. Two systems disagree, and each is internally coherent.
• Validity. A new enum value arrives and is silently bucketed as "other".
• Uniqueness. A join fanned out and duplicated rows, inflating every sum downstream.

The diagnostic order

1. Compare volumes to the historical distribution per source and per partition. Most silent failures are visible here first.
2. Check the join cardinality at each step — an unexpected fanout is the single most common cause of inflated aggregates.
3. Check the freshness of every input, not the job's completion time.
4. Reconcile against an independent source — an operational count, a finance figure — because internal consistency proves nothing.
5. Look at the most recent producer change, since silent semantic shifts almost always follow a deployment upstream.

The structural fix

Move from pipeline success monitoring to data monitoring: volume against expectation, distribution shift, null-rate change, cardinality change, freshness per input. These catch the failures that succeed.