A job enriches order events with customer tier by looking up a table built from a compacted topic. After a routine deploy, 3% of orders in the first ten minutes come out with tier null, then the rate returns to zero. Throughput and error rate are flat. What is happening?

The first three things to look at, in order

1. The time distribution of the nulls, not their count. Clustered in the first minutes after a restart is a bootstrap signature; spread evenly is a data-quality problem.
2. Whether the null keys exist in the table now. Re-run the lookup for a sample of the affected order ids. If the tier is present now and was null then, nothing was missing, the table was not ready.
3. The job's startup sequence: when the first record was processed against when the table side reported it had reached the end of its topic.

The diagnosis

The table side of the join is populated by consuming a topic, and consuming a topic takes time, but the stream side starts immediately. For the first seconds or minutes after a restart, the job is joining live orders against a partially loaded table, so lookups for customers whose records sit late in the compacted log return nothing. When the load completes, the nulls stop. The bug is not in the data, it is a race between two consumers that the job treats as one operation.

Two details make it worse than it looks. The nulls are not random: they are biased towards customers whose record appears late in the log, which usually correlates with recency, so new and recently changed customers are the ones enriched wrongly. And the join emitted a result, so downstream cannot distinguish "this customer has no tier" from "we did not know yet".

The misleading signal

Throughput and error rate being flat is exactly what makes this hard: the job is healthy by every operational measure while producing wrong output. Consumer lag is also flat, because the stream side is keeping up. The only place the problem is visible is in the output, which is why a join hit-rate metric, emitted per window, is the instrument this class of job needs and almost never has.

The fix, in order

1. Gate the stream side until the table side reports bootstrap complete. Frameworks differ in how this is expressed; the requirement is that no stream record is processed before the table reaches its end offset at startup.
2. Distinguish "not found" from "not loaded". Emit to a side output, retry, or hold the record rather than writing a null that downstream reads as a fact.
3. Alert on hit rate, not on nulls in the sink. A drop from 100% to 97% for ten minutes should page before an analyst finds it a week later.

When this is not worth fixing

If the enriched field is advisory (a label on a dashboard) and the volume affected is a few minutes of traffic per deploy, gating startup costs more than the defect. The rule: gate when the null is written somewhere durable or acted upon, accept it when the output is transient and observed by humans.