After scaling consumers from 3 to 12, downstream data shows updates applied out of order. What happened?

What the interviewer is testing

Whether you understand exactly what ordering guarantee a log provides and what it depends on.

The mechanism

Ordering is guaranteed within a partition only. If records for the same entity are spread across partitions — because the producer used a random or round-robin key, or a key that is not the entity identifier — then different consumers process them concurrently and completion order is arbitrary.

With 3 consumers this was probably still happening; it was just less visible, because fewer parallel consumers meant less interleaving and the timing windows were narrower. Scaling to 12 widened them and made an existing latent defect manifest.

The distinguishing test: check the producer's partitioning key. If it is null, random, or something other than the entity being updated, that is the cause.

The fix

Partition by the entity key — customer id, order id, account id — so all records for one entity land in one partition and are processed in order by one consumer.

The consequences to accept: throughput per key is capped by a single consumer, and a hot key becomes a bottleneck. That is the price of ordering, and it is usually the right trade.

The alternative when ordering cannot be had

Make the consumer order-insensitive:

• Include a version or sequence number in each record and reject any update older than the current version — a conditional write on the target
• Send full state rather than deltas, so applying an older record is harmless if versioned
• Use commutative operations where the domain allows

What a strong answer adds

Noting that the third option in the question — increasing partition count — is a genuine and separate hazard worth knowing: it changes the hash mapping, so a key that went to partition 2 now goes to partition 7, and records for the same entity can be in flight in both simultaneously during the transition. Repartitioning a keyed topic needs a planned procedure, not a configuration change.

Common weak answers

Reducing consumers back to 3, which hides the defect. Adding sequencing downstream without fixing the partitioning.