A team uses competing consumers to scale processing and discovers that ordering guarantees are broken. What happened, and what are the options?

What happened

Competing consumers deliberately discard ordering. Several workers pull from the same queue and process concurrently, so message two can complete before message one. That is the point — it is what provides the parallelism — and it is incompatible with any requirement that events for the same entity be applied in order.

The symptom is usually silent: a later update overwritten by an earlier one, an entity in a state it should never have reached, a balance that is wrong with no error anywhere.

The options

• Partition by entity key. Ordering holds within a partition and parallelism holds across partitions. The standard answer, and it requires the partition key to match the ordering requirement — per customer, per account, per document.
• Make the operations commutative or idempotent-with-versioning, so order does not matter. A last-writer- wins update carrying a version number can discard a stale message safely, which removes the ordering requirement rather than satisfying it.
• Serialise per key in the consumer with a lock or a per-key single-threaded executor, which preserves the shared queue and reintroduces coordination.
• Accept unordered processing where the requirement was assumed rather than real, which is more often the case than teams expect.

The mistake that recurs even after partitioning

A consumer that reads an ordered partition and hands messages to a thread pool has discarded the ordering it just paid for. This is the most common way ordering breaks in practice, and it is invisible in testing because it only manifests under load.

The transport's guarantee must be preserved by the consumer's concurrency model, which means per-key serialisation on the consumer side — the half of the design that is routinely omitted.

The scoping question that reduces the problem

Ordering demanded more broadly than the invariant requires is a scalability ceiling bought for nothing. Global ordering is almost never needed; per-entity ordering almost always is. Scope it to the smallest unit the business invariant requires, and note that consumers will quietly come to depend on any ordering you provide, which makes it hard to relax later.