A platform processes billions of events per day and requires ordering per customer but not globally. How should partition keys and consumer groups deliver that without creating a serialisation bottleneck?

Why the alternatives fail

A single partition gives total ordering and caps throughput at one consumer, which at billions of events per day is not a design, it is a queue. Sorting by timestamp at the consumer requires an unbounded buffer and a bound on lateness that a distributed producer cannot give you — clocks disagree and events arrive out of order by more than any window you would want to hold. A distributed lock per customer re-adds coordination that partitioning gives for free, and adds a lock service to your critical path.

Why partitioning by customer works

Ordering is only ever needed within the scope of the invariant. If the invariant is per customer, the partition key is the customer, and the guarantee comes from the transport rather than from application code. Parallelism is then bounded by partition count rather than by the ordering requirement.

What this decision costs you later

• Partition count is expensive to change. Increasing it rehashes keys, so events for one customer can land in two partitions during the transition and ordering breaks precisely when you are least prepared. Provision generously up front; over-partitioning costs some overhead, under-partitioning costs a migration.
• Hot keys break the model. One enormous customer sends more than one partition can absorb, and you cannot split them without losing their ordering. The options are a dedicated topic for that customer, a composite key that sub-partitions where per-key ordering is genuinely not needed, or accepting the ceiling.
• The consumer's concurrency model must preserve what the transport gave you. A consumer that reads a partition and hands events to a thread pool has discarded the ordering guarantee it just paid for. This is the most common way ordering breaks in practice, and it is invisible in testing because it only manifests under load.
• Rebalancing pauses consumption. Adding or losing a consumer reassigns partitions, and during the reassignment nothing progresses. At large scale, tuning this becomes a real operational concern.

The design principle

Scope ordering to the smallest unit the business invariant requires, and make that the partition key. Ordering demanded more broadly than the invariant is a scalability ceiling accepted for no benefit — and it is extremely difficult to relax later, because consumers will quietly have come to depend on it.