A marketplace runs a flash sale of the kind Alibaba reported for Singles' Day 2020, where peak order creation reached 583000 per second. At 00:00:00 the sale opens. By 00:00:08 the inventory read model lags the authoritative reservation store by 40 seconds. By 00:00:30 the product page is still showing stock for an item that sold out at 00:00:06, and 11000 orders are accepted for it. What failed, and which design decision allowed it?

The trigger

Demand arrived faster than the projection that feeds the read path. Nothing crashed. The event pipeline was doing exactly what it was built to do, at a lag that was acceptable at normal volume and catastrophic at sale volume.

Why it propagated

The design put the authoritative reservation in one place and the stock display in another, fed asynchronously. At steady state the lag is tens of milliseconds and the display is effectively correct. Under a demand spike the projection becomes the slowest component in the loop, and its lag is unbounded, because the input rate is set by customers and the output rate by the consumer's throughput.

Then the feedback loop closes. A page showing stock generates purchase attempts. Purchase attempts generate more events. The read model's lag increases the traffic that increases its lag, which is why 8 seconds of lag becomes 40 and not 9.

The amplifier is the accept path. If order acceptance checks the read model rather than the reservation store, every stale read becomes an accepted order for stock that does not exist. 11,000 of them is not a bug in the inventory service; it is 11,000 correct responses to a question asked of the wrong system.

Why detection lagged

The signals that were watched — error rate, latency, CPU — were all healthy. The signal that mattered was consumer lag, measured in time rather than in message count, and the alert on it was either absent or set on queue depth, which says nothing about how old the oldest unprocessed event is.

The structural fix versus the tempting local fix

The tempting fix is to speed up the projection: more consumers, more partitions, a faster store. It raises the volume at which this happens and leaves the mechanism intact.

The structural fix is to make the decision point authoritative and the display advisory.

• Reserve synchronously against a single authority per item — a counter with an atomic decrement, sharded by item, or a pre-allocated token pool for the sale. The reservation either succeeds or it does not, and nothing downstream can accept an order without one.
• Pre-allocate the sale inventory into tokens before the sale opens. This is the documented approach to extreme flash sales: convert the stock into a fixed set of claims, distribute them across shards or edge locations, and let the claim succeed or fail locally. It converts a consistency problem into a counting problem, which is the only thing that holds at hundreds of thousands of operations per second.
• Make the page say what it can honestly say. "Limited stock" rather than a number; or admit a queue and show a position. A number that can be 40 seconds old should not be displayed as a number.
• Shed before you lie. When projection lag crosses a threshold, degrade the page to the advisory form automatically. That threshold is a product decision made in advance, not an incident call.

How you would notice next time

Age of the oldest unprocessed event, alerted at a threshold set by the product's tolerance — for a flash sale, single-digit seconds. Plus one reconciliation metric: accepted orders minus successful reservations, which should be zero and is the direct measurement of overselling.

The general lesson

An asynchronously maintained view may inform a user and must never authorise a transaction. The boundary between advisory and authoritative is the most important line in an event-driven design, and it is usually drawn implicitly by whoever wrote the first order-acceptance check.

When this is over-engineering

A sale of 200 units to a few thousand customers does not need token pre-allocation or edge-local claims. One row with an atomic decrement handles thousands of operations per second and is trivially correct. The machinery above is justified by the rate, and the honest decision rule is that pre-allocation earns its place when a single authoritative row cannot serve the peak.