A grocery marketplace in Instacart's mould ships substitutions. The functional spec is complete and correct - when an item is unavailable the shopper is offered ranked alternatives and the customer approves or declines. Six weeks after launch, support volume is up and the finance team reports a rise in refunds. Error rates, latency and availability are all green. What failed, and which design decision allowed it?
Show the full answer Hide the answer
The trigger
Nothing broke. The system is doing exactly what the specification says, against data that is older than the decision it is being used for. Store inventory is read from a feed that updates on a cadence — minutes at best, hours in the tail for some retailers — and the substitution ranking treats that feed as current.
The sequence: a customer places an order; the picker arrives and the item is gone; the substitution engine ranks alternatives from the same stale feed; the top alternative is also gone; the picker improvises or marks the line unavailable. The customer gets something they did not choose, or nothing, and asks for money back. Every step returns 200.
Why it propagated
The functional requirement was complete and the non-functional requirement was absent. Nobody wrote down how fresh an availability read must be for a substitution decision to be worth making. Without that number, every component made a locally reasonable choice: the feed ingestion prioritised throughput over latency, the ranking service cached for an hour to protect the feed, and the client cached the ranking for the session. Three independent caches, each defensible, compose into a staleness nobody owns.
The amplification is in the ranking itself. A ranked list derived from stale data concentrates error on the items most likely to be wrong: the popular alternatives are the ones that sell out, so the top-ranked substitute has a higher probability of being unavailable than a random item. Ranking on stale data is worse than not ranking.
Why detection lagged
Because the failure is a business outcome, not a technical signal. The available metrics measured whether the substitution service answered, not whether the answer was true. The symptom surfaced in refunds and support tickets, which are reported weekly and owned by a different function, so the feedback loop was six weeks rather than six minutes.
The structural fix versus the tempting local fix
The tempting fix is to shorten the caches. It helps and does not address the mechanism: the underlying feed is still minutes-to-hours old for some retailers, and the system has no way to say so.
The structural fix is to make staleness a first-class value and let it change behaviour. Carry the observation timestamp with every availability record, propagate it through ranking, and define one number: the maximum age at which a substitution suggestion is offered. Past that age the product behaves differently — no ranked suggestion, ask the customer to pre-approve a category, or let the picker choose and confirm by message. Different retailers get different thresholds because their feeds differ.
Then measure the thing that actually failed: substitution acceptance rate and the share of substitutions that end in a refund, segmented by the age of the data the suggestion was based on. That curve tells you where the threshold belongs, and it is the only honest way to set it.
The general lesson
A functional specification can be complete, implemented correctly, tested fully, and still produce a defect, because it says nothing about the conditions under which the behaviour is valid. The missing requirement here is a freshness bound, and the class is wider: freshness, consistency window, accuracy, and the behaviour when a bound is exceeded.
The practical rule: any feature that reads state owned by another system needs a stated maximum age and a defined behaviour past it. If nobody will name the number, the feature's correctness is undefined, and the first report will arrive from finance rather than from monitoring.
When this is over-engineering
When the state barely moves. A feed of store opening hours is stale by days and nothing breaks, because the underlying fact changes a few times a year. Propagating timestamps through every layer costs schema changes in 4 or 5 services and buys nothing there. Spend the freshness machinery where the underlying fact changes faster than the read cadence — inventory, price, seat availability, driver position — and leave slow-moving reference data cached hard and simple.