Real-Time Serving Layer
The read path that answers queries in milliseconds from precomputed or continuously updated state — where the design question is what to compute when.
Definition
A serving layer answers user-facing queries with low latency, using state maintained by an upstream pipeline rather than computed at request time.
The central decision: what is computed when
| When | Latency | Freshness | Flexibility |
|---|---|---|---|
| Precomputed in batch | Lowest | Hours | Fixed queries only |
| Precomputed by a stream | Very low | Seconds | Fixed queries only |
| Computed at request time | Higher | Current | Any query |
| Hybrid: precomputed base plus request-time delta | Low | Current | Bounded |
Most systems that appear to require real-time computation are better served by precomputation with a small request-time adjustment. A recommendation refreshed hourly and re-ranked at request time with the current session's context is far cheaper than computing it entirely per request, and users cannot tell the difference.
What a serving layer needs
- A key-value or indexed store shaped exactly for the queries served, denormalised without apology.
- Rebuildability. The serving state must be regenerable from the authoritative source, so a bug in the pipeline is an inconvenience rather than a data-loss incident. Test the rebuild before you need it.
- Atomic switching. Build a new version alongside, verify, then switch an alias — giving instant rollback, which an in-place update does not.
- Staleness exposed, so consumers can reason about it and the product can display it honestly.
- Lag monitored as an SLI, because a stalled pipeline produces no errors: the store is healthy, the service is healthy, and the answers are silently old.
The fallback question
What is served when the serving layer is unavailable or stale? A default, a cached previous version, a simpler computation, or an explicit error. That decision is a product one and must be made in advance — serving four-day-old data silently is the failure that embarrasses people.
Failure scenarios
- The serving store becomes authoritative because something writes to it, so it can no longer be rebuilt.
- Rebuild never tested, so the one time it is needed it takes four days.
- Lag unmonitored.
- Real-time computation where precomputation was indistinguishable to users and far cheaper.
- No fallback, so pipeline problems become user-facing outages.
Interview question
"A feature needs personalised results in under 50 ms for millions of users. What is computed when?"