A feature needs personalised results in under 50 ms for millions of users. What is computed when?

What is being tested

Whether you decompose into what is precomputed and what is adjusted at request time, rather than choosing one extreme.

The answer: hybrid

Precompute the expensive base offline, and adjust cheaply at request time.

Concretely: a batch or streaming job produces a per-user candidate set — say the top few hundred items with scores — written to a key-value store shaped exactly for this lookup. At request time, read that set (single-digit milliseconds), then apply cheap adjustments: filter out what the user has already seen this session, boost by current session context, remove out-of-stock items.

That gets personalisation without per-request model inference, and it is the shape most systems that appear to require real-time computation actually converge on. Users cannot tell the difference between recommendations refreshed hourly and refreshed per request, provided the session context is applied.

Why not the extremes

Everything at request time: model inference within a 50 ms budget for millions of users is expensive and adds a failure mode to a user-facing path.

Everything precomputed: cannot reflect the current session, so the user sees an item they viewed thirty seconds ago.

What the serving layer needs

• A store shaped for the query, denormalised without apology.
• Rebuildability. The serving state must be regenerable from the authoritative source, so a pipeline bug is an inconvenience rather than a data-loss incident. Test the rebuild before you need it — one that has never been run takes four days the first time.
• Atomic switching. Build a new version alongside, verify, switch an alias. Instant rollback, which an in-place update does not give you.
• Lag monitored as an SLI. A stalled pipeline produces no errors: the store is healthy, the service is healthy, and the answers are silently old.
• A defined fallback — a default set, a cached previous version, or a simpler computation — decided by product in advance rather than by engineers at 3am.

The failure to avoid

The serving store becoming authoritative because something writes to it, at which point it can no longer be rebuilt and a projection bug becomes a data-recovery exercise.