Design the event ingestion path for a recommendation platform receiving billions of interaction events daily, where the same events must feed real-time personalisation and offline model training. How should the pipeline be structured?

The organising decision

One ingestion path, two consumption paths. Events are written once to a durable, partitioned log, and both the online and offline worlds read from it. Duplicating ingestion produces two pipelines that disagree, and training on data that differs from what serving saw is the origin of training/serving skew — the most expensive class of bug in recommendation systems, because it degrades quality silently rather than failing.

The structure

Collection. Clients batch events and send them with retries; the edge writes to the log and acknowledges. Two properties matter here: events carry a client-generated event id so duplicates from retries can be collapsed downstream, and they carry both client time and server receipt time, because client clocks are wrong in ways that are systematic, not random.

The log. Partitioned by user id, which gives per-user ordering — sufficient for almost all personalisation logic — without a global ordering bottleneck. Retention long enough to replay a meaningful window for recovery and backfill.

The streaming path. Aggregates recent behaviour into a low-latency feature store: session context, recent interactions, short-window counters. Latency budget measured in seconds. This is what makes the next recommendation reflect the last few videos watched.

The batch path. The same log lands in object storage and feeds training, long-window features, label generation and evaluation. Latency measured in hours; correctness and completeness dominate.

The feature store as the seam. Both paths write features; serving reads only from the store. This is what keeps online and offline definitions aligned — a feature computed one way in a streaming job and another way in a training pipeline is skew waiting to happen, so the definition should exist once and be executed by both.

What is kept off the request path

At recommendation time, the budget is tens of milliseconds. That forbids computing anything from raw history. The request path does only: fetch precomputed user and item features, retrieve candidates from a precomputed index, score with a model small enough for the budget, apply filters.

Everything derived from billions of historical events is precomputed. The synchronous path is a lookup and a ranking, not a computation over history. This separation is the whole architecture.

The failure modes to design for

• Hot partitions. A viral item or a bot-driven user skews one partition. Requires monitoring per partition, not in aggregate, and a strategy — salting or dedicated handling — for the hot key.
• Late and out-of-order events. Mobile clients buffer offline and deliver hours later. Windowed aggregations need watermarks and a defined policy for what happens to data arriving after the window closed. "Discard silently" is a choice, and it should be a stated one.
• Duplicate events, handled by idempotent aggregation on the event id rather than by attempting exactly-once delivery.
• Backfill contention. A reprocessing job replaying months of history will starve the real-time path if they share capacity. Separate clusters or strict resource isolation — this is not optional at this scale.
• Schema evolution. Events outlive the code that produced them. A registry with compatibility rules is required, because a breaking change to an event that has ten consumers and two years of retained history is not a deploy, it is a project.