Two-Tower Retrieval and Candidate Generation

Scoring every item for every request is impossible above a few thousand items. A model that reads user and item features jointly, which is the accurate thing to do, requires one forward pass per candidate. At a hundred million items and a 50-millisecond budget, that is not a matter of better hardware.

The two-tower architecture resolves this by forbidding interaction until the last step. A user tower maps the request context to a vector; an item tower maps item features to a vector in the same space; the score is their dot product or cosine similarity. Because the item tower never sees the user, every item embedding can be computed offline, indexed with an ANN structure, and queried in single-digit milliseconds.

That constraint is the entire design. Everything the architecture gives up, and everything it enables, follows from the towers not being allowed to talk.

The multi-stage funnel

Real systems layer stages with increasing cost per candidate and decreasing candidate count.

Retrieval produces a few hundred to a few thousand candidates from the full catalogue, usually by ANN search over two-tower embeddings, often unioned across several retrievers with different biases: one for personalised relevance, one for recency, one for popularity, one for the user's explicit follows.

Ranking scores those candidates with a much richer model that can cross user and item features, since there are now only hundreds of them. This is where the accuracy lives.

Re-ranking applies list-level concerns the pointwise ranker cannot see: diversity, deduplication, business rules, fatigue on recently shown items, and the composition of the whole page.

The stages have different objectives, and the common failure is optimising retrieval for the same metric as ranking. Retrieval's job is recall of anything the ranker might want; precision at that stage is nearly worthless because the ranker will reorder everything anyway.

Training details that decide quality

In-batch negatives treat other examples' positives in the minibatch as negatives for this user. It is efficient and it samples negatives proportionally to popularity, which biases the model against popular items. The standard correction subtracts \(\log Q(i)\), the estimated sampling probability, from the logit, and omitting it is a common and quiet quality bug.

Temperature on the softmax over in-batch negatives controls how much the loss concentrates on the hardest negatives, and it interacts strongly with batch size.

Batch size is a modelling parameter here, not just a throughput one, because it determines how many negatives each positive is contrasted against. Large batches are a substantive quality lever in two-tower training.

When it breaks

No feature crossing is a real accuracy ceiling. The model cannot represent "this user likes long videos in the evening" unless both facts are encoded within a single tower. Everything relational must be pushed into one side's features, and that is why the ranking stage exists rather than being a refinement.

The index goes stale. Item embeddings are precomputed, so retraining the item tower invalidates the entire index and requires a full re-embed and rebuild. New items are unreachable until the next build unless a separate fresh-item path exists, which is why most production systems run a distinct recency retriever alongside the learned one.

Retrieval failures are invisible downstream. If the right item is not in the candidate set, no ranker can recover it, and every metric is computed over the candidates that were retrieved. Measuring retrieval recall against a ground truth separately from end-to-end metrics is the only way to see this, and it is frequently not instrumented.

Training and serving distributions diverge. The model is trained on logged interactions produced by the previous system and serves a distribution it changes by serving. Some exploration traffic, and periodic retraining, are structural requirements rather than optional refinements.