Sequential Recommendation: SASRec and BERT4Rec

A user who has just bought a tent, a sleeping bag and a head torch is not described well by a static taste vector. Matrix factorisation averages those three purchases into one point in latent space and forgets the order. The next purchase is probably a camping stove, and the evidence for that lives in the sequence, not in the set.

Sequential recommendation models the history \(S_u = (s_1, s_2, \dots, s_n)\) as an ordered list of item IDs and predicts \(s_{n+1}\). It is the natural complement to matrix factorisation, which models long-run preference, and it usually sits in the ranking or retrieval stage of the funnel described in two-tower retrieval.

SASRec: a causal transformer over item IDs

SASRec applies a left-to-right transformer decoder to item sequences (Kang & McAuley, 2018, Self-Attentive Sequential Recommendation, ICDM, arXiv:1808.09781). Each position gets an item embedding from a table \(M \in \mathbb{R}^{|I| \times d}\) plus a learned position embedding, a causal mask stops position \(t\) from attending to anything later, and the output \(F_t \in \mathbb{R}^d\) scores every candidate item \(i\) by a dot product with that item's embedding:

Sharing \(M\) between input and output is a deliberate choice: an item's representation is the same whether it is being read or predicted. Training uses binary cross-entropy with one sampled negative \(j\) per position,

where \(\sigma\) is the logistic function and \(s_{t+1}\) is the item that actually came next. Every position in the sequence is a training example, so one user with 200 interactions yields 200 predictions per forward pass. Before SASRec, recurrent models such as GRU4Rec held this ground; the paper's argument was that attention can reach far back into the history while basing each prediction on only a few relevant items.

BERT4Rec: bidirectional attention with a cloze objective

BERT4Rec removed the causal mask and borrowed masked language modelling (Sun et al., 2019, BERT4Rec, CIKM, arXiv:1904.06690). A random fraction of items in the sequence is replaced by a [mask] token, and the model predicts each masked item from context on both sides, with a softmax cross-entropy over the full catalogue. At inference a mask is appended to the end of the history. The paper claimed consistent wins over SASRec on four datasets, and the natural reading was that seeing both sides of a masked item gives richer representations.

That reading did not survive replication.

The replicability debate

Petrov and Macdonald reviewed the published comparisons and found them inconsistent: BERT4Rec beat SASRec in some papers and lost in others. Running the original code with its defaults, they could not reproduce the original results. They could reproduce them by training up to 30 times longer than the default configuration, and their own Hugging Face-based implementation matched the reported numbers on 3 of 4 datasets with up to 95% less training time (Petrov & Macdonald, 2022, A Systematic Review and Replicability Study of BERT4Rec for Sequential Recommendation, RecSys, arXiv:2207.07483). BERT4Rec was genuinely strong, but only when trained to convergence, which many comparisons did not do.

The second blow targeted the explanation. The two models differ in two things at once: attention direction and loss. SASRec uses one sampled negative with binary cross-entropy; BERT4Rec uses a full softmax. Klenitskiy and Vasilev trained SASRec with BERT4Rec's full cross-entropy loss and found it significantly outperformed BERT4Rec in both quality and training speed (Klenitskiy & Vasilev, 2023, Turning Dross Into Gold Loss: is BERT4Rec really better than SASRec?, RecSys, doi:10.1145/3604915.3610644). Petrov and Macdonald reached a compatible conclusion from the other side: negative sampling makes the model overconfident about positives, and a generalised binary cross-entropy with more negatives closed the gap without a full softmax (Petrov & Macdonald, 2023, gSASRec, RecSys, arXiv:2308.07192).

The field has not fully settled which fix is preferable. Full cross-entropy is simple and strong; calibrated sampled losses scale further. What is settled is that "bidirectional beats causal" was a confound.

When it breaks

The full softmax does not fit at catalogue scale. Logits have shape batch by sequence length by catalogue size. At batch 128, sequence length 200 and one million items in float32, that is \(128 \times 200 \times 10^6 \times 4\) bytes, about 102 GB for the logit tensor alone. This is why sampled losses exist, and why the loss question is a systems question as much as a modelling one.

Truncation discards history. Sequences are capped, commonly at a few hundred items, so a heavy user's early behaviour is simply cut off. Long-term preference then has to come from another component.

Item IDs have no content. A new item has an untrained embedding row, so these models inherit the cold-start problem in full; see cold start and content-based hybrids.

Evaluation protocol decides the winner. The published comparisons mix leave-last-out splits, sampled ranking metrics and different training budgets. As offline evaluation and sampled metrics shows, each of those can reorder models on its own, and a result that does not state all three is not interpretable.

Repeat consumption confuses next-item prediction. In grocery or music, the correct next item is often one already in the history, and a model trained to predict novel items undervalues it. Whether to recommend repeats is a product decision that the loss silently makes for you.