Deep CTR Ranking Models: Wide & Deep, DCN and DLRM

The Criteo display-advertising dataset has 13 numerical features and 26 categorical ones, and the categorical ones include identifiers with millions of distinct values. The task is to predict whether an impression gets clicked. Almost all the signal is in combinations: this advertiser on this site at this hour. A model that cannot represent crosses of sparse features cannot do the job, and a model that enumerates them all by hand cannot be maintained.

These models live in the ranking stage, after two-tower retrieval has cut the catalogue to a few hundred candidates. Unlike the towers, they are allowed to cross user and item features freely, and that freedom is the whole point.

Wide & Deep: memorise and generalise

Wide & Deep jointly trains two components that feed one logistic output (Cheng et al., 2016, Wide & Deep Learning for Recommender Systems, arXiv:1606.07792):

Here \(\phi(x)\) is a set of hand-specified cross-product transformations, such as installed_app AND impression_app, which let a linear model memorise specific co-occurrences exactly. \(a^{(L)}\) is the final hidden layer of an MLP over embedded sparse features, which generalises to combinations never seen. The paper deployed it on Google Play, with over a billion active users and over a million apps, and reported significant gains in app acquisitions over wide-only and deep-only models online.

The weakness is visible in the formula: \(\phi\) is still feature engineering. Someone has to choose the crosses.

Deep & Cross: learn bounded-degree crosses

DCN replaces hand-picked crosses with a cross network that builds them explicitly, layer by layer (Wang et al., 2017, Deep & Cross Network for Ad Click Predictions, AdKDD, arXiv:1708.05123). With \(x_0\) the concatenated embeddings, DCN-V2 writes a cross layer as

where \(\odot\) is elementwise multiplication (Wang et al., 2021, DCN V2, WWW, arXiv:2008.13535). Each layer multiplies by \(x_0\) once more, so after \(l\) layers the network represents polynomial interactions up to degree \(l+1\), with the residual term keeping all lower degrees. Degree is controlled by depth rather than hoped for. DCN-V2 also factors \(W_l\) as a low-rank product to keep cost down on wide inputs.

DLRM: pairwise dot products and nothing more

DLRM takes a deliberately narrower view (Naumov et al., 2019, Deep Learning Recommendation Model for Personalization and Recommendation Systems, arXiv:1906.00091). Dense features pass through a bottom MLP to a vector of the embedding size; each categorical feature is looked up in its table. The interaction layer takes the dot product between every pair of these vectors, concatenates the results with the processed dense features, and passes them to a top MLP.

On Criteo that is 26 embeddings plus one dense vector, 27 vectors in all, giving \(27 \times 26 / 2 = 351\) pairwise interaction terms. Each is a scalar, so the interaction layer is small no matter how large the embeddings are. The design follows factorisation machines, and the authors argue explicitly that interactions beyond second order may not be worth their computational and memory cost. They also note a real difference from DCN: DLRM treats each embedding as one unit, while DCN crosses individual elements, including elements within the same feature vector.

The systems consequence dominates. Embedding tables hold the majority of DLRM's parameters, several tables each needing multiple gigabytes, so DLRM uses model parallelism for embeddings and data parallelism for the MLPs. In the paper's Criteo Kaggle comparison, DLRM and a DCN were sized to roughly 540 million parameters each with embedding dimension 16.

How much the architecture actually matters

Zhu and colleagues re-evaluated 24 CTR models in over 7,000 experiments consuming more than 12,000 GPU hours, and concluded that with sufficient hyperparameter search many deep models differ less than expected, and that real progress in CTR modelling is hard to demonstrate (Zhu et al., 2021, Open Benchmarking for Click-Through Rate Prediction, CIKM, arXiv:2009.05794).

This is the live disagreement. Architecture papers from large platforms, DCN-V2 among them, report offline and online gains in production. Independent benchmarks find public-dataset differences largely within tuning noise. Both can be true: an architecture that is marginal on Criteo can matter on a platform with thousands of features and the engineering to exploit it. What does not follow is that a new interaction layer will help a system whose tuning, features and data freshness have not been examined first.

When it breaks

Calibration matters more than ranking. In an ad auction the predicted probability is multiplied by a bid, so a model that ranks well but predicts 3% where the truth is 2% overpays systematically. AUC cannot see this; log loss and calibration plots can.

Embedding tables are a memory and freshness problem. Hashing identifiers into fixed-size tables causes collisions that silently merge unrelated IDs, and new IDs map to untrained rows until the next training run.

Labels carry position bias. Clicks depend on where an item was shown, so a model trained on logged clicks learns the old ranking's placement choices as preference unless position is modelled explicitly.

Tiny offline deltas are fragile. Gains in the third or fourth decimal of AUC are routinely reported. On non-stationary traffic they can vanish in an online test, which is where the decision belongs.