ByteDance's Monolith paper (ORSUM at RecSys 2022) describes a collisionless embedding table for a recommender trained on the interaction stream, where new user and item ids arrive continuously so the key space has no ceiling. What bounds the memory, and what does each bound cost in model quality?

The situation they were in

The conventional trick for a very large id space is to hash ids into a fixed-size embedding table and accept collisions. Two unrelated videos then share one row and their gradients pollute each other. At the scale of a short-video feed the long tail is most of the catalogue, so the collisions land precisely where recommendation quality is hardest to recover. Monolith rejects that trade: the table is collisionless, and every id admitted to it gets a row of its own rather than a share of one.

That decision creates the problem it solves for. A table with one row per id, fed by a stream of new ids, grows without limit and is trained online rather than rebuilt nightly, so nothing ever resets it.

What they chose

• A cuckoo-hash-based map as the table structure, so ids can be inserted and looked up without the fixed-modulus collisions of a hashed table.
• Frequency filtering — an id is only admitted after it has been observed enough times. The reasoning is that a row learned from one impression carries almost no signal and costs the same memory as a row learned from ten thousand.
• Expirable embeddings — rows not seen for a period are removed, so the table tracks the active id set rather than the accumulated history of everything the system has ever seen.
• Minute-level parameter synchronisation to the serving side, especially for the sparse parameters, which is what makes the freshness worth paying for at all.

Why it fits their constraints

The arithmetic explains the bounds. A 32-dimensional float embedding is about 128 bytes, so a billion live ids is roughly 128 GB before optimiser state, which commonly doubles or triples it. Those are my own order-of-magnitude figures, not the paper's, and they show why admission and expiry are not housekeeping: they are the difference between a table that fits on the parameter servers you have and one that does not.

What it costs

Both bounds trade tail coverage for memory. A frequency filter is a deliberate delay before the model can learn anything specific about a new item, which is worst for exactly the cold-start content a recommender most needs to evaluate. Expiry means a user who returns after a long absence is treated as new, discarding a history that existed. Neither failure raises an error; both show up as a quality metric drifting on a segment, which is why the threshold and the expiry period are model-quality parameters that happen to be implemented as memory settings.

Where copying it would be a mistake

If your id space is bounded and small — a catalogue of 50 thousand products, a few million customers — collisions and unbounded growth are not your problems, and a plain embedding table sized to the catalogue is correct. The collisionless design earns its complexity only when the id space is open-ended and you can measure quality loss on the tail. Copying the online training loop is the larger mistake: it removes the gate where a batch pipeline lets you evaluate a model artefact before it serves traffic, and that gate is worth more than minutes of freshness for almost every product that is not a feed.

Common weak answers

• "Use a bigger hash table." Collisions scale with the tail, not with the table size you chose; the tail keeps growing.
• "Evict by least-recently-used when memory is tight." That is expiry without a policy, and it makes the model's behaviour depend on memory pressure rather than on a decision anyone made.