ByteDance's Monolith paper (RecSys workshop 2022) describes training recommendation models on user feedback as it arrives rather than in nightly batches, and states that system reliability was deliberately traded for real-time learning. What does that trade actually look like in the pipeline, and when is a nightly batch the better engineering decision?

Why freshness dominates in this workload

In a recommendation feed, the distribution of what users want shifts within hours: a new video, a trend, a news event. A model trained on yesterday's interactions scores today's candidates with yesterday's beliefs. The paper's framing is that the separation of batch training from serving is the structural problem, because it fixes the maximum speed at which the model can react to feedback at one training cycle.

Two mechanisms carry the design. Collisionless embedding tables: rather than hashing sparse feature values into a fixed-size table, where two distinct video ids can land in the same slot and share an embedding, entries are stored without collision, with expiry and frequency-filtering to keep memory bounded. And online training, where the training worker consumes the interaction stream continuously and pushes updated parameters to serving.

What "trading reliability for freshness" means concretely

In a batch pipeline, a bad training run is caught before it is served: the artefact is evaluated, compared to the incumbent, and promoted. In an online loop the model being served is being modified continuously, so there is no artefact to gate. The consequences are structural, not incidental:

• A bad hour of data is already in the model. Bot traffic, a logging bug, or a broken upstream feature is learned, not quarantined.
• Rollback means restoring parameter state, which requires frequent snapshots and a documented point to roll back to, rather than re-promoting yesterday's artefact.
• A feedback loop exists: the model's own recommendations generate the interactions it then learns from, so an error can be self-reinforcing rather than self-correcting.
• Failure is quality degradation, not an error. Nothing 500s. The signal is a metric moving a few percent, which is why online training is only viable with a continuously scored evaluation set and an automatic freeze.

What you need before this is a defensible choice

A streaming path for labels with bounded lag · snapshotting frequent enough that the worst rollback loses minutes · anomaly detection on the training stream itself, not just on serving metrics · and a kill switch that freezes parameters while leaving serving up.

When not to train online: the nightly batch case

When the thing being predicted moves slower than the training cycle. Credit risk, churn propensity, demand forecasting for a warehouse: the world does not shift meaningfully in an hour, and the gate you give up is worth more than the freshness you gain. It also wins whenever the model's output is subject to review or regulatory explanation, because "the model that scored this decision" must be a reproducible artefact, and a continuously mutating parameter set is not one without extra machinery.

The honest rule: online training is justified by the decay rate of the signal, not by the sophistication of the team. Measure how much a model trained on yesterday's data loses against one trained on the last hour. If that number is small, the nightly batch is the better system.