Label Noise and Learning Through It

No real dataset has clean labels. What matters is not the noise rate but its structure, because random noise and systematic noise damage a model in completely different ways and call for different responses.

Random versus systematic

Random noise flips labels independently of the input. It is the benign case: it slows learning and increases the data required, and with enough data the model converges toward the correct function, because the errors do not point anywhere. A model trained on symmetrically noisy labels can still be well calibrated after correction.

Systematic noise correlates with the input. Annotators consistently mislabel a particular kind of case, or one guideline ambiguity is resolved the same wrong way every time. The model learns the error as signal, because from its perspective there is nothing distinguishing it from the true pattern. No amount of additional data fixes it, and it is invisible in aggregate metrics because the test set carries the same bias.

The practical implication is that noise rate is close to useless as a summary statistic. A dataset at 5 percent systematic noise concentrated in one subpopulation is worse than one at 15 percent random noise.

Memorisation happens late

Deep networks fit clean, learnable structure first and memorise noisy examples afterwards. The training curve reflects this: validation loss improves while the model learns the pattern, then degrades as it begins memorising the exceptions that are actually errors.

This gives two practical handles. Early stopping is an effective noise defence, since stopping before memorisation begins avoids fitting the noise. And examples the model finds persistently hard, high loss late in training, are enriched for mislabelling, which turns the model into a detector for its own bad labels.

Finding mislabelled examples

High-loss ranking. Sort training examples by loss after partial training and inspect the top. Cheap, effective, and it surfaces genuinely hard examples alongside mislabelled ones, which is why inspection rather than automatic removal is the right response.

Cross-validated disagreement. Train on folds and flag examples the model confidently contradicts when they were held out. This is the basis of confident learning methods, and it estimates the label noise structure rather than only listing suspects.

Ensemble disagreement. Examples where independently trained models disagree are enriched for ambiguity, mislabelling, or genuine difficulty, and separating those three requires looking.

Influence-based methods identify training points whose removal would most change a prediction, which finds the harmful examples specifically rather than the merely unusual ones, at higher computational cost.

When it breaks

Removing hard examples degrades the model. The high-loss set contains both errors and the informative difficult cases, and deleting it wholesale removes exactly the examples that taught the decision boundary. Relabelling beats removing, and inspection beats both automatic options.

Test sets have the same noise. Cleaning the training set while evaluating on a noisy test set makes a better model look worse, because the improvement is measured against errors. A small, carefully adjudicated evaluation set is worth far more than a large noisy one and is the first thing to build.

Robust losses trade against learnability. Losses designed to bound the influence of any single example, symmetric cross-entropy and its relatives, resist noise and converge more slowly and can underfit hard clean examples. They are a tool for known-noisy settings rather than a default.

Noise interacts with subgroup performance. If noise concentrates in a subpopulation, which it does when a guideline handles that population poorly, the model's performance there is capped by the labels while the aggregate looks fine. Measuring agreement per subgroup is what surfaces it.