Class Imbalance, Resampling and Reweighting

A fraud model on transactions with a 0.2% fraud rate reaches 99.8% accuracy by predicting "not fraud" every time. That observation launched a large literature on rebalancing training data. The mistake it points at is real, but it is a mistake about the metric and the threshold, not about the data, and many remedies built on it trade away something valuable to fix a problem a change to the decision rule already fixes.

Decisions come from costs, not from 0.5

A classifier that outputs a calibrated \(\hat p(x) \approx P(y=1 \mid x)\) should predict positive when that has lower expected cost. With \(c_{\text{FP}}\) the cost of a false positive and \(c_{\text{FN}}\) of a false negative, the threshold is

If a missed fraud costs 100 times a false alarm, \(t^* = 1/101 \approx 0.0099\). Defaulting to 0.5 is what produces the "always negative" classifier; imbalance merely exposes the bad default. Elkan showed that changing the class proportions in training is equivalent to moving this threshold, and gave the proportion that achieves any cost ratio (Elkan, 2001, The Foundations of Cost-Sensitive Learning, IJCAI).

What resampling does to probabilities

Random undersampling discards majority examples; random oversampling duplicates minority ones; class weights scale each example's loss. For a probabilistic model all three change the effective prior. If the training positive rate is \(\pi'\) and the deployment rate is \(\pi\), predicted odds must be corrected by

(Saerens, Latinne and Decaestecker, 2002, Adjusting the Outputs of a Classifier to New a Priori Probabilities, Neural Computation 14(1)). Train on a 50/50 sample from a 1% population, and a predicted 0.5 corresponds to a true 1%, a predicted 0.9 to 8.3%, a predicted 0.99 to 50%.

The damage when nobody corrects is easy to measure. In a small simulation with a 2% base rate and logistic regression, the uncorrected model's mean predicted probability was 0.020 with a Brier score of 0.017. Balanced undersampling pushed the mean prediction to 0.29 and the Brier score to 0.14; class weighting did the same. Applying the prior correction to the undersampled model restored a mean of 0.023 and a Brier score of 0.020. Any use of the raw scores as probabilities, in expected-loss calculations, pricing or clinical risk communication, was wrong by more than an order of magnitude.

SMOTE and the evidence

SMOTE creates synthetic minority examples by interpolation between a minority example \(x_i\) and one of its \(k\) nearest minority neighbours \(x_{nn}\):

(Chawla, Bowyer, Hall and Kegelmeyer, 2002, SMOTE: Synthetic Minority Over-sampling Technique, JAIR 16). It is among the most cited techniques in applied machine learning, and the more careful recent evidence is unfavourable. A simulation study in clinical risk prediction found that undersampling, oversampling and SMOTE all produced strong overestimation of minority-class risk without better discrimination, and that shifting the threshold matched their sensitivity and specificity (van den Goorbergh, van Smeden, Timmerman and Van Calster, 2022, The harm of class imbalance corrections for risk prediction models, JAMIA 29(9)). A large tabular benchmark found balancing helped weak classifiers but not strong ones, once metrics and hyperparameter selection were handled properly (Elor and Averbuch-Elor, 2022, To SMOTE, or not to SMOTE?, arXiv:2201.08528).

The disagreement runs between communities. Much applied work still reports SMOTE improving F1 or recall at a fixed 0.5 threshold, which critics read as threshold movement in disguise. Defenders point to settings where rebalancing does help: weak or non-probabilistic learners, compute budgets where undersampling saves real money, and optimisation that stalls on very rare classes.

When it breaks

The real problem is usually too few positives, not the ratio. With 40 fraud cases, no resampling scheme creates information that is not in those 40, and interpolations between them can land inside majority-class regions.

Resampling before splitting leaks. SMOTE on the full dataset puts synthetic neighbours of test examples into training. Resampling belongs inside each training fold, as the target leakage concept on this track warns.

Metrics must respect the base rate. ROC AUC is insensitive to prevalence, which hides how many false positives an operating point generates. Precision-recall curves, whose chance baseline equals the prevalence, and expected cost at the chosen threshold sit closer to the decision.

Deployment prevalence moves. A model corrected for a 1% prior is miscalibrated when fraud rises to 3%. Monitoring predicted versus observed positive rates and re-applying the prior correction is cheaper than retraining.

Corrections stack. Class weights plus oversampling plus a lowered threshold compound into a far more aggressive classifier than any one intended. Pick one mechanism, preferably the threshold, and document it.