Where Bias Enters the Pipeline

"The data was biased" is a true statement that identifies nothing actionable, because disparity enters at several distinct points with distinct remedies. Locating which one is operating is the difference between a fix and a gesture.

The points

Historical bias. The world the data describes is unequal, so a perfectly collected, perfectly representative dataset encodes that inequality. A hiring model trained on accurate records of past hiring learns past preferences. No sampling or modelling fix addresses this, because the data is correct; the question is whether to reproduce what it records.

Representation bias. The sample under-represents a population, so the model has less signal about it and performs worse. This is the case that more or better-targeted data genuinely fixes, and it is why it is the mitigation everyone reaches for even when the problem is something else.

Measurement bias. The features or the label measure different things for different groups. Arrest is a proxy for crime whose relationship to the underlying quantity differs by policing intensity. Healthcare cost is a proxy for health need whose relationship differs by access to care, which is the mechanism behind a widely cited case where a care-management algorithm systematically under-referred Black patients because it predicted cost rather than illness. Fixing this requires changing what is measured, which is usually a product decision rather than a modelling one.

Aggregation bias. One model is fitted where the relationship between features and outcome genuinely differs across groups, so the model is optimal for the majority and wrong for the rest. The remedy is group-specific models or group-interacted features, and the constraint is that using the protected attribute as a feature may be legally restricted.

Learning bias. The objective and the architecture make choices with disparate effects. Minimising average loss allocates capacity to the majority; a compression method that maintains aggregate accuracy can degrade rare subgroups; regularisation that suppresses rarely-used features suppresses those that mattered for a minority.

Deployment bias. The system is used differently from how it was evaluated: a different population, a different decision threshold, a human who overrides in a patterned way, or a feedback loop where predictions shape the data the next model sees. Predictive policing directing patrols to areas that then generate more recorded incidents is the canonical example.

Matching remedy to stage

Reweighting and resampling address representation. Changing the target variable addresses measurement. Group-specific modelling addresses aggregation. Constrained optimisation and post-processing address learning. Monitoring, threshold review and human-decision auditing address deployment. Historical bias is addressed by deciding not to reproduce the pattern, which is a policy decision the pipeline cannot make.

When it breaks

One mitigation is applied to every diagnosis. Balancing the dataset is the reflex, and it does nothing for measurement, aggregation or deployment bias. The diagnosis has to come first, and it usually requires understanding how the data was generated rather than examining the data itself.

Fixing one stage moves disparity to another. A balanced dataset feeding a model deployed with one threshold across populations with different score distributions reintroduces the disparity at the decision.

The label is where the most damage is done and where least attention goes. Substituting a convenient proxy for the quantity of interest is the single most consequential design decision in most systems, and it is typically made early, casually, and without review.

Measurement requires the attribute. Every diagnosis above needs group membership to detect, which regimes discouraging its collection make harder. Separate, restricted collection for fairness assessment only is the arrangement that resolves it, and it needs a lawful basis and careful access control.