Bias in Generative Models

Fairness for a classifier is a well-posed measurement problem: there are groups, outcomes and rates. A generative model has no decision, no groups defined by the task, and an unbounded output space. Every metric described for classification is undefined, and the harms are real, so the measurement problem has to be reconstructed rather than transferred.

The harm categories

Representational harm. The model's outputs reinforce a stereotype or erase a group. Occupational images skewing to one demographic, a model completing a sentence about a nationality with a stereotype, or a translation system defaulting a gender-neutral pronoun to a gendered one in a profession-dependent way.

Quality of service disparity. The model works less well for some people. Higher error rates on dialects and accented speech, worse performance in lower-resource languages, worse handling of names outside a dominant convention. This is the closest analogue to classification disparity and the most directly measurable.

Allocative harm appears when generation feeds a decision: a resume screener, a triage assistant, a content moderation system. Here classification fairness metrics do apply, to the decision rather than to the model, which is the right place to measure.

Disparate refusal. The model declines legitimate requests more often for some topics, dialects or framings, which is a quality-of-service harm produced by the safety layer rather than the base model, and it is usually measured separately from either.

What evaluation looks like

Templated probes vary a single attribute across an otherwise identical prompt and compare outputs. Sentence completion benchmarks and their relatives work this way. They are systematic, they produce comparable numbers, and their coverage is limited to the templates someone wrote, which encodes the authors' assumptions about what to check.

Distributional analysis of generated content samples many outputs for a neutral prompt and examines the distribution of depicted attributes. "A photo of a doctor" generated a thousand times has a measurable demographic distribution, and comparing it against a reference, which requires deciding what the reference should be, is where the substantive judgement sits.

Paired quality evaluation measures the same task with inputs differing only in a group signal, comparing quality rather than content. This targets service disparity directly and generalises across tasks better than content probes.

Red-teaming with affected communities finds harms no template anticipated, and remains the only method that reliably surfaces new categories rather than new instances.

When it breaks

The reference distribution is a value judgement. Should generated doctors match the current demographic distribution of doctors, the population, or something else? Each answer encodes a position, and reporting a deviation from an unstated reference presents an ethical choice as a measurement.

Benchmarks are gamed by training on them. Once a bias benchmark is public it enters training corpora and models improve on it without a general change. Held-out probes are what measure the property.

Mitigations produce visible new failures. Interventions that adjust generated demographics have produced outputs that are historically or contextually wrong in ways that attracted substantial criticism. Blanket rewriting of prompts is a blunt instrument, and the failure mode is public.

English-centric evaluation misses most of the problem. Most bias evaluation exists in English, so the languages where quality-of-service disparity is largest are the ones with the least measurement. A model reporting good bias scores has usually reported them in the language where it works best.