Model Extraction and Membership Inference

A model behind an API is not a black box in any meaningful sense. Query access alone permits reconstructing an approximation of the model, determining whether a specific record was in its training set, and in some cases recovering training data verbatim. These are distinct attacks with distinct defences and a shared root cause: the model's outputs carry more information than the answer they were asked for.

Extraction

Model extraction trains a substitute model on input-output pairs obtained by querying the target. With enough queries the substitute approximates the target's decision boundary well enough to be useful, either as a free replacement or as a white-box platform for crafting adversarial examples that transfer back.

Query efficiency depends on what the API returns. Full probability vectors carry far more information per query than a top-1 label, so returning only the label, or rounding probabilities, raises the attack cost substantially. Active learning strategies that query near the decision boundary reduce the number needed by a large factor over random querying.

For large language models the analogous concern is distillation: generating a corpus from the target and training a student on it, which is effective enough that provider terms of service prohibit it explicitly. Prohibition is not prevention.

Membership inference

Membership inference asks whether a particular record was in the training set. The basic mechanism is overfitting: models are more confident and have lower loss on examples they trained on than on comparable examples they did not. Thresholding the loss is the simplest attack and it works.

Stronger attacks calibrate against a reference. Likelihood-ratio approaches train shadow models with and without the target record and compare the target model's behaviour against both distributions, which is far more effective than a global threshold because it accounts for examples that are intrinsically easy or hard.

The privacy consequence depends entirely on context. Membership in a dataset of clinical trial participants is itself sensitive information regardless of what the record contains, which is why membership inference is treated as a privacy violation rather than a curiosity.

Attribute inference and extraction go further, recovering feature values or, for generative models, verbatim training sequences. Extraction of memorised sequences from language models is well documented and is most effective on rare, high-entropy strings, which is precisely the category that includes credentials and personal identifiers.

When it breaks

Defences cost utility. Returning only labels degrades every legitimate use that needs probabilities. Adding noise to outputs reduces calibration. Rate limiting slows real users. Each defence is a tax paid by legitimate traffic to raise the attacker's cost.

Differential privacy is the only defence with a guarantee, and its guarantee is a bound. DP training bounds the influence of any single example, which bounds membership inference success. The epsilon values used in practice often permit more leakage than the formalism's reputation implies, and the utility cost at strong epsilon is substantial.

Deduplication is the cheapest large win against memorisation. Sequences appearing many times in training are memorised far more readily, so aggressive deduplication reduces extractable content substantially at no utility cost, and it is the intervention with the best ratio of benefit to effort.

Detection is difficult. Extraction queries look like use. Distinguishing systematic boundary probing from a legitimate heavy user requires behavioural analysis, and the detection threshold trades false accusations against undetected extraction.