Model Extraction and Weight Stealing

The final layer maps a hidden state of dimension \(h\) to \(|V|\) logits, and \(h \ll |V|\). So every logit vector the model can emit lies in an \(h\)-dimensional subspace of \(\mathbb{R}^{|V|}\).

Collect logit vectors for many prompts, stack them, take the SVD: the count of non-negligible singular values is exactly \(h\). The left singular vectors then give the projection matrix up to an unknown rotation.

Carlini et al. (2024) recovered \(h = 1024\) for Ada and \(2048\) for Babbage, with the full matrices costing under $20.

Because a logit bias parameter lets you move any token into the top-k. Query repeatedly with different biases and you can reconstruct the full logit vector for a prompt, one slice at a time, up to an additive constant.

That is why several providers changed their APIs after disclosure, restricting logit bias or removing logprobs entirely.

The general rule: an API returning real-valued scores rather than discrete choices leaks linear-algebraic structure, at a rate set by the precision it returns.

Realistic: architectural facts (hidden dimension, vocabulary and tokeniser behaviour), the final projection layer up to symmetry, and task-specific behaviour via distillation on collected outputs.

Not realistic: full weight recovery. Carlini et al. recovered one layer, not a model; the query complexity for hundreds of billions of parameters from black-box access is absurd.

Membership and memorisation leakage are separate attack classes with their own literature; do not fold them into "model stealing".

Perplexity-based evaluation needs per-token scores. Constrained and structured decoding implementations need them to mask and renormalise. Calibration work and selective prediction pipelines that abstain below a confidence threshold need them. So does much interpretability tooling.

Adding noise instead is not free either: it raises an attacker's query count, but a determined attacker averages the noise away, while legitimate consumers of the scores get permanently degraded signal.

There is no configuration that leaks nothing and remains useful.

Because a model that emits its chain of thought supplies more than labels. The student receives the intermediate reasoning steps, which is dense supervision on how to reach the answer rather than just what the answer is.

That is the same signal that makes reasoning distillation efficient in the first place, and it is available to anyone who can pay for the queries.

It is one of the practical reasons several providers now summarise or hide reasoning traces rather than returning them verbatim.

The final layer maps a hidden state of dimension \(h\) to \(|V|\) logits, and \(h \ll |V|\). So every logit vector the model can emit lies in an \(h\)-dimensional subspace of \(\mathbb{R}^{|V|}\).

Collect logit vectors for many prompts, stack them, take the SVD: the count of non-negligible singular values is exactly \(h\). The left singular vectors then give the projection matrix up to an unknown rotation.

Carlini et al. (2024) recovered \(h = 1024\) for Ada and \(2048\) for Babbage, with the full matrices costing under $20.

Because a logit bias parameter lets you move any token into the top-k. Query repeatedly with different biases and you can reconstruct the full logit vector for a prompt, one slice at a time, up to an additive constant.

That is why several providers changed their APIs after disclosure, restricting logit bias or removing logprobs entirely.

The general rule: an API returning real-valued scores rather than discrete choices leaks linear-algebraic structure, at a rate set by the precision it returns.

Realistic: architectural facts (hidden dimension, vocabulary and tokeniser behaviour), the final projection layer up to symmetry, and task-specific behaviour via distillation on collected outputs.

Not realistic: full weight recovery. Carlini et al. recovered one layer, not a model; the query complexity for hundreds of billions of parameters from black-box access is absurd.

Membership and memorisation leakage are separate attack classes with their own literature; do not fold them into "model stealing".

Perplexity-based evaluation needs per-token scores. Constrained and structured decoding implementations need them to mask and renormalise. Calibration work and selective prediction pipelines that abstain below a confidence threshold need them. So does much interpretability tooling.

Adding noise instead is not free either: it raises an attacker's query count, but a determined attacker averages the noise away, while legitimate consumers of the scores get permanently degraded signal.

There is no configuration that leaks nothing and remains useful.

Because a model that emits its chain of thought supplies more than labels. The student receives the intermediate reasoning steps, which is dense supervision on how to reach the answer rather than just what the answer is.

That is the same signal that makes reasoning distillation efficient in the first place, and it is available to anyone who can pay for the queries.

It is one of the practical reasons several providers now summarise or hide reasoning traces rather than returning them verbatim.