Many-Shot In-Context Learning

Few-shot prompting was named for a constraint, not a preference. With a 4k or 8k window, five examples were what fit, and an entire prompt-engineering folklore grew up around choosing those five well. Million-token windows removed the constraint, and the natural experiment became affordable: what happens at 100 examples, or 1,000? The systematic answer came from a DeepMind study across translation, summarisation, planning, mathematics, and classification: performance keeps improving well into the hundreds-of-shots regime, often substantially, on tasks where few-shot performance had looked saturated (Agarwal et al., 2024, Many-Shot In-Context Learning, arXiv:2404.11018).

What more shots actually buy

The gains are not uniform polish; they concentrate where a handful of examples cannot pin the task down. Low-resource machine translation improves because 1,000 sentence pairs sketch a bilingual dictionary that 5 pairs cannot. Classification with many labels, or with unusual label semantics, improves because the decision boundary finally has enough support. Most strikingly, many-shot prompts can override preferences baked in at pretraining: Agarwal et al. show many-shot ICL overcoming pretraining biases on sentiment tasks where few-shot prompts consistently failed to, and learning abstract input formats (linearised functions, symbol-remapped labels) that few-shot ICL cannot touch. In-context learning at this scale starts to behave less like instruction-following and more like a temporary fine-tune, assembled at inference and discarded afterwards.

When you run out of labelled examples

The obvious blocker is that tasks with thousands of high-quality worked examples are rare, and for reasoning tasks the expensive part is the rationale, not the answer. The same paper proposes two workarounds that have aged well. Reinforced ICL replaces human rationales with model-generated chains of thought, sampled and then filtered to keep only those reaching a correct final answer; the filtered traces become the shots. Unsupervised ICL drops rationales and answers entirely and fills the window with problems alone, betting that the examples' role is partly to locate the task in the model's ability space rather than to teach it. Both match or beat prompts built from human-written rationales on math and reasoning benchmarks in the many-shot regime, which is a strange and useful result: the value of a shot is not only its correctness but its power to specify the distribution.

The economics, and why caching changed them

A 500-shot prompt can run 50k to 500k tokens, and paying prefill on that for every query is absurd. What makes many-shot deployable is that the shot block is identical across queries, which is precisely the case prompt caching serves: the KV cache of the shared prefix is computed once and reused, so marginal cost falls to the query suffix (see prompt caching infrastructure). The design consequence runs the other way too: to benefit, the shot block must be byte-identical and sit before anything that varies, so many-shot prompts are built as frozen prefixes, not assembled per request. Against fine-tuning, the trade is capability you can version-control and revert instantly, at the price of latency and per-query cost that a weight update would amortise away.

When it breaks

• Gains are not monotone. On several tasks, accuracy peaks at some shot count and then declines as the prompt grows further; the optimal number is task- and model-specific and has to be swept, not assumed.
• It inherits every long-context pathology. Shots in the middle of a 200k-token prompt are subject to the same positional unevenness as any other content there (see context rot); a model that cannot use its window well cannot learn from it well.
• Next-token loss stops being a guide. Agarwal et al. found negative log-likelihood trends uncorrelated with downstream many-shot performance, so you cannot cheaply predict where the gains stop; you have to evaluate the task itself.
• Self-generated shots can entrench errors. Reinforced ICL filters on final-answer correctness, which admits traces that reach right answers through wrong reasoning; at hundreds of shots those systematic flaws become the in-context training distribution.
• The comparison point is a moving target. Where parameter-efficient fine-tuning is cheap and traffic is high, a LoRA adapter often beats a permanent 100k-token prefix on cost within days; many-shot wins when tasks change faster than you can retrain, or when you cannot touch the weights at all.