Prompt Format Sensitivity

Take a task prompt, change Question: to Q:, swap a colon for a newline, and re-run the benchmark. Nothing about the meaning has changed, and on LLaMA-2-13B the spread across plausible formatting variants of the same task reached 76 accuracy points (Sclar et al., 2024, Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design, arXiv:2310.11324). That is larger than the gap between most model generations. Any evaluation that reports one number for one format is reporting a draw from a distribution it did not characterise.

Three distinct sensitivities

Surface form. Separators, capitalisation, whitespace, and field names. The Sclar et al. work searches this space systematically with FormatSpread, which samples plausible formats under a grammar and reports the resulting performance interval instead of a point estimate. The finding that matters for practice: format quality does not transfer well between models, so a format tuned on one model tells you little about another.

Example order. With the same few-shot examples in a different order, accuracy on some tasks ranges from near state-of-the-art to near chance, and the good permutations do not transfer across model sizes (Lu et al., 2021, Fantastically Ordered Prompts and Where to Find Them, arXiv:2104.08786). Their fix uses the model itself to generate a probing set and selects orderings by an entropy statistic over predicted label distributions, requiring no extra labelled data.

Option position. For multiple-choice questions, models carry a prior over the option identifiers themselves, preferring to answer "A" or "C" independently of content. Permuting the options changes the answer often enough that a model can appear to know a fact under one arrangement and not another, an effect traced to token bias over the ID tokens rather than to reasoning (Zheng et al., 2023, Large Language Models Are Not Robust Multiple Choice Selectors, arXiv:2309.03882).

Why it happens

Nothing in pretraining or instruction tuning enforces invariance to these transformations. The model has seen Q:/A: transcripts, markdown, XML tags, and JSON, each in different corpora with different downstream statistics, so different surface forms genuinely activate different learned priors. Chat templates make it worse rather than better: the same user text embedded in two different templates produces two different token sequences, and a model fine-tuned on one template is out of distribution under the other.

Instruction tuning reduces the spread but does not remove it, and neither does scale within the range tested. This is not a bug awaiting a fix; it is a property of models trained by likelihood on text where format correlates with content.

Working with it rather than against it

• Report intervals. For any evaluation used to make a decision, sample several plausible formats and report the range, or at minimum fix and publish the exact format.
• Debias multiple choice. Evaluate under a few option permutations and average, or use a label-free debiasing correction. Cyclic permutation of options is cheap and catches most of the position effect.
• Fix the template, then optimise inside it. Chasing separator variants by hand is a poor use of time; freezing the template and letting an automatic optimiser search instructions and demonstrations is a better one.
• Regression-test the format. Treat the exact prompt string, template version, and model version as one artefact. A silent chat-template change on a provider's side is indistinguishable, from your metrics' point of view, from a model regression.

When it breaks

• Format search overfits. Picking the best of forty formats on a 100-example dev set buys you a number, not a capability. The gain frequently vanishes on held-out data.
• Sensitivity confounds every comparison. Two papers reporting different numbers for the same model on the same benchmark are usually reporting different formats, not different findings. This is a large part of why leaderboard replication fails.
• Averaging hides the variance you care about. A model with a mean of 70 and a range of 40 to 85 is not equivalent to one with a mean of 70 and a range of 68 to 72. In production you get one format, and you want the one whose worst case you can live with.