Automatic Prompt Optimisation

Hand-tuning a prompt is hill climbing with a sample size of about three, no held-out set, and an evaluator who already knows the answer. It works until the pipeline has four stages, at which point the search space is the cross-product of four instructions and their few-shot sets, and human iteration stops being able to cover it. Automatic prompt optimisation replaces the human in that loop with a search procedure: a training set, a metric, a proposer that writes candidate prompts, and a selection rule.

The search has no gradients, so it borrows their vocabulary

There is no derivative of accuracy with respect to a string. Every method here substitutes something else for the gradient.

Sample and score. APE has an LLM propose instruction candidates from input-output demonstrations, scores each by execution accuracy on held-out data, and keeps the best, with an optional resampling step around strong candidates (Zhou et al., 2022, Large Language Models Are Human-Level Prompt Engineers, arXiv:2211.01910).

Textual gradients. ProTeGi feeds a minibatch of failures back to an LLM and asks what is wrong with the current instruction. The natural-language criticism plays the role of a gradient, the edit plays the role of a step in the opposite direction, and beam search plus a bandit allocation of evaluation budget handles the rest (Pryzant et al., 2023, arXiv:2305.03495). TextGrad generalises this into a full backpropagation-shaped framework over arbitrary computation graphs whose intermediate values are text (Yuksekgonul et al., 2024, arXiv:2406.07496).

Optimisation by prompting. OPRO gives the model a trajectory of past prompts with their scores and asks for a better one, iterating. On GSM8K, its discovered prompts beat human-written ones by up to 8 points, and on Big-Bench Hard tasks by up to 50 points (Yang et al., 2023, Large Language Models as Optimizers, arXiv:2309.03409).

Compile the whole program. DSPy reframes the unit of optimisation: you declare modules with typed signatures, and a compiler searches instructions and demonstrations jointly for every stage against your metric (Khattab et al., 2023, arXiv:2310.03714). MIPRO adds program-aware and data-aware instruction proposal, mini-batch evaluation with a surrogate model of the objective, and Bayesian selection over the joint space, reported to beat baseline optimisers on five of seven multi-stage programs by as much as 13 points with Llama-3-8B (Opsahl-Ong et al., 2024, arXiv:2406.11695).

Reflective evolution. GEPA maintains a Pareto front of prompts across tasks and mutates them using natural-language reflection on execution traces. Its reported comparison is against reinforcement learning, not just against other prompt optimisers: better than GRPO by about 6 points on average and up to 20 on individual tasks, using as much as 35× fewer rollouts, and more than 10 points ahead of MIPROv2 (Agrawal et al., 2025, arXiv:2507.19457).

Why this beats fine-tuning surprisingly often

A prompt edit is one API call and zero GPUs. It transfers across model upgrades with a re-run rather than a retrain, it stays legible so a human can audit what the system was told, and it applies to closed-weight models. When the failure is one of instruction rather than capability, and it usually is, searching instructions is the cheaper lever. The GEPA result sharpens the claim: on tasks where an RL run needs tens of thousands of rollouts, a few hundred reflective prompt evaluations reached better accuracy.

When it breaks

• The metric becomes the specification, including its flaws. Optimise against an LLM judge with a length bias and you will get a longer prompt that produces longer answers. Every reward-hacking failure from RL reappears here.
• Overfitting to a small dev set is the default outcome. Optimisers routinely gain 10 points on a 50-example training split and 1 point on held-out data. Hold out a genuine test set, and treat the optimiser's reported score as training accuracy.
• Optimised prompts are brittle across models. A prompt compiled against one model is tuned to that model's quirks. Re-run the optimiser on a version bump rather than assuming transfer.
• The optimised artefact is often ugly. Discovered instructions accumulate odd emphasis, redundant restatements, and task-specific jargon. They work and they are hard for a human to maintain, which matters when the prompt also encodes policy.
• Cost is real. A MIPRO-style run is hundreds to thousands of LLM calls. Budget it like a training job, and cache aggressively.