Prompts as Programs: What Changes When You Optimise the Prompt Instead of the Weights

The standard mental model says that if a language model is not good enough at your task, you fine-tune it. The 2025 result that complicates this model is GEPA, a prompt optimiser that mutates instructions using natural-language reflection over execution traces and never updates a single weight. Against GRPO, a reinforcement learning method that does update weights, it reported an average advantage of roughly 6 points, gains up to 20 points on individual tasks, and, most importantly, reached those results with as much as 35 times fewer rollouts (Agrawal et al., 2025, GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning, arXiv:2507.19457).

The interesting claim there is not that prompting beats training. It is about information per rollout. An RL step extracts a scalar reward from a rollout and pushes it through a policy gradient. A reflective prompt optimiser reads the entire trace, in language, and can conclude "the retrieval step returned the right document and the answer step ignored the date field." One of those signals is several bits wide and the other is a paragraph.

Why this matters: Most production LLM systems are multi-stage programs with three to ten prompts, each hand-written by a different engineer at a different time, none with a held-out test set. Prompt optimisation turns that into an ordinary machine learning problem with a training set, a metric, and an overfitting risk you can measure. It also imports every failure mode of ordinary machine learning, which is the part teams discover late.

TL;DR

• Prompt optimisation reframes the prompt as a parameter and the pipeline as a program, then searches instruction and demonstration space against a metric on a training set. There is no gradient, so every method substitutes something: sampled scores, natural-language critique, or a trajectory of past prompts with their scores.
• OPRO, which simply shows an LLM its own history of prompts and scores and asks for a better one, reported gains of up to 8 points on GSM8K and up to 50 points on Big-Bench Hard tasks over human-written prompts (Yang et al., 2023, arXiv:2309.03409).
• MIPRO optimises instructions and demonstrations jointly across all stages of a multi-stage program, using mini-batch evaluation plus a surrogate model of the objective, and beat baseline optimisers on five of seven programs by as much as 13 points with Llama-3-8B (Opsahl-Ong et al., 2024, arXiv:2406.11695).
• Demonstrations frequently matter more than instructions, which inverts how most humans allocate their prompt-writing effort.
• The dominant failure is overfitting to a small development set. Gains of 10 points on a 50-example dev split routinely become 1 point on held-out data, and the optimiser's reported score is training accuracy by construction.
• The metric is the specification. Optimise against an LLM judge with a length bias and the search will find verbosity, exactly as reward hacking does in RL.
• Optimised prompts do not transfer cleanly across models. Treat a compiled prompt as a binary built for one target and recompile on version bumps.

At a Glance

flowchart LR
  P["Program with N prompts"] --> E["Evaluate on train split"]
  E --> S["Score plus execution traces"]
  S --> R["Proposer: LLM writes candidates"]
  R --> Sel["Selection under a rollout budget"]
  Sel --> P
  Sel --> T["Held-out test: the only honest number"]
  classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
  classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
  class P blue
  class E,S,R purple
  class Sel amber
  class T emerald

The loop is unremarkable as machine learning goes. What is unusual is that the parameter is a string, the proposer is itself a language model, and every evaluation costs an API call, which makes the rollout budget the binding constraint on the whole design.

[IMAGE: Side-by-side comparison of two optimisation loops. Left: RL fine-tuning, showing a rollout collapsing into a scalar reward then a gradient. Right: prompt optimisation, showing a rollout expanding into a full trace then a natural-language critique then an edited instruction. Caption: "Same rollout, different bandwidth of feedback."]

Before the Prompt Was a Parameter

The idea that prompts could be searched rather than written predates instruction-tuned chat models. AutoPrompt used gradient-guided search over discrete tokens to build cloze-style prompts for masked language models, showing that automatically discovered prompts elicited more accurate factual knowledge on LAMA than manually written ones, and that the resulting prompts were largely unreadable (Shin et al., 2020, arXiv:2010.15980). Prefix-tuning then dropped the requirement that the prompt consist of real tokens at all, optimising a continuous prefix of virtual tokens with roughly 0.1% of the parameters of full fine-tuning (Li and Liang, 2021, arXiv:2101.00190).

Both approaches need gradients and therefore model weights, which excluded the closed-weight models that most teams were actually deploying. The move that unlocked the field was to give up gradients and use the model itself as the proposer. APE had an LLM infer candidate instructions from input-output demonstrations, scored them by execution accuracy on held-out data, and resampled around the best ones (Zhou et al., 2022, arXiv:2211.01910). Everything since is a refinement of the proposer, the credit signal, or the selection rule.

timeline
    title From written prompts to compiled programs
    2020 : AutoPrompt searches discrete tokens with gradients
         : Discovered prompts beat manual ones and are unreadable
    2021 : Prefix-tuning optimises continuous virtual tokens with 0.1 percent of parameters
    2022 : APE uses an LLM to propose instructions, scored by execution accuracy
    2023 : ProTeGi turns failures into textual gradients with beam search and bandit selection
         : OPRO feeds the model its own prompt-score history and asks for a better prompt
         : DSPy makes the program, not the prompt, the unit of optimisation
    2024 : MIPRO optimises instructions and demonstrations jointly across stages
         : TextGrad backpropagates natural-language feedback through arbitrary graphs
    2025 : GEPA evolves a Pareto front of prompts by reflecting on execution traces

[IMAGE: Timeline strip where each entry shows the artefact that method produces: a string of nonsense tokens for AutoPrompt, a vector block for prefix-tuning, a readable instruction for APE, and a structured multi-module program for DSPy.]

How Prompt Optimisation Actually Works

The problem, stated honestly

You are solving

where \(\Phi_\pi\) is your program under prompt configuration \(\pi\), \(m\) is your metric, and \(\Pi\) is the space of instruction strings and demonstration sets for every module. Three properties make this hard in a specific way. The space is discrete and combinatorial: with \(k\) modules and \(c\) candidates each, there are \(c^k\) configurations. Each evaluation is expensive, on the order of one LLM call per module per example. And the objective is stochastic, because the model is sampled and the metric is often itself a model.

That combination rules out exhaustive search and rules out anything requiring many thousands of full evaluations. Every practical method is an answer to "how do I spend a few thousand rollouts well."

Where candidates come from

The naive proposer asks an LLM to write a better instruction. The effective proposers ground the request in context that the model would otherwise have to guess: a summary of the training data's characteristics, the program's own source code and control flow, the current instruction, and the traces of examples the program got wrong. MIPRO's contribution is largely here, in program-aware and data-aware proposal, plus a meta-optimisation step where the LLM refines how it constructs proposals over time.

There is a second, cheaper source of candidates that is easy to overlook: the training data itself. Bootstrapped demonstrations are traces the program produced on training inputs that passed the metric, promoted to few-shot examples. Empirically, searching over which demonstrations to include often gives a larger gain than rewriting the instruction, because a demonstration specifies format, style, and edge-case handling simultaneously and unambiguously, while an instruction only describes them.

Credit assignment without a gradient

With a scalar metric and a multi-module program, the classic credit-assignment problem appears immediately: the answer was wrong, but which prompt caused it?

ProTeGi's answer is to ask. Feed a minibatch of failures to an LLM with the current instruction and request a critique; treat that critique as a natural-language gradient; apply an edit in the "opposite semantic direction"; explore with beam search and allocate evaluation budget with a bandit algorithm (Pryzant et al., 2023, arXiv:2305.03495). TextGrad generalises this into a full backpropagation analogue where each node of an arbitrary computation graph receives textual feedback from its successors (Yuksekgonul et al., 2024, arXiv:2406.07496).

GEPA pushes on the richness of the signal. Rather than only the final score, it reflects on the full execution trace, including intermediate module outputs and tool results, and mutates the prompt of the module the reflection implicates. It also maintains a Pareto front across tasks rather than a single incumbent, which prevents the search from collapsing onto a prompt that is excellent on one task family and mediocre everywhere else.

[IMAGE: Two-module program drawn as a graph with the metric at the output, showing a natural-language critique propagating backwards: the answer module receives "ignored the date field in passage 2" and the rewriter receives "query dropped the year". Caption: "Textual gradients are credit assignment carried in language rather than in a scalar."]

Selection when every evaluation costs money

Suppose 30 candidate configurations and 200 training examples. Fully evaluating everything is 6,000 program executions, and with two modules that is 12,000 LLM calls before you have learned anything.

The standard economisation is a two-stage procedure. Score every candidate on a small random minibatch, fit a cheap surrogate model that predicts full-set performance from minibatch performance and configuration features, then spend the remaining budget fully evaluating only the candidates the surrogate ranks highest. This is Bayesian optimisation with a noisy, cheap proxy, and it is the core of MIPRO's efficiency. The failure mode is that minibatch noise at \(n = 25\) has a standard error of roughly \(\sqrt{p(1-p)/25} \approx 0.10\) for \(p = 0.6\), so differences under about 10 points between candidates are not distinguishable at that sample size, and the surrogate will confidently rank noise.

Seeing It in Motion

One optimisation round, with the actors that matter:

sequenceDiagram
  participant O as Optimiser
  participant Pr as Program under test
  participant M as Metric or judge
  participant L as Proposer LLM
  O->>Pr: run current prompts on a minibatch
  Pr->>M: outputs plus traces
  M-->>O: scores per example
  O->>L: failures, traces, current instruction, data summary
  L-->>O: k candidate instructions and demo sets
  O->>Pr: evaluate candidates on the minibatch
  Note over O: surrogate ranks candidates; only the top few get a full evaluation
  O->>O: update incumbent or Pareto front

Where the signal comes from, compared with weight-based methods:

flowchart TB
  subgraph Weights["Weight updates"]
    W1["Rollout"] --> W2["Scalar reward"]
    W2 --> W3["Policy gradient"]
    W3 --> W4["Millions of parameters change"]
  end
  subgraph Prompts["Prompt updates"]
    P1["Rollout"] --> P2["Full trace in language"]
    P2 --> P3["Reflection names the failing module"]
    P3 --> P4["One instruction string changes"]
  end
  W4 --> R["Better task performance"]
  P4 --> R
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
  classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
  class W1,W2,W3,W4 purple
  class P1,P2,P3,P4 teal
  class R emerald

[IMAGE: Scatter plot of candidate prompts, x-axis minibatch score at n=25, y-axis full dev-set score, with a wide error band showing how many rank inversions occur below a 10-point gap. Caption: "Minibatch ranking is only trustworthy above the noise floor."]

Watch It Run

By the Numbers

Sources: AutoPrompt, Prefix-tuning, APE, ProTeGi, OPRO, MIPRO, GEPA. Every figure is the authors' own, on their own task suites and base models, and none has been independently replicated across the others' benchmarks. Cross-row comparison is not meaningful; within-row comparison against that paper's baseline is.

A Concrete Example

A two-module retrieval program: a rewriter turns the user's question into a search query, and an answerer produces a response from the retrieved passages. Metric is answer F1 against a reference. You have 200 training examples and 200 held-out test examples that the optimiser will never see.

Step 1, baseline. Hand-written instructions score 0.52 F1 on the training split.

Step 2, propose. The optimiser generates 8 instruction candidates per module, grounded in the program source, a data summary, and traces of 20 failures. Joint space: \(8 \times 8 = 64\) configurations, plus demonstration sets.

Step 3, price the grid. A full evaluation of one configuration is 200 examples × 2 LLM calls = 400 calls. Evaluating all 64 is 25,600 calls. At roughly 1,200 tokens per call that is about 31 million tokens, which is a real budget line rather than an experiment.

Step 4, minibatch instead. Sample 30 of the 64 configurations and score each on a 25-example minibatch: \(30 \times 25 \times 2 = 1{,}500\) calls. Minibatch scores range 0.44 to 0.68. Note the noise floor: at \(n = 25\) and \(p \approx 0.6\), one standard error is about 0.10, so the difference between the 0.68 candidate and the 0.61 candidate is inside the noise, and the surrogate should be treated as a filter rather than a ranking.

Step 5, promote and fully evaluate. Take the top 5 by surrogate score and run full training-set evaluations: \(5 \times 400 = 2{,}000\) calls. Best configuration scores 0.63 on the training split. Total spend: about 3,500 calls against 25,600 for the grid, roughly a 7× saving.

Step 6, the number that counts. Run the winner once on the untouched 200-example test set: 0.57 F1. The honest improvement is +5 points over the 0.52 baseline, not the +11 the training split advertised. Six of the eleven points were dev-set overfitting, and this is the typical ratio rather than a pathological case.

[IMAGE: Funnel chart of the worked example's budget: 64 candidate configurations at 25,600 calls if fully evaluated, narrowing to 30 configurations on minibatches at 1,500 calls, then 5 full evaluations at 2,000 calls, then 1 winner, with the 7x total saving annotated. Caption: "Every efficient prompt optimiser is an evaluation-allocation strategy."]

Step 7, look at what won. The winning rewriter instruction turns out to be short, and the winning answerer configuration is mostly a set of four bootstrapped demonstrations rather than a cleverer instruction. That is the common outcome, and it should change where you spend your own effort.

Where It Breaks

The metric becomes the specification

If your metric is an LLM judge, the optimiser will find the judge's biases faster than it finds task competence. Length bias produces verbose prompts; position bias in pairwise judging produces prompts tuned to the presentation order; a judge that rewards confident phrasing produces prompts that suppress hedging even when hedging is correct. This is reward hacking with a different search algorithm, and the mitigations are the same: judge against rubrics with explicit criteria, include adversarial examples in the training set, and periodically check a sample of optimiser-preferred outputs by hand.

Dev-set overfitting is the default outcome

An optimiser performs hundreds of evaluations against the same split, which is hundreds of chances to encode its idiosyncrasies. The worked example above lost more than half its apparent gain on held-out data with a 200-example split; with a 50-example split the loss is usually worse. Maintain three splits, not two: one the optimiser trains on, one it uses for selection, and one nobody looks at until the decision is made.

[IMAGE: Paired bar chart of training-split gain versus held-out gain for several optimisation runs at different dev-set sizes (50, 200, 1000 examples), showing the gap narrowing as the split grows. Caption: "The reported gain is training accuracy. The gap is the size of the split."]

Prompts do not transfer across models

A compiled prompt encodes the target model's quirks: its formatting preferences, its default verbosity, which instructions it needs repeated. Moving it to a different model, or the same model one version later, is not a no-op, and the failure is usually silent degradation rather than an error. The operational answer is to treat compilation as part of the build: pin the model version, store the compiled prompts as artefacts, and recompile when the version changes.

The artefact becomes unmaintainable

Discovered instructions accumulate strange emphasis, redundant restatements, and task jargon. AutoPrompt made this vivid by producing prompts that were effective and unreadable, and modern optimisers produce a milder version of the same thing. It matters because the system prompt is often also where policy lives, and an instruction a human cannot read is an instruction a human cannot review. Keep the human-authored policy block separate from the optimised task block.

It cannot manufacture a missing capability

Prompt search reallocates behaviour the model already has. If the model cannot do multi-digit arithmetic reliably, no instruction fixes that; a tool call does. The diagnostic is cheap: if a strong human-written prompt and a weak one produce similar errors of the same kind, the ceiling is capability, and optimisation will find at most a couple of points.

Cost and non-determinism compound

A MIPRO-style run is thousands of calls, and re-running it does not reproduce the same prompt because the proposer is sampled. That means an optimisation run is not a reproducible build step unless you fix seeds where you can and store the resulting artefact, which is the only part that is genuinely reproducible.

Alternative Designs

The row that deserves the most scrutiny is the last one. GEPA's comparison against GRPO is the strongest published argument that prompt search should be tried before RL, and it is one paper's own comparison on its own tasks. The defensible reading as of mid-2026 is a sequencing claim rather than a dominance claim: prompt optimisation is cheap enough that running it first is nearly free relative to an RL run, and if it closes the gap, the RL run was unnecessary.

How It Is Used in Practice

The teams getting value from this are not the ones running an optimiser once. They are the ones who built the evaluation set first, because everything here is downstream of having a metric worth maximising. In practice that means fifty to a few hundred labelled examples drawn from production traffic, a metric that correlates with what users complain about, and a split discipline that survives contact with a deadline.

The pattern that has settled in production is compilation as a build step. Prompts live in version control as templates. A compile job runs the optimiser against pinned model versions and a frozen training split, and it emits compiled prompt artefacts that are reviewed and deployed like any other build output. The evaluation harness runs on every change, and a model version bump triggers a recompile rather than a manual audit. This is unglamorous and it is the difference between a technique that works in a notebook and one that survives.

Where it does not fit: single-prompt applications with fuzzy success criteria, anything where the metric would have to be a human, and systems whose prompt encodes negotiated policy that legal or safety teams own. In that last case the optimiser is not allowed to edit the part that matters, and the honest scope is the task block only.

Insights Worth Remembering

1. The bandwidth of the feedback signal, not the presence of weight updates, is what determines sample efficiency. A scalar reward per rollout is a lossy summary of a trace that contained the diagnosis. That is the mechanism behind the reflective-optimiser results, and it predicts where they will and will not generalise.

2. Demonstrations often beat instructions. A bootstrapped example pins format, tone, and edge-case handling simultaneously, where an instruction only describes them. Most humans spend their effort on the instruction because that is the part that feels like writing.

3. The reported gain is training accuracy until proven otherwise. Optimisers do hundreds of passes over the same split. Any number quoted without a held-out set is an upper bound on what you will see in production, typically by a factor of two.

4. Your metric is your specification, including the parts you did not intend. This is the same lesson RL taught, arriving through a different door, and it arrives with the advantage that the resulting artefact is readable, so at least the hack is visible.

5. A compiled prompt is a build artefact, not source. Pin the model, store the output, recompile on version changes, and keep the human-owned policy text separate from the machine-owned task text.

6. Search cost is dominated by evaluation, not proposal. Proposing candidates is a handful of calls; evaluating them is thousands. Every efficient method in this literature is an evaluation-allocation strategy wearing a different name.

7. Prompt optimisation cannot create capability. It reallocates behaviour the model already has. When a strong and a weak prompt fail the same way, stop optimising and add a tool, retrieve better context, or change models.

Open Questions

Does the prompt-versus-RL result hold outside its own benchmarks? GEPA's comparison against GRPO is measured and specific to its task suite and base models. Whether the ordering survives on tasks with dense verifiable rewards and long horizons, where RL's advantages are strongest, is not established by independent replication.

How much of the gain is instruction versus demonstration? Joint optimisers report combined results. The ablations that exist suggest demonstrations carry a large share, but the split is task-dependent and there is no general account of when instruction search is worth its cost.

Can optimised prompts be made transferable? Nothing currently produces a prompt with a stability guarantee across model versions. A method that optimises for robustness across a family of models, rather than peak performance on one, would change the operational picture substantially, and it is unclear whether that costs much peak accuracy.

What is the right regularisation for prompt search? In weight space there are decades of theory for controlling capacity. In prompt space the analogous knobs, prompt length limits, candidate diversity, Pareto fronts, are chosen heuristically. Whether there is a principled complexity measure for a prompt is open.

Do these methods survive non-stationary judges? When the metric is itself an LLM, and that judge is upgraded, every previously compiled prompt was optimised against a slightly different objective. Nobody has published a clean account of how much accumulated drift this introduces in long-lived systems.

Sources and Further Reading

1. Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., & Singh, S. (2020). "AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts." EMNLP 2020. arXiv:2010.15980
2. Li, X. L., & Liang, P. (2021). "Prefix-Tuning: Optimizing Continuous Prompts for Generation." ACL 2021. arXiv:2101.00190
3. Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., & Ba, J. (2022). "Large Language Models Are Human-Level Prompt Engineers." ICLR 2023. arXiv:2211.01910
4. Pryzant, R., Iter, D., Li, J., Lee, Y. T., Zhu, C., & Zeng, M. (2023). "Automatic Prompt Optimization with 'Gradient Descent' and Beam Search." EMNLP 2023. arXiv:2305.03495
5. Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., & Chen, X. (2023). "Large Language Models as Optimizers." ICLR 2024. arXiv:2309.03409
6. Khattab, O., et al. (2023). "DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines." arXiv:2310.03714
7. Opsahl-Ong, K., Ryan, M. J., Purtell, J., Broman, D., Potts, C., Zaharia, M., & Khattab, O. (2024). "Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs." EMNLP 2024. arXiv:2406.11695
8. Yuksekgonul, M., Bianchi, F., et al. (2024). "TextGrad: Automatic 'Differentiation' via Text." arXiv:2406.07496
9. Agrawal, L. A., et al. (2025). "GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning." arXiv:2507.19457
10. Sclar, M., Choi, Y., Tsvetkov, Y., & Suhr, A. (2024). "Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design." ICLR 2024. arXiv:2310.11324
11. Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2021). "Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity." ACL 2022. arXiv:2104.08786
12. Zheng, C., Zhou, H., Meng, F., Zhou, J., & Huang, M. (2023). "Large Language Models Are Not Robust Multiple Choice Selectors." ICLR 2024. arXiv:2309.03882
13. Wallace, E., Xiao, K., Leike, R., Weng, L., Heidecke, J., & Beutel, A. (2024). "The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions." arXiv:2404.13208
14. Liu, N. F., et al. (2023). "Lost in the Middle: How Language Models Use Long Contexts." TACL 2024. arXiv:2307.03172

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.