Prototyping AI Features Before the Model Is Ready

In the early 1980s J. F. Kelley at IBM wanted to build a natural-language calendar for business professionals who had never used a computer. Rather than guess what they would type, he sat in another room and answered their requests himself while they believed a program was responding. The first run's transcripts seeded a language processor; over fifteen further participants the dictionaries grew and he phased himself out of the loop. Tested unassisted on six new users, the finished system, built on a deliberately simple non-parsing algorithm, responded correctly to between 86 and 97 percent of requests depending on the criterion, a level Kelley argued the simple model could never have reached without the empirical loop (Kelley, 1984, An Iterative Design Methodology for User-Friendly Natural Language Office Information Applications, ACM TOIS 2(1)). He called the setup the OZ paradigm, and it is still the cheapest way to answer a question AI teams usually answer last: what will people actually ask for?

Scoping under capability uncertainty, the neighbouring concept, covers the afternoon-long feasibility check. Prototyping is broader. It separates three questions that a demo blurs together, and each has a different instrument.

Three questions, three instruments

Is it wanted? A Wizard-of-Oz study answers desirability with no model at all. A human produces the outputs behind a realistic interface, so the team observes the real input distribution, the phrasing users choose, the follow-ups they attempt and the errors they tolerate. The wizard's transcript becomes the first draft of the dataset.

Can a model do it, on these inputs? A model-in-the-loop prototype wires a current general-purpose model, prompted rather than trained, into the same interface. ProtoAI formalised this for designers: model outputs are placed directly into interface mock-ups and the design is evaluated across many inputs, so breakdowns are seen in the UI rather than in a spreadsheet (Subramonyam, Seifert & Adar, 2021, ProtoAI: Model-Informed Prototyping for AI-Powered Interfaces, IUI). The frontier model is a useful ceiling: if the strongest available model fails on the wizard's transcripts, a cheaper production model is unlikely to succeed without training.

How good is good enough? An eval set written before the build answers this, and it doubles as the specification. Each item pairs a realistic input with a pass criterion. The feature ships when the pass rate clears a threshold on that set, which makes "done" a number rather than a feeling.

Eval sets as specifications, and their arithmetic

A specification written as examples inherits sampling error. With \(n\) items and an observed pass rate \(\hat p\), the normal-approximation 95% interval is

At \(\hat p = 0.8\) and \(n = 50\) the half-width is \(1.96\sqrt{0.16/50} \approx 0.11\), so the true rate could plausibly be anywhere from 69% to 91%. At \(n = 400\) it narrows to about \(\pm 3.9\) points. A team that sets an 85% bar and measures 80% on fifty examples has not learned whether it failed. For small \(n\) or rates near 0 or 1, a Wilson interval behaves better than this approximation, but the budgeting lesson is the same: halving the interval width costs four times the items.

Where the eval-first discipline meets resistance is in the criteria themselves. Shankar et al. observed criteria drift: people need criteria to grade outputs, but grading outputs is how they discover the criteria, and some criteria only become visible once specific outputs exist (Shankar et al., 2024, Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences, arXiv:2404.12272). This cuts against the product advice to fix the quality bar before measuring. A workable reconciliation is to fix the threshold and the sampling frame in advance, which prevents moving goalposts, while allowing the rubric to be revised in versioned steps as outputs reveal failure types nobody anticipated.

When it breaks

The wizard is too good. A human wizard understands context, asks sensible clarifying questions and never hallucinates. Users calibrate their expectations and their phrasing to that competence, so a desirability result can describe a product no model will deliver. Constraining the wizard to templated responses, or injecting errors at a planned rate, trades realism of the experience for realism of the capability.

Prompted prototypes flatter the tail. A model-in-the-loop prototype is usually tried on the inputs the team thought of. The failures that dominate production sit in the long tail the prototype never saw, which is why the wizard's real transcripts, not invented examples, should feed it.

The eval set becomes the target. Once a pass rate gates a launch, prompts get tuned against those items, and the set stops estimating performance on new traffic. Holding back a portion that the builders never see, and refreshing it from production samples, keeps the specification honest.

Deception and scale have costs. Wizard-of-Oz studies mislead participants about what they are talking to, which calls for debriefing and often ethical review, and every session consumes a wizard's time. Kelley iterated with fifteen participants and validated with six: enough to see phrasing and failure tolerance, far too few to estimate rates to within a few points, as the interval arithmetic above makes plain.

Model change invalidates the ceiling. A prototype that proved feasibility on one model version says little about the next, so the eval set has to be rerun whenever the underlying model moves.