Scoping Under Capability Uncertainty
Why you cannot specify an AI feature the way you specify software, the cheap experiments that resolve the uncertainty, and the scoping decisions that determine whether a feature is buildable at all.
A conventional feature can be specified before it is built: the behaviour is known, and the work is implementing it. An AI feature cannot, because whether the model can do the task at an acceptable rate is unknown until it is tried, and the answer depends on the exact framing of the task. Product planning has to accommodate a dependency that ordinary planning does not have.
Resolve capability before committing
The cheapest thing available is a manual prototype. Take twenty representative inputs, run them through a model by hand with a reasonable prompt, and look at the outputs. This costs an afternoon and answers the question everything else depends on: is this in reach, near reach, or not.
The failures it catches are the expensive ones. A task the model cannot do at any acceptable rate is discovered before a quarter is committed. A task it does easily is discovered before an elaborate pipeline is designed for it.
The important discipline is that the twenty inputs must be representative, including the awkward ones. A prototype run on clean examples answers a question about clean examples, and real traffic is not that.
Scope by narrowing until it works
The variable that most often makes an infeasible feature feasible is the scope of the task rather than the quality of the model.
Narrow the input distribution: one document type rather than any. Narrow the output: a structured extraction rather than free prose. Narrow the decision: suggest rather than decide, so a human absorbs the error rate. Narrow the population: one segment where the data is cleanest.
Each narrowing reduces the error rate and the consequence of an error simultaneously, and shipping a narrow feature that works beats shipping a broad one that does not, because the narrow version generates the usage data that tells you where to widen.
Define acceptable before measuring
The quality bar has to be set before the evaluation, or it will be set by whatever the evaluation produced. Setting it requires answering what the alternative is, since a system replacing a manual process competes with human error rates rather than with perfection, and what an error costs, which is where the human-in-the-loop decision is actually made.
When it breaks
Demo quality is not product quality. A curated demonstration establishes possibility. The distance from there to a reliable feature is usually most of the work, and it is where roadmaps built on demos go wrong.
The long tail is where the effort goes. Getting to 80 percent is fast and the remaining 20 percent is most of the project, because it is composed of many distinct rare cases rather than one general weakness. Planning as though the last stretch resembles the first is the standard estimation error.
Model improvements arrive from outside. A capability out of reach today may be routine in six months, so a decision not to build is a decision for now. Re-testing infeasible ideas periodically is cheap and is rarely done systematically.
Prompt-level iteration hides architectural problems. A feature failing because retrieval is returning the wrong documents will not be fixed by any prompt, and teams can spend weeks on the wrong layer. Diagnosing which component is failing before iterating is what avoids that.
12 flashcards for this concept
Click a card to reveal the answer.