Value of Information
also called Information Value, Analysis Budget, Cost of Deciding
The ceiling on what an evaluation can be worth - the amount by which its result could change the outcome - used to size a bake-off or a spike before anyone starts one.
A platform group spends five weeks comparing three message brokers - feature matrix, load test, scoring sheet - and concludes with the broker two of the three engineers already preferred. Nobody can say afterwards which measurement changed anyone's mind.
That evaluation cost something like 85k dollars: 7.5 engineer-weeks plus the teams that stalled waiting. Nobody asked at the start how much the answer could possibly be worth. Information has value only when it could change what you do, and its value is capped by the spread between the options - the framing of Howard's 1966 information-value paper.
Why it matters
Two failures are endemic without this frame. The first is the bake-off that costs more than the decision: three candidates whose three-year costs differ by 20k dollars, evaluated over six weeks. The second is the one-way door decided in an afternoon - a public field name, a stored identifier format, a tenancy model.
A third case feels productive and is not: the evaluation that cannot change the answer, because one candidate is already in production and the organisation will not operate two.
Implementation patterns
- Estimate the spread first, in money or in weeks - two candidates, three-year totals, operations fraction included - then spend under roughly 10% of it on deciding. A 300k spread justifies a few weeks; a 20k spread justifies an afternoon and a default.
- Name the one question the evaluation must answer, and the observation that would change the recommendation. If nobody can name the observation, there is no value to compute.
- Run a dominance screen first. Drop every candidate that no criterion favours; what remains genuinely trades against the others, and it is usually two.
- Attack the dominant assumption, not the feature list. Where the spread is driven by an operations fraction - 0.2 versus 0.5 of an engineer is a 75k-a-year swing - the cheapest information is a measurement of the system you already run. Time-box it as a spike with that one question.
Industry example
The idea is older than software and clearest outside it. In petroleum decision analysis, "should we drill an appraisal well before committing to field development?" is answered by computing the value of the information the well would produce: if the development decision would be the same whatever the well showed, the well is not worth drilling, however interesting its data. Howard's 1966 paper gave the method its name, and the same arithmetic is standard in clinical trial design. Software teams rediscover it as "the two-week spike with one question", and the discipline that makes a spike pay is the same: write down in advance which result would change the decision.
Failure scenarios
- The scoring sheet with weights. Numbers assigned to "developer experience" produce false precision, and the weights are argued because that is where the disagreement lives.
- Analysis paralysis with a deadline attached. The evaluation runs long, the delivery date does not move, and the implementation is rushed - the project pays twice.
- Perfect information about the wrong variable: throughput benchmarked to three figures while nobody measured the restore time.
Trade-offs
| Choose | Gains | Pays |
|---|---|---|
| Size the analysis from the spread | Decisions in days where the spread is small | Some decisions get less scrutiny than their advocates want |
| Evaluate deeply anyway | Confidence and a paper trail | Delay cost, which is frequently the largest line in the whole comparison |
The cost is political: telling a team its favourite comparison is not worth running is harder than letting it run, and the saving never appears in any number.
When not to use it
When the decision is irreversible, the spread stops being the ceiling. A data format customers write into, a public contract, a tenancy model: being wrong costs a migration, not a price difference, so set the budget from the migration cost. The test is one question - if this is wrong in eighteen months, do we change a config, a service, or everyone's stored data?
It is also wrong where the options are not comparable in money - two designs with different failure behaviour - and scenario-based comparison does that work instead.
Interview question
Q: A team asks for four weeks to evaluate three candidate databases for a new service. How do you decide whether to approve it, and what would you ask them to produce?
What a strong answer covers: estimating the three-year spread including operations time; a 10%-of-spread budget; naming the single question and the observation that would change the recommendation; screening dominated candidates out first; the delay cost as a line item; and the exception - a format written into customer data gets its budget from migration cost instead.
Quick check
Quiz: Two brokers differ by about 30k dollars over three years and a six-week evaluation by two engineers is proposed. Approve or not? No: at roughly 4.8k dollars an engineer-week that costs about 58k, twice the entire spread. Pick the one the team already operates and write the trigger that would reopen it.
Flashcard: What caps the value of any architectural evaluation? - The spread between the options: the amount by which its result could change the outcome. Spend under about 10% of that spread, and nothing when no result could change the choice.