Three engineers spend five weeks at roughly half their time comparing three message brokers for a 14-engineer platform group. Roughly what did that evaluation cost, what difference between the options would have justified it, and which assumption dominates the error?

The assumptions, stated

• Fully loaded cost of an engineer in a high-cost market: roughly 250k dollars a year, so about 4.8k dollars an engineer-week. Use your own number; the structure survives the substitution.
• Evaluation effort: 3 engineers x 5 weeks x 0.5 = 7.5 engineer-weeks.
• The decision it gates: a broker that four services will be built against over the next three years.

The arithmetic

Direct cost: 7.5 x 4.8k is about 36k dollars. That is the small term. The larger one is delay - five weeks during which the services waiting on the choice either stall or build against a placeholder they will rewrite. If two of the fourteen engineers are blocked, that is another 10 engineer-weeks, about 48k dollars, so the real bill is roughly 85k dollars, call it 50k to 120k.

Now price the spread the evaluation was trying to resolve. Managed broker at the group's volume: order of 15k dollars a year. Self-managed on its own nodes: cheaper in infrastructure and about 0.2 to 0.5 of an engineer in operations, which is 50k to 125k dollars a year. Over three years the two candidates differ by something like 150k to 350k dollars, plus a rework risk if the loser's limitations bite.

The number, and what it rules in

An evaluation is worth at most the amount by which it can change the outcome. That is the whole of information value as decision analysis has framed it since Howard's 1966 paper: information has value only when it could change what you do, and its value is capped by the spread between the options. Choose the length of the bake-off from that spread. The spread here is 150k-350k over three years, so an 85k evaluation is defensible but close to the ceiling - the usual working rule is spend under about 10% of the decision's spread on deciding it, which would have bought two weeks, not five.

The reverse case is the common one: three candidates whose three-year costs differ by 20k dollars, and a six-week bake-off. Then the analysis costs more than the entire decision, and the correct move is to pick the one the team already operates and move on. An evaluation that cannot change the answer - because the platform is standardised, or because one candidate is already running in production - has an information value of zero, and it fails in a particular way: it produces a document that justifies what was going to happen anyway, which is worse than no document because it looks like diligence.

Which assumption dominates the error

Not the licence price - the operations fraction. The 0.2-versus-0.5 engineer swing is worth 75k dollars a year on its own, more than every other line combined. So the evaluation that would have paid for itself is not a feature comparison at all: it is two weeks of measuring how much time the team currently loses to the datastore it already self-manages, which is the same quantity under a different name and is already in the on-call record.

When this is the wrong answer

When the decision is not reversible, the spread ceases to be the ceiling. A storage format that customer data is written into, or a public contract, can be wrong in ways no three-year cost model captures, and there the analysis budget is set by the cost of migration rather than by the price difference. The test is one question: if this choice turns out wrong in eighteen months, do we change a config, a service, or everyone's stored data?