Reference Call

A vendor's benchmark answers the question the vendor chose. A code review tells you about code. An analyst report tells you about a market. None of them predicts the thing that decides whether you regret a datastore, which is what happens at 2 a.m. in year two.

A reference call is the practice of getting that information from someone who has already lived it. Two qualifiers make it worth the effort: at your scale, because a product that is delightful at 200 GB is often a different product at 20 TB, and for over a year, because a year covers a major version upgrade, a capacity increase and at least one incident. Those three events are exactly what a sales cycle cannot show you.

Why it matters

Technology selection errors are expensive in a specific asymmetry: the cost of a wrong choice is paid in migration, and migration cost scales with how much data the thing holds and how many services depend on it. Evidence gathered before the commitment is therefore worth far more than evidence gathered after, and a one-hour call is the highest-yield hour available.

The second reason is that the failure modes that matter are not measurable in a week. Operability, upgrade behaviour and support responsiveness have no benchmark, and they are what teams cite when they describe a choice they regret. "Fast but unoperable" is the common regret, not "slower than advertised".

Implementation patterns

• Ask what their on-call gets paged for. The single most diagnostic question, because nobody has a rehearsed answer, and the list is a direct preview of your own rota.
• Cover the six areas deliberately: what broke and how they found out; what the last major upgrade actually involved; how support behaved during a real incident; what they had to build themselves; how many people it takes to run; and what they would choose again with what they now know.
• Ask what they wish they had known, which reliably surfaces the undocumented constraint.
• Find your own references rather than only accepting the offered ones, since those are selected. Conference talks, engineering blogs, issue trackers, the project's chat and job adverts naming the technology all identify practitioners. An engineer who has filed detailed bug reports is a better reference than a logo on a slide.
• Ask the vendor for a reference at your scale who has completed a major upgrade. A refusal is itself information, and it is a reasonable request.
• Write the call up against your own decision criteria the same day, and circulate it. An undocumented call becomes an opinion within a week.
• Two calls, not one. The second exists to test whether the first was idiosyncratic, and disagreement between them is more useful than either alone.

Industry example

The pattern is standard in enterprise procurement and rare in engineering-led selection, which is odd given that engineers bear the consequence. Where it does happen the evidence is visible in public: the most useful material about any widely deployed datastore is a practitioner's post-mortem write-up rather than the documentation — accounts of sharding migrations, of GC pauses discovered in year two, of upgrade paths that required a full dump and restore. Those write-ups are reference calls conducted in public, which is why they circulate so widely.

Failure scenarios

• Taking only vendor-supplied references, which produces a call with a happy customer at a different scale, and a decision made on a selected sample.
• A call with a manager rather than an operator, which yields strategy and no pager history.
• No scale match, so the reference's experience does not transfer and the team discovers the difference at 10x.
• Not writing it up, after which the call's findings are remembered selectively by whoever preferred that option.
• Doing the call after the commitment, as validation rather than evaluation, which is common and worthless.

Trade-offs

A call costs an hour of two engineers' time plus the effort of finding the right person, and it produces anecdote rather than measurement. A single reference may be unrepresentative, which is why two are better and why the call should be combined with a benchmark on your own access pattern.

What it buys is the only forward-looking evidence about operability available before you commit. The honest comparison: a benchmark on your workload takes most of a week and tells you about throughput; two calls take three hours and tell you about the next two years. Do both, in that order of priority, and if only one is possible, make the calls.

When not to use it

If the decision is cheap to reverse, stop evaluating and run it. A component you can swap in a fortnight deserves a fortnight of production traffic rather than a research programme, and that traffic is better evidence than any conversation. Research effort should scale with reversal cost, so the full treatment belongs to the primary datastore, the message bus everything subscribes to, and the identity provider. Calls are also the wrong tool for a genuinely novel technology with no production users at your scale: there, the honest position is that you are the reference, and the decision should be sized accordingly with a stated exit.

Interview question

Q: You have a week to choose between two databases for a system that will hold five years of customer records. Tell me how you spend the week, and what you would refuse to decide on.

What a strong answer covers: ranking evidence by what it predicts — reference calls for operability and upgrades, a benchmark on the real access pattern for throughput, documentation last; the specific questions, with the on-call one named; sourcing references independently of the vendor; proportioning the effort to reversal cost, and noting that a five-year data store is the case that deserves it; and refusing to decide operability from a benchmark or code quality from an analyst report.

Quick check

Quiz: Which single question makes a reference call worth the hour? "What does your on-call get paged for with this?" — it has no rehearsed answer and previews your own operational load.

Flashcard: Why must a reference be at your scale and over a year old? — A year covers an upgrade, a capacity increase and an incident, and behaviour at 200 GB rarely predicts behaviour at 20 TB.