intermediate 3 min answer Multiple choice

You are evaluating a database whose only production references are the vendor's own case studies. The benchmark in their documentation is excellent. You have one week. Which single piece of evidence would most change your confidence?

evaluationevidencereference-callprocurementresearch
Pick one
Show the full answer Hide the answer

The reasoning

Benchmarks answer the question the benchmark author chose. A reference call answers the questions you cannot construct, because the failure modes that decide whether you regret a datastore are not throughput — they are what happens at 2 a.m. in year two.

What the call is for, specifically: what broke, what the upgrade path was actually like, how support responded when it mattered, what they had to build themselves, and what they would choose again. Ask what they wish they had known, and ask how many people it takes to run. The single most diagnostic question is "what does your on-call get paged for with this?" — nobody has a rehearsed answer, so you get the truth.

Two qualifiers matter. At your scale, because a datastore that is delightful at 200 GB may be a different product at 20 TB. For over a year, because a year covers a major version upgrade, a capacity increase, and at least one incident, and those are the three things the sales cycle cannot show you.

Why the other options fail

  • Reproducing the benchmark on your workload. The most tempting, and it is the second-best answer rather than the best. It is genuinely valuable — a benchmark on your access pattern with your data distribution regularly collapses a vendor's headline number, because the shape of your keys and the mix of reads and writes is where the number came from. But it takes most of your week, it tells you nothing about operability or upgrades, and a system that is fast and unoperable is the more common regret. Run it second, and run it only on the access pattern you actually have.
  • A code review of the open-source core. A week of reading gives you an impression of code quality and no evidence about behaviour under load, recovery, or the vendor's responsiveness. It also flatters engineers into a comfortable activity that does not reduce the decision's risk.
  • An independent analyst report. Useful for a shortlist and for internal argument. It is a survey of vendor claims and market position, at the wrong altitude to predict your incident in year two.

How to get the call when the vendor controls the references

Do not only accept the references offered, because those are selected. Find users yourself: conference talks, engineering blogs, issue trackers, the project's chat, job adverts mentioning the technology. An engineer who has filed detailed bugs is a better reference than a logo on a slide, and asking the vendor for a reference at your scale who has completed a major upgrade is a reasonable request whose refusal is itself information.

When this is the wrong answer

If the decision is cheap to reverse, stop evaluating and run it. A component you can swap in a fortnight does not deserve a week of research; a fortnight of production traffic is better evidence than anything on this list. The research effort should scale with reversal cost, and the systems that deserve the full treatment are the ones whose data you cannot easily move: the primary datastore, the message bus everything subscribes to, the identity provider.