An architect must evaluate an unfamiliar technology for a significant decision. What research approach produces a reliable judgement?

The approach

1. Start from the questions the decision needs answered, not from the technology's documentation. Which properties would make this right or wrong for us? That list determines what to investigate and prevents an open-ended read that produces familiarity rather than judgement.

2. Read the failure literature first. Public postmortems, "why we moved off X" accounts, and issue trackers tell you the failure modes, which is what the marketing material and the getting-started guide never do. Failure modes are the most decision-relevant information available and the least published in official documentation.

3. Time-boxed spike against your actual workload. Not a benchmark. Public benchmarks answer questions you do not have, on data that is not yours, at a scale you may not reach. Test the thing you are worried about — filtered query performance at realistic selectivity, behaviour under the failure you fear, an upgrade, a restore.

4. Talk to people operating it at your scale, which surfaces the operational realities that no document contains — what breaks, what is hard, what they would do differently.

5. Check the operability questions, which are frequently decisive and rarely investigated: how does it fail, can the failure be diagnosed, what does an upgrade involve, what does a restore involve, and who on the team could debug it at 3 a.m.

What produces unreliable judgements

• Evaluating on the getting-started experience, which is designed to impress and says nothing about operation at scale.
• Benchmarks, which are the vendor's workload rather than yours.
• Feature comparison matrices, which reward breadth over fitness.
• Community enthusiasm, which correlates with novelty rather than with suitability.
• One person's deep dive with no independent check, which is how a preference becomes a decision.

The output

A written comparison with the criteria named first, each option's position stated concretely, the dominant criterion identified, a recommendation with visible reasoning, and what would change it.

That is reviewable and challengeable on grounds rather than on authority — which is the point of doing the research at all.