Research Skills

Definition

The ability to answer a technical question well and quickly: does this technology fit, has this problem been solved, what actually happens at scale.

The hierarchy of sources

1. Your own measurement. A two-day spike against your workload and your data beats any amount of reading. Most performance and integration questions are settled this way and should be.
2. Someone operating it at your scale. Worth more than all documentation, and usually obtainable through a network or a user group.
3. Operational documentation — upgrades, backup, monitoring, troubleshooting. Where you will actually live, and always documented last.
4. Published engineering accounts and incident reports, read critically.
5. Foundational papers, for genuinely understanding a mechanism.
6. Vendor material, which describes the happy path and the vendor's workload.

Reading engineering blogs critically

The most common research error is reasoning by analogy from a company whose constraints differ from yours. A published architecture reflects their scale, team size, existing estate and history, and adopting it without those conditions is how organisations end up operating machinery for a problem they do not have.

Ask of every account: what scale, what team size, what constraint drove this, and does that constraint apply here?

What to research about a technology

• How it fails. Node failure, disk full, upgrade, network partition. This finds more than any benchmark.
• What operating it involves. Backup and tested restore, online schema change, monitoring, upgrades.
• Who maintains it, and their trajectory.
• What people who left it say. Search for migration-away accounts; they are more informative than adoption ones.
• The cost model at your projected scale, not today's.

The spike, done well

• Timebox it, and state the question it answers before starting.
• Use realistic data volumes and realistic skew. Uniform synthetic data avoids exactly the hot-key problems that break production.
• Test the failure path, not the happy path.
• Write down what you learned, including what you could not determine.

Failure scenarios

• Deciding on benchmarks, which describe someone else's workload.
• Copying an architecture whose constraints do not apply.
• Unbounded research, which never concludes.
• Not testing failure, so the finding is about the happy path only.

Interview question

"How would you evaluate an unfamiliar technology for a critical workload in one week?"