Operational Rehearsal
also called Worst-Day Evaluation, Day-2 Bake-Off
Evaluating a candidate technology by running the operations that will hurt later - a restore, a major upgrade, a node loss, a hot partition - instead of benchmarking its best day.
Every bake-off measures throughput and latency on a clean cluster with a synthetic workload. Almost none measures the restore. So the decision is made on the one dimension hardware improves anyway, and is silent on the ones that produce the outage: how long a recovery takes, what a major-version upgrade requires, what happens when one node is slow rather than dead, and whether the team can diagnose it at 03:00.
An operational rehearsal evaluates the candidate's worst day, before the decision rather than after it. It is usually cheaper than the benchmark it replaces, because it needs no careful load generation - only a willingness to break things on purpose and write down what happened.
Why it matters
A datastore is owned for years and chosen in weeks, and over that life the costs that dominate are operational: upgrades, restores, rebalancing, and how many engineers can safely touch it. Benchmarks say nothing about these and vendor documentation describes the happy path.
A technology 20% slower is a capacity line item. One nobody can restore is an existential incident, and the time to find out is before the data is in it.
Implementation patterns
- Restore drill with a stopwatch. Back up, destroy the instance, restore to a point in time, and record the elapsed minutes and the recovery point achieved. That number belongs in the decision record next to the throughput figure.
- Run the major-version upgrade before choosing, and record the downtime and the rollback story.
- Fail a node, then - harder - make one slow. Slow breaks naive clients, because requests queue rather than error. Watch the client library, not only the server.
- Create the pathological data shape - a hot partition, a key with a million values - and name the first warning signal.
- A parallel run against the incumbent where the stakes justify it: both take the real workload, one serves, and you compare failure rates before retiring the old one.
- Rotate the operator. Someone who did not do the evaluation performs the restore from the runbook, which tests the documentation you will actually depend on.
Industry example
Grab's engineering blog described this discipline around a major infrastructure change in November 2025: before retiring a rented macOS build fleet for a colocation cluster of more than 200 machines, the team ran the new cluster alongside the old one for a long period, comparing failure rates and job performance, and only then ended the vendor contracts. The published performance figures - p50 pipeline time falling from 43 to 33 minutes - come from that comparison rather than from a vendor claim.
The independent form is Jepsen, publishing adversarial analyses of distributed databases since 2013 by inducing partitions, clock skew and process pauses against real clusters - repeatedly showing that systems behave differently on their worst day than their documentation implies.
Failure scenarios
- The restore nobody timed, so the recovery time objective is a wish and the discovery happens during the incident, with the business asking for an estimate.
- An upgrade path tested on an empty database, hiding the hours a large table's rewrite takes.
- A rehearsal run by the technology's advocate, who knows the undocumented workaround, leaving the runbook wrong.
- Node failure tested, slowness not - the gap that produces cascading timeouts.
Trade-offs
| Choose | Gains | Pays |
|---|---|---|
| Rehearse the worst day | The costs that dominate ownership become visible before commitment | Days of deliberate breakage, and a team willing to publish an unflattering result |
| Benchmark only | A fast answer and a clean chart | A decision made on the one axis that hardware fixes anyway |
Rehearsals produce uncomfortable findings about systems already chosen - their value, and the reason they get quietly dropped.
When not to use it
Skip it where the choice is reversible and small - a library, a build plugin, a cache in front of one endpoint. The cost of being wrong is a sprint, so a day of deliberate breakage costs more than the mistake. The practice belongs to things that hold state or carry production traffic for years.
Skip it when the spread is tiny - two candidates differing by 20k dollars over three years, both already operated by your team.
Interview question
Q: A team hands you a 30-page evaluation recommending a new database with excellent benchmark numbers. Which three exercises do you require before approving it, and what single result would make you reject it anyway?
What a strong answer covers: a timed point-in-time restore, a major-version upgrade on a realistically sized dataset, and a slow-node test observed from the client library; the runbook executed by someone who did not write it; and a rejection criterion - a recovery time beyond the business's stated objective, or an upgrade needing downtime the product cannot take.
Quick check
Quiz: Which exercise most often reverses a decision made on benchmarks? The timed restore. Recovery time and achievable recovery point bind the business, and a benchmark never produces them.
Flashcard: What does an operational rehearsal measure that a benchmark cannot? - The worst day: restore time and recovery point, upgrade downtime, behaviour when a node is slow rather than dead, and whether the runbook works in other hands.