A vendor's datastore is documented at one million writes per second. Your cluster does 40000 writes per second and p99 write latency is 900 ms. Nothing is erroring. Work from first principles - where do the claim and your reality diverge?

The first three things to look at, in order

1. What the vendor counted as a write. Batched? Acknowledged at the coordinator or after replication? Durable on disk or in a page cache? A million small writes batched into 10000 network round trips is a different measurement from 40000 individually acknowledged durable writes, and both are honestly labelled "writes per second".
2. Your durability and replication settings. Every synchronous replica and every fsync is a latency floor you cannot optimise past. A commit requiring two remote acknowledgements across availability zones pays roughly 1–2 ms of network per hop before any work happens; one requiring a disk flush pays whatever the device's flush latency is.
3. The shape of your keys. A benchmark uses uniformly distributed keys. Production rarely does. A hot partition converts a distributed system into a single-node system with extra hops, and the symptom is exactly this: low aggregate throughput, high p99, no errors, idle capacity elsewhere.

The diagnosis this shape usually points to

900 ms p99 with no errors and throughput 25× below the claim is not a capacity problem. Capacity problems produce errors, rejections or saturation; this produces waiting. Something is serialising: a hot key, a lock, a single coordinator, or a queue in front of a resource with one server.

Work it from the physics. 40000 writes per second at 900 ms p99 implies, by Little's law, roughly 36000 requests in flight at the tail. If your client pool allows 500 connections, the rest of that concurrency is queued in your own application, and the datastore may be perfectly healthy. Check the client side before the server side.

The misleading signal

Server CPU at 30% reads as "the database is fine, the problem is elsewhere". It is consistent with a hot partition, with fsync-bound writes, and with client-side queueing — three different causes, all of which leave the servers idle. Low CPU rules out compute saturation and nothing else.

The fix, and the alert that would have caught it

Measure per-partition write rate and the ratio of your busiest partition to your median one. Alert when that ratio crosses a threshold you choose deliberately — skew, not volume, is what breaks partitioned stores, and no default dashboard shows it.

When the vendor number is not wrong

It usually is not wrong; it is answering a different question. Vendor benchmarks establish an upper bound under the vendor's assumptions, which is genuinely useful for ruling a system out. Use it that way: if the claim is below what you need, stop. If it is above, the claim has told you nothing about whether you will get there.