A nightly billing batch that ran in 4 hours 10 minutes on a 2016 on-premises server takes 9 hours 20 minutes after a lift-and-shift onto a cloud instance with twice the vCPUs and four times the memory. CPU sits near 12 percent across all cores and the instance is never network-throttled. Where is the time going?
Show the full answer Hide the answer
The first three things to look at, and why in that order
- Per-core utilisation rather than the average. 12% across 16 cores is about two busy cores. The average is the misleading signal in the stem: it says the machine is idle, when what it says is that the job cannot use the machine. One or two threads are running flat out and the other fourteen cores are decoration.
- Sustained clock, not core count. A 2016 workstation-class or single-socket server commonly ran 3.4 to 3.7 GHz on a lightly threaded load. A general-purpose cloud instance commonly sustains 2.4 to 2.9 GHz. That is a factor of roughly 1.3 to 1.5 on the CPU-bound portion, before anything else.
- Commit latency. A local battery-backed RAID controller or local NVMe acknowledges an fsync in roughly 0.05 to 0.15 ms. Network-attached block storage is typically 0.4 to 1 ms. A batch that commits once per row pays that difference 10 million times.
The diagnosis
Rehosting gives more throughput and less latency, and a serial batch is a latency workload. The arithmetic fits: if the job is 60% CPU-bound and 40% commit-bound, a 1.4x clock penalty and a 6x fsync penalty produce roughly 0.6 × 1.4 + 0.4 × 6, or about 3.2x on paper; the observed 2.24x says some of the commit cost is absorbed by write caching. Either way the shape of the answer is the same, and no amount of vCPU or memory touches it.
Why the other options fail
- "Undersized — move to a larger class." This is the instinct the dashboard invites and it makes things worse: larger general-purpose instances usually add cores at the same or lower sustained clock, so the bill rises and the runtime does not move. It would be right if utilisation were pinned near 100% across all cores.
- "Add read replicas." Correct treatment for a read-heavy workload competing with serving traffic. Here the batch is alone on the box at night and is not waiting on query throughput; it is waiting on its own serial progress. Replicas add a consistency problem and no speed.
- "The network to the database is saturated." Explicitly excluded by the stem, and the per-core picture rules it out anyway: a saturated link shows up as high iowait with cores parked, not as two cores busy. This is the distractor that costs a week of packet captures.
The fix, in order
- Batch the commits. 1,000 rows per transaction turns 10 million fsyncs into 10,000. This is usually the single largest win and it is a code change of a few lines in the loop.
- Move to a high-clock instance family for this one host. Compute-optimised families sustain higher clocks; the batch host does not have to match the rest of the estate.
- Put the write-ahead or temp volume on local instance storage if the job's durability model allows a rerun from the start, which batch jobs usually do.
- Only then parallelise by account or date partition, which is a real change with real correctness risk and should not be attempted while the serial version is 6x off its own commit cost.
- Alert on batch duration as a fraction of the window, not on completion. 9 hours 20 in a 10-hour window is already an incident.
When this is the wrong answer
If the job genuinely runs many parallel workers and still shows 12%, the bottleneck is downstream and the analysis above is a dead end: look at the database's lock waits first. And if the batch window is 14 hours, none of this is worth doing. Rehosting a serial batch and expecting it to get faster is the error; the plan should have budgeted for the regression and verified the window before the lease ran out.