Peak Readiness Rehearsal

Provisioning instances is the easy and visible part of preparing for a scheduled peak. The failures happen elsewhere, and they are consistent enough to be a checklist.

Why it matters

Every item on the list is invisible in a capacity calculation and fatal in production. A fleet with correct instance count and cold caches sends every request to the database in the first minute of the peak, which is precisely when the database has least margin.

Implementation patterns

Before the peak:

• Warm the caches, and verify it. Not an optimisation — a required step, and one that must be confirmed rather than assumed.
• Warm the connection pools. Establishing thousands of connections simultaneously is itself a load spike, and TLS handshakes at that volume are meaningful CPU.
• Allow runtime warm-up time, since a freshly started process is measurably slower for its first few thousand requests.
• Pre-scale on a schedule rather than on a metric, because the ramp outruns any autoscaler.
• Confirm every downstream ceiling — provider rate limits, notification quotas, identity throughput, database connection limits, egress capacity. The peak finds whichever is lowest, and they are frequently owned by someone else.
• Freeze changes, with an explicit exception process, and check that no backup, index rebuild or batch job is scheduled inside the window.

During:

• A named person watching with authority to shed, disable features and roll back without seeking approval.
• A shedding policy already rehearsed in production, because a path that has never run will not work the first time.

After:

• Compare forecast with actual and record it. The distribution of forecast error is the most valuable artefact a team with recurring peaks can build, and without it the same shortfall repeats.

Industry example

Brokerages such as Zerodha and Groww face a scheduled peak every trading day, and fantasy-sports platforms such as Dream11 face one before every major fixture. The shared property is that the timing is known precisely, which makes reactive scaling both unnecessary and insufficient — and which makes the operational checklist, rather than the architecture, the thing that most often determines the outcome.

Failure scenarios

• Correct capacity, cold caches, so the database absorbs the first minute.
• A deployment or batch job landing in the window, a recurring self-inflicted cause.
• Dependency ceilings unverified, so the constraint is someone else's quota.
• A shedding path that has never run.
• No post-event comparison, so forecasting never improves.

Trade-offs

Pre-provisioning and freezing costs money and delivery velocity, and applying the full checklist to a minor peak is disproportionate. The judgement is proportional to what the peak represents: for an event carrying a large share of the day's revenue with a hard deadline, all of it is cheap.

The property that distinguishes mature operations is that nothing about the peak is discovered on the day — which is a process outcome rather than an architectural one, and is frequently the gap in technically excellent systems.

Interview question

"Your capacity plan for tonight's peak is correct and you have provisioned exactly what it says. List everything that could still go wrong, in order of likelihood."