intermediate 3 min answer

Peak trading day is six weeks away and expected to be four times normal traffic. What do you do in those six weeks?

capacityload-testingpeakreadiness
Show the full answer Hide the answer

What the interviewer is testing

Whether you can run a readiness programme rather than just "add servers", and whether you know what fails at peak that does not fail in a load test.

Weeks 1–2: model and measure

Model first, cheaply. For each tier, at 4× traffic and current per-request latency, what concurrency is required (Little's Law), and how does that compare to the configured pool sizes, connection limits and provider quotas? This is arithmetic and it finds the connection-limit ceiling in an afternoon — the most common first failure and the least visible in advance.

Inventory the things that do not autoscale: database primaries, third-party API quotas, licence-limited components, IP address space, account-level cloud quotas capping the autoscaler, and any single-instance component (a scheduler, a cron worker).

Start the lead-time items now. Quota increases, reserved capacity purchases, and third-party limit raises are commercial conversations with lead times measured in weeks. Six weeks is enough only if these begin immediately.

Weeks 2–4: load test properly

The three things that make load tests lie, and how to avoid them:

  • Production-sized data, or everything sits in cache and the database never does real work.
  • Realistic traffic mix, including the concentration peak actually produces. Peak trading is not uniform — it is one promoted product taking a disproportionate share, which creates a hot key or hot partition that a uniform test will never reveal.
  • Ramp, not step, so autoscaler lag and cold starts are exercised. Then also run a step test, because a doorbuster promotion is a step.

Push past 4× until something breaks. The purpose is to find the first bottleneck and its failure mode, not to get a pass mark. Expect three rounds: fixing the first bottleneck reveals the second.

Weeks 4–5: build the protections

Because the forecast will be wrong, and the design must degrade rather than collapse:

  • Rate limiting so excess is rejected cleanly rather than degrading everyone.
  • Load shedding with a priority order agreed with the business: checkout and payment protected, recommendations and personalisation shed first.
  • A static or degraded fallback for the catalogue.
  • Cache warming for the promoted items, and jittered TTLs so nothing expires in unison.
  • Queue the write path where possible, so a database spike becomes backlog rather than errors.
  • Circuit breakers with fallbacks on every third-party dependency, especially payment and fraud.

Week 5–6: freeze and rehearse

  • Change freeze ahead of the day, with a defined exception process. Most peak incidents are self-inflicted by a deployment.
  • Game day: rehearse the failure scenarios — a dependency down, a zone lost, cache cleared — and time the response.
  • Dashboards and alerts specifically for the day, including business metrics (orders per minute, conversion) alongside technical ones. A technically healthy system with a collapsed conversion rate is an outage.
  • Runbooks and staffing, with named people and a decision-maker for shedding.
  • Pre-scale rather than relying on autoscaling for the known start time. Autoscaling reacts in minutes; a doorbuster arrives in seconds.

What a strong answer adds

The point most people miss: the cache is the risk, not the compute. A system running at 4× traffic on a warm cache is fine; the same system with a cold cache cannot recover, because the database saturates and nothing completes, so the cache never fills. Ask explicitly what happens if the cache tier is lost at peak — and if the answer is not "we degrade to a static catalogue", that is the work for week five.