Anatomy · Performance & capacity · Simulation

A web stack under load, as a marble run

Every steel ball is one request. A load balancer queues it, an app server holds it for its service time, the cache answers most of them, and the rest wait for one of a handful of database connections. Nothing here is drawn by hand: a discrete-event simulation moves every ball, and every number is checked against the queueing theory architects size systems with. Turn the traffic up until it breaks.

A request being served: on a drum, or in the cacheDrums: app servers (left) and database connections (right)Troughs: the queues, one slot mark per waiting placeLeft dial: p99 latency, 0 to 1 s, from the hidden full-speed run. Right dial: app-tier utilisation λ/cμ, with the last fifth of the scale ruled heavier. The hopper releases balls at the traffic rate; a full trough refuses the next ball into the bin.
Try
Defaults: 60 requests/s against 3 app servers of 40 ms each (75/s of capacity), a 70% cache hit ratio, and 2 database connections of 30 ms.
Watch

Every request, as it finishes

Each dot is one ball landing in the tray: when it finished, and how long it took from the hopper. The line is what queueing theory says the average should be at the current settings. Most requests are fast; the ones that queued behind a burst make the tail.

Refused requests are the cross marks at the top: the trough was full when they arrived, so the load balancer answered 503 rather than make the queue longer.

Go deeper Tail latency Queueing delay Load shedding
Predict

The hockey stick

Average response time against utilisation for the app tier at its current size. It is flat for a long way, then turns up hard. For a single server the response time is s/(1−ρ), so going from 50% to 80% busy multiplies it by 2.5, and going from 80% to 95% multiplies it by another 4.

The dot is where the stack is now. More servers move the curve right, which is why the same traffic that breaks three servers barely registers on four.

Go deeper Utilisation target Queueing theory Capacity planning
Check

Simulation against theory

A second copy of the same simulation runs out of sight at full speed, so its averages settle in seconds rather than minutes. Each tier is an M/M/c queue: Poisson arrivals, exponential service, c servers sharing one line. Erlang's formula gives the chance a request waits at all, and Little's law ties the queue length to the wait: L = λ·W, always.

Go deeper Little's law Kingman's formula Connection pool sizing
Theory

Little's law

In any stable system, the average number of things inside it equals the rate they arrive times the average time each one stays: L = λ·W. It needs no assumption about distributions, which is why it is the first tool for sizing anything. At 60 requests a second and 100 ms each, there are 6 requests in flight on average, so a thread pool of 6 is the floor, not a comfortable size.

The table above checks it live on the hidden run: the measured average in flight against λ times the measured latency.

Little's law, the topic · in practice
Theory

Why the tail explodes first

For an M/M/c queue the chance of waiting longer than t falls off as C·e−(cμ−λ)t: the spare capacity cμ−λ sets the decay. As traffic approaches capacity that spare shrinks toward zero, so the tail stretches long before the average looks alarming. p99 is the number that pages people, and it degrades several times faster than the mean.

Fan-out makes it worse: a page that waits on 100 backend calls hits at least one of them at its p99 63% of the time (1 − 0.99100).

Tail latency · Tail amplification
Theory

Variability costs as much as load

Kingman's approximation for one server, Wq ≈ (ρ/(1−ρ)) · ((ca² + cs²)/2) · s, separates the two culprits: how busy the server is, and how irregular arrivals (ca) and service times (cs) are. This model uses exponential arrivals and service, both with a coefficient of variation of 1. Halve the service-time variance (cs² from 1 to 0.5, by caching the slow path or capping query time) and the wait falls by a quarter at the same utilisation.

Kingman's formula · Queueing delay
Practice

What architects do with it

Plan for headroom, not averages. Target 60 to 70% utilisation at peak so a spike or a lost node lands on the flat part of the curve. Treat the cache as capacity. The database sees only the misses, λ(1−h): drop the hit ratio from 70% to 10% and the database load triples, which is what the cold-cache scenario shows. Bound every queue. An unbounded queue turns overload into timeouts for everyone; a bounded one refuses some requests fast and keeps the rest healthy. Size pools from Little's law, not from defaults.

Pool sizing · Load shedding · Retry amplification · Autoscaling
The model, stated

What is simulated, and what is simplified

Arrivals are Poisson. The app tier is one shared FIFO line in front of c identical servers, which is how a least-outstanding-requests balancer behaves, with exponentially distributed service of mean 40 ms. A cache hit takes 2 ms and never queues. A miss waits in a second line for one of the database pool's connections, with exponential query time of mean 30 ms. Each hop between tiers costs 3 ms. Each line holds at most 12 waiting requests, and arrivals beyond that are refused. Real systems add retries, timeouts, garbage-collection pauses and correlated slowness, all of which make the tail heavier than this; the shape of the curve is the part that transfers.

Sources

  1. Little, 1961. A Proof for the Queuing Formula: L = λW (Operations Research 9(3))
  2. Kingman, 1961. The single server queue in heavy traffic (Proc. Cambridge Phil. Soc.)
  3. Harchol-Balter, 2013. Performance Modeling and Design of Computer Systems: Queueing Theory in Action. Cambridge University Press (M/M/c and Erlang C, chapter 14)
  4. Dean and Barroso, 2013. The Tail at Scale (Communications of the ACM)
  5. Google SRE book, chapter 21: Handling Overload

Built with the anatomy kit · balls, dials and charts are driven by one simulation, checked against Erlang C and Little's law