Burst Credit Depletion

A reporting database runs well for months. A new nightly job starts at 22:00; by 04:00 queries that took 200 ms take 9 seconds; by 07:00 everything is normal again. CPU reads 100% through the bad window and the instance's average utilisation for the day is 20%.

Nothing is broken. Burstable instance families and older volume types do not provide their headline performance continuously. They accrue credits at a baseline rate and spend them whenever the workload exceeds baseline. While the balance is positive the instance behaves like a much larger one; when it reaches zero, performance is capped at baseline - a fraction of a single vCPU on the small sizes - and the transition is a cliff rather than a slope.

The same model governs storage. AWS's gp2 volumes provision 3 IOPS per GiB with a burst bucket that lets smaller volumes reach 3,000 IOPS, and a 100 GiB volume delivers 300 IOPS once the bucket is empty.

Why it matters

It is a capacity limit that no utilisation metric predicts. Average CPU is a mean over a period far longer than the depletion window; instantaneous CPU at 100% during the window looks like an ordinary sizing problem and sends the investigation towards query tuning; and the symptom resolves itself, which makes it look transient and discourages a root-cause hunt.

It is also a pricing model that rewards exactly the wrong instinct. Burstable classes are cheapest, so they are what gets selected for a service with low average load - and average load is precisely the wrong statistic. The question is not "how much CPU does this use on average" but "for how long at a time does it exceed baseline".

Implementation patterns

• Alert on credit balance trending towards zero, which is the only predictive signal. Everything else is after the fact.
• Use fixed-performance families for anything with a predictable peak, and provisioned-IOPS volumes (gp3-style) where throughput and IOPS are set explicitly rather than earned.
• Enable surplus-charging mode where the provider offers it - AWS's T-unlimited - which converts the performance cliff into a cost increase. That is a far better failure mode, and it needs a budget alert so it does not become a surprise invoice.
• Move batch work off serving instances, which is the fix teams should reach for regardless of instance class.
• Size from the duration of the peak, not its height. A workload that exceeds baseline for 20 minutes an hour is a good fit; one that exceeds it for six hours a night is not, at any size.

Industry example

AWS's documentation for the T instance families and for gp2 volumes sets this out explicitly: baseline rates per size, accrual rates, maximum accrued balance, and the published credit-balance metrics. The design is honest - it is a cheaper class for genuinely spiky load, and the provider publishes everything needed to predict the behaviour.

The recurring failure is organisational, not technical. The instance class is chosen during a proof of concept when the load is small, the service grows, the nightly job is added a year later by a different team, and nobody revisits the class - because the only metric that would have flagged it was never added to the dashboard. The lesson generalises to every credit-or-bucket-based cloud resource: if a resource has a balance, the balance is the metric.

Failure scenarios

• The nightly cliff, as above, with recovery before the morning so it never gets investigated.
• A failover onto a burstable standby whose credits have been accruing but whose baseline cannot carry production load, so the recovery itself is the incident.
• Storage credits exhausted on a small volume, giving a database 300 IOPS and a query plan that looks suddenly terrible.
• Surplus charging left on and unmonitored, turning a performance incident into an invoice nobody approved.
• Load tests that never last long enough to drain the bucket, so the configuration passes every pre-production check and fails in production at 04:00.

Trade-offs

When not to use it

Do not choose a burstable class for anything on a recovery path or with a predictable sustained peak: a standby database, a nightly batch target, a queue consumer that drains a backlog. In all three the workload's defining characteristic is a long period above baseline.

Conversely, do not migrate every burstable instance away on principle. For development and test environments, internal tools and small services with genuinely spiky load, the class is cheaper and the model is a fair trade - the error is choosing it without the credit-balance alert, not choosing it at all.

Interview question

Q: A managed database degrades by a factor of 40 between 02:00 and 07:00 every night and recovers on its own. Average CPU for the day is 20%. Tell me what you suspect, how you confirm it in five minutes, and what you change.

What a strong answer covers: the credit model and why it is a cliff; checking the credit-balance metric rather than CPU; why average utilisation and instantaneous CPU both mislead; the storage equivalent with gp2's 3 IOPS per GiB; the immediate options - fixed-performance class, provisioned IOPS, or surplus charging with a budget alert; moving batch work off the serving instance; and the generalisation that any cloud resource with a balance needs the balance monitored.

Quick check

Quiz: Which metric predicts a burstable instance's performance cliff, and why do CPU metrics not? The credit balance. Average CPU is a mean over a period longer than the depletion window, and 100% instantaneous CPU during the window is indistinguishable from an ordinary sizing problem.

Flashcard: What is a burstable instance's performance once its credits are gone? Its baseline - a fixed fraction of a vCPU for the size - reached as a cliff rather than a gradual decline, and it recovers only when load falls below baseline long enough to re-accrue.