A multiplayer-game platform in the mould of Roblox stores player data encrypted with envelope encryption. Each service call decrypts a data key through the regional key management service, 6,000 requests per second, with a 5 s client timeout and 200 worker threads per instance. The key service becomes slow (p50 2 s) but not unavailable. What happens, second by second, and what stops it?

Second by second

In the first seconds every request that needs a data key waits about 2 s instead of a few milliseconds. By Little's law the number of requests in flight in the service is the arrival rate times the latency: 6,000 x 2 s = 12,000 in-flight requests, spread over the fleet. If 40 instances serve this, each instance has 300 requests waiting against 200 threads, so thread pools are exhausted within seconds.

Once threads are saturated, requests that do not need a key (health checks, cached reads) queue behind those that do. The load balancer's health check times out, instances are marked unhealthy and removed, and the remaining instances take more load. The p50 of 2 s becomes timeouts at 5 s, and clients retry, adding load to the key service that is already slow.

Where it amplifies

Retries. Each timeout produces another call to the key service, doubling its load and deepening its slowness. Health checks coupled to the dependency turn a latency problem into capacity loss. A slow dependency is worse than a dead one because failures are fast and slowness holds resources.

What the user sees

Game sessions fail to load profile data, some succeed after long waits, and matchmaking times out. Reads that were previously cached continue to work.

What stops it

Mechanisms, not monitoring:

1. Data key caching. Keep decrypted data keys in memory for a bounded time (minutes) or a bounded number of uses, so the key service is called once per key, not once per request. AWS Encryption SDK documents this as data key caching, with explicit limits on age and bytes encrypted. It reduces 6,000 calls a second to a few per minute.
2. Short timeouts and a bulkhead. Limit concurrent key-service calls to a fixed pool (for example 50 per instance), fail fast beyond it, and keep the rest of the threads free.
3. Circuit breaking on slow calls, not only errors, plus retry with a budget and jitter.
4. Decouple health checks from the key dependency.

What would have to be true for it to self-heal

Keys cached before the slowdown keep serving; the key service recovers; the retry budget prevents a rebound. With a warm cache and a bulkhead, the user-visible effect is a few cold-key requests failing quickly.

The cost of the fix

A cached data key lives in process memory for its lifetime, which widens the window in which a memory dump exposes it, and key revocation takes up to the cache age to propagate. Choose the cache lifetime from the revocation requirement, not from convenience.

When this is over-engineering

At 50 requests a second the key service call is cheap and caching adds a way to hold stale keys. Cache when call rate times latency approaches your thread budget.

Common weak answers

• "Add more instances." More instances make more calls to a slow dependency.
• "Increase the timeout." Longer waits hold threads longer.