Discord's Read States service was fast most of the time but showed latency spikes every two minutes. The pattern looked familiar to anyone who has seen a service under load. The cause published in 2020 was Go's garbage collector scanning a large LRU cache. What does this teach about pattern recognition, and which check would have got there faster?

The situation they were in

Read States tracks which messages a user has read, and it is touched on every connect, every message sent and every message read. The symptom was periodic latency spikes rather than load-correlated ones, and Discord's 2020 write-up attributes them to the Go runtime forcing a collection every two minutes, with the collector having to scan a large in-memory LRU cache to establish what was still referenced. They rewrote the service in Rust, which has no garbage collector, and the spikes went away.

Why pattern recognition misleads here

An experienced engineer seeing p99 spikes on a hot service has a ready-made pattern: contention, queueing, a saturated dependency. Every one of those produces latency spikes, and the standard moves follow — add replicas, raise connection limits, look at the database.

The distinguishing evidence is in the shape of the signal, not its magnitude. Load-driven latency tracks load, so it is correlated with traffic and irregular. A spike that arrives every two minutes regardless of traffic is periodic, and periodicity implicates a timer, not a queue: a garbage collector, a cache refresh, a cron job, a metrics flush, a certificate reload, a leader lease renewal.

That is the check worth internalising. Before matching a pattern, ask what the symptom is correlated with. Correlated with request rate, look at capacity and contention. Correlated with wall-clock time, look for something scheduled. Correlated with a specific key or tenant, look at distribution and hot partitions. Correlated with a deploy, look at the diff.

Why the honest answer is uncomfortable

The pattern match was not stupid; it was a valid prior with an unexamined precondition. Load-driven latency assumes the resource under contention is one your load controls. Runtime behaviour breaks that assumption: the collector's work is a function of live heap size and object count, so a service with a big cache pays a pause that has nothing to do with the request in flight. Discord's point was specific — the cache was large enough that scanning it was expensive even when almost nothing had become garbage.

What to do rather than rewrite

A rewrite was the right call for them and is the wrong first move for most teams, because it costs months and the mechanism is usually addressable:

1. Move the big object out of the traced heap — fewer, larger objects, off-heap or arena storage, or a cache keyed by primitive values rather than a graph of pointers.
2. Cap the cache. Pause cost scales with what the collector must examine.
3. Tune the trigger so collections are driven by allocation rather than by a timer where the runtime allows it.
4. Measure before choosing. Runtime pause telemetry answers this in a day; a language migration answers it in a quarter.

When this is the wrong answer

If the periodic spike costs you nothing that anyone can measure in user terms, leave it. Many services have a two-minute hiccup of 30 ms that no user notices, and hunting it is a hobby. The case for acting is a product requirement the service is failing: Discord's was that Read States sits in the path of every connect, so its tail is the tail everyone sees. Language choice is the last resort, and it is only defensible when the mechanism — not the fashion — is the reason.

Common weak answers

• "Add more instances." The pause is per process and happens on all of them, so the spike is unchanged and the cost rises.
• "It is a Go problem, so rewrite in Rust." Sometimes true, and stated without the measurement it is a preference. The transferable skill is the correlation test, not the conclusion.
• "Alert on p99." They already knew. The gap was diagnosis, not detection.