Discord's Read States service, written in Go, showed latency spikes every two minutes like clockwork. The team had written it carefully with very few allocations, and the spikes appeared regardless of load. Their published account from 2020 explains the cause and the rewrite that followed. What was happening, and what does it teach about periodic latency?

The symptom, and what it rules out

Periodic at 2 minutes, fixed interval, independent of load. That combination eliminates most candidates immediately. Load-dependent causes vary with traffic. Dependency problems vary with the dependency. A spike on a fixed period points at a timer: a scheduled job, a cache refresh, a metrics flush, a certificate reload — or the language runtime.

Read States is on the hot path: it is consulted whenever anyone connects, sends a message or reads one, so a two-minute sawtooth in latency is visible to a very large number of people.

The diagnosis

Go forces a garbage collection at least every two minutes even when the heap has not grown. Discord's service allocated very little, so no allocation-driven collection was ever triggered — and the forced one still ran.

The crucial mechanism is what makes that forced collection expensive: the cost of the mark phase is proportional to the live set, not to the amount of garbage. A service holding 10 GB of live cache pays for all 10 GB on every forced sweep, and the request unlucky enough to arrive during it simply waits. The service held a large cache of live objects, so every forced sweep walked all of it.

The counter-intuitive part is the whole lesson. Writing code that allocated less made the observed behaviour worse, not better, because low allocation removed the ordinary collections and left only the forced ones, running against a heap that was almost entirely live.

Why the usual tools missed it

CPU utilisation averaged over a minute conceals a sub-second stall. Distributed tracing shows the latency and attributes it to the service, which is true and useless. The instrument that answers this is runtime-level telemetry — GC pause histograms, live heap size, allocation rate — which most services do not export, and so this class of problem gets attributed to the network.

The fix, and what it cost

Discord rewrote the service in Rust, which has no garbage collector: memory is released deterministically and there is no periodic global pause. They reported the spikes eliminated along with lower CPU and memory use.

The cost is a rewrite, a second language in the estate, and the hiring and tooling that follow from it. That is a large bill and it was justified here because the large live set was the product — the cache was the service.

The tempting local fix, and when it is enough

Before a rewrite, the cheaper moves target the same mechanism:

• Shrink the live set. Move the cache out of process into a shared store, or into an off-heap structure the collector does not scan. This attacks mark cost directly and is usually sufficient.
• Shard the service so each instance holds proportionally less.
• Tune the collector — pacing, ballast, soft memory limits — which buys headroom rather than a fix.

Where copying it would be a mistake, and when not to rewrite

"Rewrite in Rust" is not the transferable lesson. Most services have sub-millisecond pauses that nobody would ever notice, and changing language is among the most expensive decisions an organisation can make.

The transferable lessons are cheap: a fixed-period latency spike is a timer, and the runtime is a timer; GC cost tracks live-set size rather than garbage produced; and a large in-process cache under a managed runtime is a combination that should be examined before it is defended. Put GC pause and live heap size on every service dashboard, and this becomes a five-minute diagnosis instead of a project.