Twitch described in 2019 how its Go API frontend, Visage, spent a large share of CPU in garbage collection while holding a live heap under 450 MiB on 64 GiB hosts. What did their profiles show, what did they change, and where would copying it today be a mistake?

The situation they were in

Twitch's engineering blog (Ross Engers, April 2019) described Visage, a Go service that fronts the site's API. At steady state it ran garbage collection roughly 8 to 10 times per second, with over 30% of CPU in GC-related calls, and refresh storms from clients made it worse. The live heap was small (under about 450 MiB) on hosts with 64 GiB of memory, so the machine had nearly all of its memory free while the collector ran constantly.

How they found it

They took production profiles with pprof and ran the execution tracer for a few seconds each hour. The traces showed request goroutines performing GC mark assist, where a goroutine that allocates during a collection is made to help with marking. That is the mechanism that turned a CPU cost into a latency cost: requests were slowed directly, not just starved by background work.

What they chose

They allocated a 10 GiB byte slice at startup and kept a reference to it, a "ballast". Because the collector's trigger is a multiple of live heap, the ballast moved the trigger to roughly 20 GiB. A byte slice has no pointers, so marking it is cheap, and the pages are never touched, so they stay virtual and use no physical memory. The post reports about a 99% fall in collections, about 30% lower CPU per host and about 45% lower p99 API latency at peak, which let them shrink the fleet by a similar fraction.

Why it fit their constraints

The service was allocation-heavy with a small live set and had abundant memory, so the default collector pacing was the wrong policy. The decisive step was not the trick; it was seeing GC cost in production data rather than guessing from a load test.

What it cost

A tuning hack that relies on runtime internals, a number (10 GiB) that must track the host size, and a risk if memory per container shrinks.

When this is the wrong answer

When the CPU profile shows little GC, or the service has a large live heap, a ballast changes nothing and costs a tuning hack.

Where copying it would be a mistake

• On a current Go release, use the runtime's soft memory limit (GOMEMLIMIT, Go 1.19, 2022) together with a higher GOGC. It states the intent directly and works with container limits. Verify against your release notes.
• In a memory-constrained container, a ballast that is counted as live by the collector can interact badly with cgroup limits and with accounting tools that read virtual size.
• Without a profile. If your CPU is not in GC, none of this applies. Check the GC share of the CPU profile first, and the mark-assist time in a trace, before touching a knob.

What a strong answer adds

The general lesson is the order of work: profile continuously, find the share, change one policy, and re-measure at p99 not mean, because the gain here was in the tail.