Twitch wrote in April 2014 that it had been sending all of its logging to a hosted analytics product and that the volume had reached billions of events per week - growing far faster than its user base - so it built its own ingest path targeting under 24 hours of data latency, running at around 5 hours at the time, with a faster layer planned for the real-time dashboards. Why does event volume outrun user growth, and what should you size a new pipeline on?

The situation they were in

A hosted analytics product is priced per event. That is fine while events track users, and it stops being fine at the point where events per user is itself growing. Twitch described reaching billions of events a week and moving to its own storage because of the cost of continuing.

Why events grow faster than users

Events per user per day is not a constant of nature. It is a number the engineering organisation sets, and it only moves one way:

• Every new feature adds events. A feature that ships with three tracked interactions adds three events per user who touches it, permanently.
• Every experiment adds an exposure event per user per experiment, and experiment counts grow with the size of the product team.
• Every debugging need adds a field or a heartbeat, and a heartbeat multiplies by session length rather than by sessions.
• Session length grows on a video product as the catalogue grows, and a per-second or per-minute playback event scales with watch time rather than with people.

So volume is roughly users × events per user, and both factors grow. A pipeline sized on the user forecast is sized on one of the two terms.

What to size on, and what to watch

Size on events per active user per day, projected forward, and assume it roughly doubles over a 12 to 18 month horizon independent of user growth unless someone is actively removing instrumentation. Then multiply by the user forecast.

Put events per active user on a dashboard next to total volume. It is the leading indicator and the only one that distinguishes "the product is growing" from "someone added a heartbeat", and those two have completely different responses. A step change in that ratio with no user change is a release to investigate on the day it happens, not at the next invoice.

The decision they made, and the rule in it

They built the bulk path first, at hours of latency, and planned a fast layer for the real-time dashboards rather than for everything. The whole pipeline does not have to be as fast as its fastest consumer. Most queries over event data are answered perfectly well by data that is a few hours old; the handful that are not can be carved out and served separately at a fraction of the cost of making everything current.

When this is the wrong answer

If the events drive an operational decision - a fraud block, a dispatch, a live moderation action - the hours-of-latency bulk path is not a cheaper version of the right answer, it is the wrong answer. Separate the two populations of events before sizing anything: operational events get the fast path and are usually a small share of volume, while analytical events get the cheap path.