advanced 2 min answer

Which windowing strategy fits which metric, and what is the property that breaks naive windowing?

windowingtumblingslidingsessionlate-datadatadogdesign
Show the full answer Hide the answer

The strategies

Tumbling windows — fixed, non-overlapping. Natural for periodic aggregates: requests per minute, errors per five minutes. Cheapest, and each event belongs to exactly one window.

Sliding windows — overlapping, advancing by an increment. Suited to "rate over the last five minutes, updated every ten seconds", where a tumbling window's boundary would produce a sawtooth. More expensive, since each event belongs to several windows.

Session windows — bounded by inactivity rather than by time. Suited to user-behaviour aggregation where the unit is a session of unknown length.

The property that breaks naive windowing

Event time versus processing time.

A window defined on when events arrive produces results that depend on network conditions and processing lag — the same events produce different aggregates on a rerun, which makes reprocessing inconsistent with the original and makes the result meaningless during a delay.

A window defined on when events occurred is reproducible and correct, and it introduces the problem that defines streaming: events arrive out of order and late, so the window cannot be closed simply because its end time has passed.

What follows

Watermarks — an assertion that events older than a threshold are not expected — with an explicit policy for what happens to data arriving after: dropped, sent to a side output, or triggering a window update.

Every option is a choice, and the failure is having no policy and discovering the framework's default during an incident.

The observability requirement

Every windowed metric needs an as-of timestamp and a completeness indicator. A five-second window computed over data arriving thirty seconds late reports the past as the present, and a consumer cannot tell.

For a platform where customers act on these metrics, presenting a stale aggregate as current is worse than presenting a gap.

The multi-tenant addition

Windowing multiplies cardinality. A sliding window over a high-cardinality dimension produces many window instances per key, and state grows accordingly — which is why cardinality control at ingestion matters more for streaming aggregation than for storage.