Etsy released StatsD in 2011: a small daemon that receives metric samples from applications over UDP, aggregates them in memory, and flushes to the metrics backend on a fixed interval. Why was UDP the right choice for that design, and what did it make permanently impossible?

The situation Etsy was in

Etsy's engineering culture at the time was built on measuring everything and deploying continuously, which only works if adding a measurement is close to free for the engineer adding it. Any instrumentation call that could block, throw, or slow the request path is a call that reviewers will argue about and engineers will omit.

What they chose and why it fit

UDP is fire-and-forget. The application writes a datagram and continues. There is no connection to establish, no acknowledgement to wait for, no retry, and no error to handle. That has three consequences that were the entire point:

• Instrumenting a hot path cannot add latency. The send is a syscall with no round trip.
• A metrics outage cannot become a product outage. If the daemon is down, datagrams go nowhere and the application does not notice. Compare a TCP-based client with a bounded queue, where a slow backend eventually applies back-pressure into the request path.
• The cost of a new metric falls to one line, so the number of metrics grew to the tens of thousands — which was the cultural goal, not a side effect.

Aggregation in a separate local daemon rather than in the process also meant one flush per metric per interval regardless of request volume, so the backend saw a rate set by the interval rather than by traffic.

What it cost them

• Loss is silent and unmeasurable. Datagrams are dropped by a full socket buffer, by the kernel under load, and by any network hop in between. You cannot distinguish a lost sample from a real zero, which means counters are approximate at precisely the moments — overload — when you most want them exact.
• Client-side sampling distorts the distribution. StatsD's sample-rate mechanism scales counters back up correctly, but a timer sampled at 10% gives percentiles computed from a tenth of the events, and the tail is the part of the distribution most damaged by thinning.
• Percentiles are computed per daemon and cannot be recombined. A p99 from each of 200 hosts cannot be averaged into a fleet p99; this is a property of quantiles, not of StatsD, but the architecture bakes it in.
• The request context is gone. By the time a sample reaches the daemon it is a name and a number. There is no trace id, no tenant, no build. You can see that the tail moved and never which request moved it.

What replaced it, and why

In-process histograms with fixed buckets — the Prometheus and OpenTelemetry model — keep the distribution as counts per bucket, which do aggregate correctly across instances, and exemplars attach a trace id to a bucket so the jump from "p99 rose" to "here is one of those requests" is one click. That recovers exactly what the daemon discarded.

Where copying it would be a mistake, and when not to make this trade

Never put a number that has to be right on a lossy transport. Billing counters, compliance evidence and anything that reconciles against money belong in an acknowledged write path. The StatsD trade is correct for operational signals where a 0.5% error is irrelevant and a blocked request path is unacceptable — and it is the wrong trade the moment someone starts invoicing from the number.