RED and USE Pairing

At 3 am an engineer opens an unfamiliar service's dashboard and meets forty panels. CPU looks fine, memory looks fine, and users say it is slow. Twenty minutes go on panels that do not discriminate between hypotheses. A method fixes the order of questions so that each answer narrows the next. RED and USE are two short checklists that between them cover the request side and the resource side of any service.

RED (Rate, Errors, Duration) was described by Tom Wilkie in 2015 for microservices: for each service, measure requests per second, failed requests, and the distribution of latency. USE (Utilisation, Saturation, Errors) comes from Brendan Gregg's performance work: for each resource, how busy it is, how much work is waiting for it, and what errors it reports. Google's SRE book names four golden signals (latency, traffic, errors, saturation), which are the union of the two lists.

Why it matters

The two methods answer different questions. RED asks whether the service is doing its job for callers, and matches the complaint a user makes. USE asks which resource is the reason, which is where the fix lives. Using only RED tells you the service is slow and stops; using only USE sends you hunting through resources when the problem is a single endpoint. Ordering matters as well: starting with RED proves the symptom is real and bounds it before spending effort on machinery.

The part most often missed is saturation. Utilisation is an average, and queues form in bursts inside the average. A pool that is 40% used over a minute can be fully used for two seconds in every ten, and requests wait during exactly those seconds.

Implementation patterns

• RED per endpoint, from the request side: requests per second, error ratio and latency percentiles (p50, p95, p99) by route template, not raw path.
• USE per resource, with the saturation line defined explicitly: CPU run-queue length, memory pressure and swapping, disk queue depth, network drops and retransmits, thread pool queue, connection pool wait time and waiting callers.
• Standard dashboards generated from one template, so every service has the same first screen and an unfamiliar one is not a puzzle.
• Saturation metrics at the pools you own: time to acquire a connection (p99), queue length, rejected tasks. Add them before an incident; they do not exist by default.
• The triage order: scope (who, since when, what changed), RED on the affected endpoint, USE on resources behind it, follow the slow dependency with a trace and run RED on the callee. Mitigate (roll back or shed load) before explaining.
• Pair with traces for the dependency walk, since USE shows a local constraint and a trace shows where waiting happens.

Industry example

An archetypal ride-hailing platform has a pricing service whose p99 latency doubles each evening while CPU stays at 25%. RED shows duration up on one endpoint with errors flat. USE on the service shows no CPU or memory saturation, but the connection pool to a shared profile store shows p99 acquire time rising from 2 ms to 400 ms: a saturation signal that utilisation averages hid. A downstream slowdown from 20 ms to 200 ms needed ten times the connections for the same traffic, by Little's Law. This is an archetype used to show the method, not a documented incident at a named company.

Failure scenarios

• RED without a rate check: a failure of work that never arrives, such as clients that cannot connect, shows no errors. Alert on a drop in traffic as well as on a rise in errors.
• USE without saturation instrumentation: utilisation is green and the pool wait is invisible.
• Aggregating across endpoints: one slow endpoint at 2% of traffic disappears into a service-wide p99.
• Averages: mean latency hides the tail where users complain; use percentiles or histograms.
• Mismatched windows: a one-minute average on one panel against a ten-second view on another produces contradictory pictures.

Trade-offs

When not to use it

Batch and streaming pipelines are not request-shaped: replace RED with throughput, lag and freshness, and keep USE for the resources. For a failure with an obvious trigger, such as an error spike right after a deploy, roll back first and analyse afterwards; the method is for the case when the cause is not already known. Neither method finds a correctness bug that returns wrong answers quickly with no errors; that needs business-level checks.

Interview question

Q: You are paged for a slow service you have never seen. Describe your first ten minutes and say where RED and USE each fit.

What a strong answer covers: scoping and recent changes first; RED to confirm and bound the symptom; USE with saturation to find the constrained resource; a trace to follow waiting into dependencies; mitigation before explanation; the blind spots (traffic that never arrives, uninstrumented pools); and who communicates while someone investigates.

Quick check

Quiz: A pool is 40% utilised on average and requests still wait. Which USE term shows it? Saturation, such as p99 time to acquire a connection.

Flashcard: What does RED measure and what does USE measure? — RED measures the work a service does for callers; USE measures the resources behind it.