A service's p99 latency has tripled over three months with no single obvious change. How do you investigate?

Establish the shape before touching anything

Is it everything or something? Break the metric down by endpoint, tenant, region, instance and version. A tripling spread evenly across all traffic points to a shared resource; a tripling concentrated on one endpoint or one tenant is a different investigation entirely.

Is it a step or a slope? A step change has a cause on a date — a deployment, a configuration change, a dependency version, a data volume threshold crossed. A gradual slope points at growth: data volume, cache working set, a queue slowly deepening. The distinction determines where to look and it takes two minutes.

Is it the tail only? p99 tripling with a flat p50 means a subset of requests is affected — cold caches, GC pauses, lock contention, retries, one bad instance. p50 moving too means everything is slower.

Then walk the request path top-down

Metrics to establish scope. Traces to find which hop consumed the time. Logs filtered by trace ID for that hop's reason. Profiles if the hop is slow with no obvious cause.

Each transition depends on correlation being in place; without it this becomes four searches joined by timestamps and guesswork.

And audit resources bottom-up

For every resource — CPU, memory, network, disk, and the software ones people forget: thread pools, connection pools, locks — check utilisation, saturation and errors.

Saturation is the strongest signal and the least instrumented. Utilisation at 100% may be healthy; queued work is not. And errors are frequently discarded silently — dropped packets, pool timeouts, allocation failures.

The causes that produce this specific pattern

Data volume growth — a query whose plan flipped when statistics went stale or a threshold was crossed, and which produces exactly this "fast for months, then slow, no deployment" signature. Compare estimated against actual row counts in the plan.

Cache hit rate decay — the working set outgrew the cache. A fall from 99% to 95% multiplies backend load by five, which is enough to move latency substantially with nothing else changing.

Utilisation creep. Waiting time scales as 1/(1−utilisation). Growth from 70% to 90% roughly doubles queueing delay with no code change anywhere.

Accumulated small regressions — no single commit responsible, which is exactly the condition a performance budget in CI exists to prevent.

Dependency degradation, which is someone else's regression arriving as yours.

Fix in the right order

Only the bottleneck's improvement counts, and the bottleneck moves after each fix. Amdahl's Law bounds the return: a component taking 5% of the time cannot yield more than 5%, however good the optimisation.

So: measure, fix the dominant term, re-measure, repeat. And add a performance budget in CI for the endpoint, so the next three months of erosion fails a build instead of becoming another investigation.