A payments company in Razorpay's mould sees CI pipeline duration with p50 at 11 minutes and p99 at 70 minutes. Runner CPU averages 30%, the queue is almost always empty, and no single job stands out in the average. Engineers complain that CI is slow. Where do you look first, and what is the diagnosis?

The first three things I would look at, and why in that order

1. The duration distribution split by cache outcome. A bimodal distribution with a healthy p50 and a terrible p99 almost always means two populations of runs, not one slow run. The first hypothesis is dependency-cache hit versus miss, because that is the largest single difference between two otherwise identical builds.

2. What invalidates the cache, and how often. Usually the lockfile or the dependency manifest. Any pull request touching it misses, and so does every branch created from a commit after the change until the cache is repopulated.

3. Whether the cache is shared or runner-local. A cache on the runner's local disk is cold on every newly scheduled runner, which at 30% average CPU with autoscaled runners means cold caches are common precisely because the fleet is elastic.

I would deliberately not start with the queue or with CPU, because both are already reported healthy and both are the signals people chase.

The diagnosis

Two populations. Roughly 9 runs in 10 restore a warm dependency cache and finish in about 11 minutes. Roughly 1 in 10 misses, resolves and compiles dependencies from scratch, and takes 60 to 70 minutes.

The misleading clue is average CPU at 30%. It says the fleet is not saturated, which is true and irrelevant: the slow runs are not slow because they are waiting for CPU, they are slow because they are doing far more work. An empty queue says the same thing — there is no contention, so the duration is all work.

The second misleading signal is that no job stands out in the average. Averaged across hit and miss, the dependency step looks moderate. Split by outcome it is the entire story, which is why the first move is a split rather than a deeper average.

Why the p99 matters more than the p50 here

Because of who experiences it. A 1-in-10 miss rate means an engineer pushing 5 times a day hits a 70-minute build roughly every other day. The perception "CI is slow" is formed by the worst experience, not the median, and the team's reported complaint is therefore accurate while the dashboard says the pipeline is fine.

It also drives behaviour: engineers who expect an hour stop watching, batch more work per push, and merge later, which degrades everything downstream of CI.

The fix, in order

1. A shared remote cache keyed by the dependency manifest's hash, so a newly scheduled runner restores rather than rebuilds, and a miss is a one-time cost per manifest change rather than per runner.
2. Pre-warm on the default branch. Populate the cache for a manifest change once, on merge, so the branches created afterwards hit.
3. A base image containing the resolved dependencies, rebuilt when the manifest changes. This converts the miss cost into an image pull.
4. Alert on the p95 of pipeline duration and on cache hit rate, not on the mean. The mean was healthy throughout.

When this is the wrong diagnosis

If the distribution is not bimodal, this is a different problem. A long smooth tail points at variable test duration or at a flaky test being retried; a tail concentrated at a particular hour points at contention with a scheduled job; a tail correlated with one repository in a monorepo points at change-detection selecting too much work.

Check the shape before accepting the cache hypothesis — the reasoning above depends entirely on two distinct clusters being visible.

The alert that would have caught it earlier

Cache hit rate per pipeline, with a floor. It is one number, it would have sat at 90% all along, and nobody was looking because the pipeline was green and the mean was good.