A content pipeline auto-approves items above a confidence threshold and routes the rest to human review. Review queue p95 waiting time was 4 minutes and is now 70 minutes. Item volume is flat, reviewer headcount is unchanged, and the model has not been redeployed. Where is the time going?

The first three things I would look at, and why in that order

1. The escalation rate - the share of items routed to humans - over the same window. It is the one quantity that can change the queue without changing either volume or headcount, and almost nobody charts it.
2. The composition of escalated items: new locale, new content type, a campaign that introduced a format the model has not seen. The model is unchanged; its inputs are what moved.
3. Reviewer utilisation, not reviewer throughput. Throughput will look normal, because a saturated queue keeps everyone busy.

The diagnosis

The input distribution shifted, so more items fall below the confidence threshold, so the escalation rate rose - and a human review queue near saturation responds to a small rise in arrivals with a large rise in waiting time.

Make it concrete. Reviewers clear 500 items an hour. Escalations were 300 an hour, so utilisation was 0.6. A distribution shift pushes escalations to 470 an hour, so utilisation is 0.94. For a single queue, mean waiting time scales roughly with utilisation over one minus utilisation:

0.6 / 0.4  = 1.5
0.94 / 0.06 ≈ 15.7

A 57% rise in arrivals produces about a tenfold rise in waiting time. That is the whole explanation, and it is why "volume is flat" misleads: total volume is flat, and the volume that reaches humans is not.

The misleading signal

"The model has not been redeployed", which is true and invites the conclusion that the model is not involved. A fixed threshold on a model's confidence is a policy whose behaviour depends on the input distribution, so an unchanged model plus changed inputs is a changed system. The second misleading signal is reviewer throughput, which is at its maximum precisely because the queue is full.

The fix, in order

1. Shed or prioritise now. Define what happens when the queue exceeds a target age: auto-approve the lowest-risk categories, or hold the lowest-value ones. An unbounded queue is a decision nobody made.
2. Make the threshold adaptive to a target escalation rate rather than a fixed confidence value. The business constraint is reviewer capacity, so the control variable should be the escalation rate, with the confidence threshold derived from it. This is the change that prevents recurrence.
3. Size for utilisation, not for the mean. A human queue should run at 70 to 80% utilisation. Above that, waiting time is dominated by the queue rather than by the work, and every small arrival change is an incident.
4. Then investigate the distribution shift and decide whether it needs retraining, a new category or different guidance - on a normal timeline, because the queue is no longer on fire.

The alert that would have caught it earlier

Escalation rate, with its own threshold, charted next to reviewer utilisation and age-of-oldest-item. Not queue length, which depends on throughput, and not average wait, which lags. Escalation rate would have moved hours before the wait time did, because the queue absorbed the change until it could not.

Common weak answers

• "Hire more reviewers." It works and it is the slowest and most expensive lever, and it will be needed again at the next distribution shift. Prefer fixing the control variable first, then size capacity to the target escalation rate.
• "Retrain the model." Days to weeks, while the queue is 70 minutes behind now. Do it after shedding and after the threshold is adaptive.
• "Lower the confidence threshold to escalate less." Directionally right and dangerous as a reflex: it trades review coverage for wait time with no statement of what error rate the business accepts. That trade-off has to be made explicitly, with the auto-approved error rate measured, since this is a published-content path where a wrong auto-approval costs more than a slow review.
• "Add an alert on queue length." Length depends on throughput, so the threshold is wrong at every volume. Age and escalation rate are the signals.