Retrieval returns 100 candidates and a cross-encoder reranks all of them. The service must hold 300 queries per second at a 400 ms p95 budget for the whole retrieval stage. Roughly what does that rerank cost in hardware and time and does it change the design?

The assumptions worth stating

• Rerank depth 100, so 300 QPS becomes 30,000 query-document pairs per second.
• A small cross-encoder, 6 to 12 transformer layers, scoring a pair truncated to about 256 tokens.
• Throughput of such a model on one modern inference accelerator with good batching: on the order of 1,000 to 3,000 pairs per second. Treat this as an order of magnitude, not a benchmark; it moves by 3x either way with layer count and sequence length.

The arithmetic

30,000 pairs per second at roughly 2,000 pairs per second per accelerator is about 15 accelerators at 100% utilisation. Nobody runs an interactive tier at 100%, so at 50% headroom for traffic peaks and rolling deploys the real number is around 30, plus replicas in a second availability zone.

Latency is the harder constraint. To reach 2,000 pairs per second the model must batch, and a batch of 100 pairs at 256 tokens is a single forward pass of perhaps 30 to 60 ms. Add batch-forming wait, network and deserialisation, and the rerank stage alone eats 80 to 150 ms of a 400 ms budget before first-stage retrieval has been paid for.

What the number rules out

Thirty accelerators for reranking on a service doing 300 QPS is almost never proportionate. The depth is the dominant term and it is the cheapest thing to change.

• Cut depth to 25. Cost falls by 4x to roughly 7 accelerators. The question is what that costs in quality, and it is measurable: compute the fraction of queries whose correct passage sits in the first-stage top 25 versus top 100. If first-stage recall at 25 is 0.94 and at 100 is 0.96, you are spending 4x to recover two points.
• Rerank only where it pays. Short navigational queries are usually resolved by the first stage. Route by query length or by the score gap between candidate 1 and candidate 5; a wide gap means the ordering is already confident.
• Cache. Support and documentation traffic is heavily repeated. A rerank result keyed by query plus candidate id set often serves 30 to 60 percent of traffic in that kind of corpus.
• Use a late-interaction model that precomputes document token embeddings, trading index size for scoring cost, if depth genuinely cannot fall.

Which assumption dominates the error

Per-accelerator throughput, and it is dominated by sequence length rather than by model size. Truncating pairs from 512 to 256 tokens roughly halves the work because attention cost grows faster than linearly. Measure your own throughput at your own truncation length before buying anything; the model choice is a second-order effect next to that.

When this is the wrong answer

At 5 QPS the entire question is moot: one accelerator, or even a CPU, reranks depth 100 comfortably, and the engineering time spent on routing and caching buys nothing. Rerank depth only becomes an architectural decision above roughly 10,000 pairs per second.

Choose depth 100 only if the measured first-stage recall gap between 25 and 100 is worth more to the product than four times the fleet; otherwise prefer depth 25 with a cache. The failure mode to watch for is not a crash: when the reranker saturates it forms larger batches, p95 climbs past the budget, and the retrieval stage silently starts returning first-stage order because the timeout fires. Alert on rerank timeout rate, not on accelerator utilisation, because a fleet that has been sized correctly in production sits near full utilisation by design.