An internal API call takes 40 ms consistently while the server reports 2 ms of processing. Where is the time going?

What is being tested

Whether you can diagnose a consistent, suspiciously round latency floor — which almost always indicates a protocol or configuration interaction rather than work being done.

The characteristic suspects

1. Nagle's algorithm interacting with delayed acknowledgement. The classic 40 ms floor, and the number itself is the clue — 40 ms is a common delayed-ACK timer.

Nagle buffers small writes until the previous data is acknowledged. Delayed ACK holds acknowledgements briefly hoping to piggyback them on outgoing data. Together they deadlock briefly: the sender waits for an ACK before sending, the receiver waits for data before acknowledging, and the delayed-ACK timer resolves it — at 40 ms.

This appears in request/response protocols that write headers and body as separate small writes. Resolved by disabling Nagle (TCP_NODELAY) or by writing the message in a single call, which is the better fix.

2. A new connection per request. TCP handshake plus TLS is two to three round trips. On a low-latency internal network that is a few milliseconds, but with any distance it accumulates quickly. Connection pooling and keep-alive eliminate it.

3. DNS resolution per request, if not cached — or cached forever by a runtime that ignores TTL, which produces a different and worse problem.

4. A proxy or mesh sidecar in the path, adding two userspace traversals plus its own buffering. Two proxy hops is inherent in a sidecar mesh; more than two means something is misconfigured.

5. Cross-availability-zone routing on every call, adding both latency and transfer charges. Zone-aware balancing removes it.

6. Timer or scheduling granularity in a batching layer waiting for a fixed interval.

How to confirm

• Packet capture on both ends, which shows the gap directly and identifies which side is waiting.
• Compare with a raw socket test between the same hosts to establish the network floor.
• Test with TCP_NODELAY set, which confirms or eliminates Nagle in one experiment.
• Check whether latency is constant or variable. A consistent floor points at a timer or protocol interaction; variable latency points at queueing or contention.

The general principle

Consistent, round-numbered latency is a timer, not work. Work produces a distribution; timers produce a floor. 40 ms, 100 ms, 200 ms and 1 second all suggest a specific configured or protocol interval rather than something being computed.

The measurement point worth naming

Server-side timing reports 2 ms because it measures only handling. It excludes connection setup, TLS, queueing before the handler, and transfer. Measure at the client — the difference between the two numbers is precisely the thing you are looking for, and this comparison is the fastest diagnostic available.