A service runs 600 instances behind a load balancer with 3 zonal nodes, each probing every instance every 5 seconds. Each instance also has kubelet readiness and liveness probes at 5 seconds. The health endpoint pings the database and two downstream APIs. At 3 am the service handles 50 user requests per second. Estimate the probe traffic and say what happens when the database slows to 2 seconds.

The assumptions, stated

600 targets, 3 load-balancer nodes each checking every target, a 5-second interval, plus two local kubelet probes per instance at the same interval. Every probe runs the deep check: one database ping and two downstream calls.

The arithmetic

• Load balancer probes: 600 x 3 / 5 s = 360 requests per second.
• Kubelet probes: 600 x 2 / 5 s = 240 requests per second.
• Total probes arriving at the service: 600 per second, against 50 per second of real users. The health traffic is twelve times the user traffic at night.
• Each probe makes three downstream calls, so the database sees 600 pings per second and each downstream API sees 600 calls per second, around the clock, independent of any user.

At a healthy 2 ms per database ping, probes hold 600 x 0.002 = 1.2 connections busy on average. Invisible.

When the database slows to 2 seconds

Little's Law: concurrent connections held = arrival rate x time held = 600 x 2 = 1,200 connections, from probes alone. A typical pool across the fleet is a few hundred. The pool exhausts, probes time out, instances fail readiness and drop out of the load balancer, and real requests now queue behind the same exhausted pool. Worse, the timed-out probes retry on the next interval, so the load does not fall. The check that was meant to detect the slowdown became the largest consumer of the scarce resource.

Which assumption dominates

The fan-out multiplier. Load balancer nodes multiply, not add: three nodes means three times the checks, and many teams count the load balancer as one prober. The second is the depth of the check; a shallow check costs nothing downstream.

What the number rules in or out

It rules out deep dependency checks on the per-instance, per-prober path. The fixes, in order:

1. Make the endpoint answer from a cached result refreshed by one background task every 10 to 15 s, so downstream load is 1 check per instance regardless of how many probers ask.
2. Keep dependency health out of liveness and, for shared dependencies, out of readiness; expose it as a metric and alert instead.
3. Raise the interval to 10 s where recovery time permits. It halves the load and adds 5 s to detection.

When this is the wrong analysis

At 20 instances the same maths gives 20 probes per second, which no database notices. The problem is a function of fleet size and prober count, and a small service with a deep check is fine. Do the multiplication when the fleet passes a few hundred instances or when the dependency is shared with other services.