A reverse proxy in front of an upload service has request buffering disabled to reduce latency. What happens when 5000 clients on slow mobile links each upload a 50 MB file?

Second by second, what happens

With buffering off, the proxy opens a connection to the application as soon as the request starts and streams the body through at whatever rate the client can manage. A 50 MB upload on a 2 Mbps mobile link takes about 200 seconds. For that entire time, a worker or connection on the application is held open, doing almost nothing, waiting for bytes.

5,000 of those at once means 5,000 concurrent application-side connections for upload traffic alone. An application sized for a few hundred concurrent requests runs out of workers, and then requests that have nothing to do with uploads start queueing behind the ones that are waiting on a phone.

Where it amplifies

Little's Law explains the size of the effect without any mystery: concurrency equals arrival rate times residence time, so residence time of 200 seconds instead of 2 seconds multiplies required concurrency a hundredfold. The application has not got slower and is not short of CPU; it is short of the one resource the slow clients are consuming.

The second amplification is retries. Mobile clients on bad links disconnect and restart uploads, so some of the 5,000 are second and third attempts of the same file, consuming concurrency without producing a completed upload.

What the user sees

Unrelated pages failing or timing out, while the upload pages work slowly. That combination is the signature: the damage lands on paths that share the application's worker pool and do not themselves involve uploads.

What stops it

• Turn buffering back on. The proxy accepts the body to memory or a spool file and sends it to the application in one fast burst, so the application's residence time becomes the time to process 50 MB, not the time to receive it. This is the whole purpose of the feature, and it is why request buffering is the default in nginx, which only gained a proxy_request_buffering switch to turn it off in version 1.7.11 (2015): the proxy is built to hold thousands of slow connections cheaply, because that is all it does, while an application worker holding one costs a thread and its stack.
• Bound the spool. Buffering trades the application's concurrency for the proxy's memory and disk, so cap in-memory size, set a spool directory with a monitored disk, and limit maximum body size.
• Upload directly to object storage using a pre-signed URL, which removes both the proxy and the application from the data path and is the right architecture at this volume.
• Cap concurrent uploads per client and in total, and return a clear retry signal rather than queueing indefinitely.

When not to buffer

Buffering is wrong for streaming and for genuinely large bodies where the application must process incrementally: a server-sent-events endpoint, a live media ingest, a multi-gigabyte restore that would spool to disk only to be read straight back. The rule: buffer when the client is slow and the body is bounded; stream when the body is unbounded or must be processed as it arrives - and when you stream, size the application's concurrency for the client's speed rather than the server's.