Inference & Serving
24 min
Goodput, Not Throughput: The Metric That Decides Whether Your LLM Deployment Works
Two servers run the same model on the same GPUs. One reports 4,200 tokens per second and is unusable; the other reports 2,600 and feels instant. Throughput is a property of the server, latency is a property of the request, and th…