A pipeline moves from writing records one at a time to batches of 500, and throughput rises roughly twentyfold. What has been given up, and at what batch size does the trade stop paying?

What is gained, and why

Each write pays a fixed cost — a network round trip, a transaction begin and commit, a durable log append, index maintenance — plus a variable cost per record. Batching amortises the fixed cost across the batch.

With a 2 ms fixed cost and a 20 µs per-record cost: one record takes 2.02 ms, about 495 per second. Five hundred records take 2 + 10 = 12 ms, about 41,000 per second. That is the twentyfold improvement, and it is why batching is the first lever in every ingestion path.

What is paid

• A latency floor. A record now waits for the batch to fill or the flush timer to expire. During quiet periods nothing is ever faster than the timer, so p50 becomes roughly half the flush window and p99 becomes the whole window.
• Blast radius. A failed batch is 500 failed records. If one poison record causes it, you either fail all 500 or you need per-record error reporting and a retry path that splits the batch — which most client libraries handle badly or not at all.
• Memory, at the worst moment. In-flight batches are held in the producer: batch size × partitions × producer instances. The failure is an out-of-memory kill during a downstream slowdown, which is exactly when batches are largest and retries are accumulating.
• Duplicates on retry. Retrying a partially applied batch without idempotency duplicates the applied portion, and partial application is the normal failure mode for a batch that timed out.

Where the trade stops paying

When the variable cost begins to dominate the fixed cost. Once batch_size × per_record_cost greatly exceeds fixed_cost, doubling the batch doubles the latency and adds almost nothing to throughput.

The knee is at roughly fixed_cost ÷ per_record_cost records. In the numbers above that is 2 ms ÷ 20 µs = 100 records, at which point the amortised fixed cost per record has fallen to 20 µs, equal to the variable cost. Going from 100 to 500 buys perhaps another 20% of throughput while multiplying latency and blast radius by five.

Compute that ratio rather than guessing a round number, then cap the result by the latency budget and by the memory you are willing to hold, and always pair it with a flush timer so a quiet period cannot hold records indefinitely.

When not to batch at all

When per-record failure semantics matter more than throughput. A payment instruction, a legal notification, a trade: the operator needs to know precisely which one failed, and "batch 4411 failed" is not an acceptable answer. The same applies wherever a record must be durable before a user-visible acknowledgement, because the batch window is dead time the user is waiting through.