Batch and Asynchronous LLM Workloads

A nightly job classifies two million support tickets, each about 1,200 input tokens with a 50-token label. That is 2.5 billion tokens. Against a hypothetical limit of 2 million input tokens per minute, the synchronous endpoint needs 1,250 minutes of perfectly shaped traffic, almost 21 hours, and pays full price for every token. Nobody is waiting for the answers, yet the job is built as if someone were.

Offline work is a different product from interactive inference, and providers price it that way. The design questions become throughput, deadlines, duplicate suppression and the requests that never came back.

What the batch contract trades

OpenAI's Batch API charges 50% less than the synchronous endpoints, promises completion within a 24-hour window, accepts up to 50,000 requests or 200 MB per batch, and draws on a rate-limit pool separate from the per-model limits (OpenAI, Batch API guide). Anthropic's Message Batches API also discounts by 50%, caps a batch at 100,000 requests or 256 MB, reports that most batches finish within an hour, expires unfinished work at 24 hours and keeps results for 29 days (Anthropic, Batch processing).

The price of a job becomes

where \(N\) is the number of requests, \(t^{\text{in}}_i\) and \(t^{\text{out}}_i\) are the input and output tokens of request \(i\), and \(p_{\text{in}}\), \(p_{\text{out}}\) are the per-token list prices. What you give up is any guarantee tighter than the window: 24 hours is a ceiling, not a service level, and the next stage must be designed around it.

Two contract details shape the code. Results arrive in arbitrary order, so every request carries a custom_id and joins back to its input by that key. And completion is per request, not per batch: a finished batch can hold successes, errors and expired requests side by side, and you are billed only for what completed.

Shaping traffic when a day is too long

When the deadline is hours rather than a day, the job runs against the synchronous endpoint and the constraint becomes the rate limiter. Anthropic documents that its limits use the token bucket algorithm, with capacity replenished continuously rather than reset each minute, and that a per-minute limit may be enforced over shorter intervals (Anthropic, Rate limits). A bucket with capacity \(B\) refilled at rate \(r\) admits at most \(B + rT\) units in any interval of length \(T\), so bursts beyond \(B\) fail no matter how idle the previous minute was.

The useful planning quantity is sustainable concurrency. With a token limit \(L\) per minute, a mean of \(\bar{t}\) tokens per request and a mean request duration of \(d\) seconds,

At \(L = 400{,}000\), \(\bar{t} = 1{,}250\) and \(d = 4\) seconds, the limit allows 320 requests per minute, and about 21 workers in flight keep you at it. Running 100 workers does not go faster; it converts the excess into 429 responses and retries.

Retries themselves need shaping. Exponential backoff alone synchronises clients that failed together, so they collide again on the next attempt. Full jitter draws the sleep uniformly from \([0, \min(\text{cap}, \text{base}\cdot 2^{a})]\) for attempt \(a\), which spreads retries out and reduced total work in Brooker's simulations (Brooker, 2015, Exponential Backoff and Jitter, AWS Architecture Blog). A retry-after header, when present, overrides the local schedule.

Idempotency and reconciliation

Queues deliver at least once: a worker that crashes after the model call but before acknowledging will see the message again. The standard construction derives a key from everything that determines the output, for instance \(k = \mathrm{hash}(\text{model snapshot}, \text{prompt version}, \text{input id})\), writes results with an upsert on \(k\), and gates any external side effect on the same key. It is the pattern Stripe popularised for payment APIs, where a retried charge with the same idempotency key is executed once (Leach, 2017, Designing robust and predictable APIs with idempotency, Stripe).

Reconciliation closes the loop. Treat each request as a small state machine: pending, submitted, succeeded, failed or expired, resubmitted. After each batch, diff the returned custom_id set against the submitted set, route failures to a retry batch, and move requests that fail repeatedly to a dead-letter store for a person to inspect. A job is finished when every key is terminal, not when the provider says the batch ended.

Contrast fallbacks, timeouts and degradation, where a user is waiting; here the retry budget buys completeness, not latency.

When it breaks

Deadlines compound. A map step and a reduce step, each in a batch, is a 48-hour worst case.

A bad prompt fails at scale. A synchronous job surfaces a broken template after ten requests; a batch surfaces it after 100,000 billed completions. A small canary batch first is cheap insurance. This is also why some practitioners prefer a shaped synchronous queue at double the price, valuing early failure detection and ordering control, while others argue the separate rate-limit pool alone justifies batches. The answer turns on what a wasted run costs relative to the token bill.

Caching does not compose as neatly as the price sheet suggests. Anthropic states that batch and prompt-caching discounts stack, but because batch requests run concurrently, cache hits are best-effort and typically range from 30% to 98%. A cost model that assumes the top of that range will be wrong.

Snapshots drift inside a job. Resubmitted requests that reference an alias rather than a dated snapshot may run on a newer model, mixing two labelling functions in one dataset.

Duplicates leak through side effects. Idempotent storage does not help if the worker emailed a customer before writing the row. Any action outside the result store needs its own key check.