Serving Many LoRA Adapters

A company fine-tunes Llama-3 70B for 500 enterprise clients, one per customer domain. Each adapter is roughly 50 MB. The base model is 140 GB in BF16. Naively, serving each adapter in its own GPU instance would require 500 x 140 GB of VRAM - an absurd number. The practical question is: how do you share one copy of the base model weights across all 500 adapters while routing each incoming request to the right fine-tuned variant, with throughput close to single-model serving?

That question is what the LoRA serving literature addresses, and the answer is more subtle than it first looks.

Why Naive Batching Breaks

Standard batched inference works because all requests in a batch share identical weight tensors: every matrix multiply is the same operation applied to different input vectors, so a single GEMM kernel covers the whole batch. The moment you introduce per-request adapter weights, that assumption collapses.

Recall the LoRA parameterisation for a weight matrix W:

output = x W^T  +  x (A B)^T
                    ^^^^^^^^
                    adapter contribution

where A (d x r) and B (r x k) are the low-rank matrices, and r is the rank (typically 4-64). Different requests in the same batch have different A and B. A naive loop over requests (one GEMM per request) serialises the adapter contributions and destroys GPU utilisation. You need a kernel that can compute the adapter contributions for all requests in a heterogeneous batch in one pass.

The second problem is memory. If you hold all adapter weights in GPU HBM simultaneously, even at rank 16 across all transformer layers you quickly exhaust VRAM for large adapter counts. The system must page adapters between CPU DRAM and GPU HBM based on which adapters are currently scheduled.

Punica: The Segmented Gather-Scatter Kernel

Punica (Chen et al., 2023) introduced the key kernel primitive: Segmented Gather Matrix-Vector Multiply (SGMV). The idea is compact enough to state precisely.

Given a batch of N requests, group them by adapter identity. For each group g with requests indexed by a segment index:

# pseudocode for the adapter contribution pass
for each request i in batch:
    y[i] += x[i] @ A[adapter_id[i]] @ B[adapter_id[i]]

The SGMV kernel executes this in a single GPU launch by treating the adapter index as a gather index into a packed tensor of A and B matrices. All requests sharing the same adapter_id hit the same memory region, enabling cache reuse. Different adapter groups run as independent warps within the same kernel.

The result: a GPU holds exactly one copy of the base model. The adapter matrices for the currently scheduled requests are loaded into a side buffer, and the SGMV kernel stitches their contributions onto the base model's output in one shot. Punica reported 12x higher throughput versus naive vLLM serving of multiple LoRA models on the same hardware.

S-LoRA: Unified Paging and Tensor Parallelism

S-LoRA (Sheng et al., 2023) extended the Punica insight to production-scale systems - thousands of adapters across multiple GPUs. Three mechanisms matter:

Unified memory pool. Instead of allocating fixed GPU buffers per adapter, S-LoRA treats adapter weight pages and KV-cache pages as fungible resources managed by a single allocator (inspired by PagedAttention in vLLM). This eliminates internal fragmentation when adapters have heterogeneous ranks: a rank-8 adapter and a rank-64 adapter can coexist in the same pool without wasting chunks of memory.

CPU offloading with prefetch. Adapters not currently in a scheduled batch live in CPU DRAM. The scheduler predicts which adapters will be needed in the next iteration and begins prefetching them over PCIe while the GPU is computing the current batch. The overlap hides most of the transfer cost when request arrival is bursty.

Tensor parallelism for adapters. When the base model is sharded across multiple GPUs (tensor parallelism), the A and B matrices must also be split consistently. S-LoRA introduces a partitioning scheme where A is column-partitioned and B is row-partitioned to match the column-parallel linear layers common in transformer inference, avoiding extra all-reduce collectives on the adapter path.

Taken together, S-LoRA demonstrated serving thousands of LoRA adapters on a single GPU or across multiple GPUs with up to 4x throughput improvement over HuggingFace PEFT and naive vLLM, while the memory overhead per additional adapter was linear only in the adapter size (not the base model size).

vLLM's Built-in LoRA Support

For teams who do not want to run a custom fork, vLLM (from v0.3 onward) ships native multi-LoRA serving. The key configuration parameters are:

Adapters are registered at server start via --lora-modules name=path and exposed as separate model names through the /v1/models endpoint. A request selects its adapter by passing the adapter's registered name as the model field, so existing OpenAI-compatible clients need zero code changes.

Dynamic loading (registering adapters at runtime without restart) is also supported but carries a security caveat: an arbitrary path can be passed to the engine, so dynamic loading should only be enabled in isolated, fully trusted environments.

When It Falls Down

Rank heterogeneity overhead. The SGMV kernel's efficiency degrades when the batch contains many different ranks simultaneously. Unified paging mitigates memory fragmentation, but the kernel itself pads lower-rank adapters to the maximum rank in the batch. A workload mixing rank-4 and rank-64 adapters in the same batch pays a 16x compute penalty on the rank-4 adapters' contribution pass.

Cold-start latency spikes. If an adapter is not in GPU HBM when its first request arrives, it must be fetched from CPU DRAM (PCIe bandwidth: 16-64 GB/s, vs. HBM bandwidth: 1-3 TB/s). For large adapters or low-bandwidth PCIe slots, this can add tens of milliseconds to the first token latency - noticeable in interactive applications.

Tensor parallelism at high TP degrees. Splitting adapters across 8 or 16 GPUs with high rank creates non-trivial communication volume for the B matrix all-reduce. The overhead grows with TP degree and rank, and can outweigh the compute benefit for small batch sizes.

Adapter count vs. accuracy. This is a systems paper concern, not a modelling one, but worth stating: the serving infrastructure does nothing to prevent two adapters from conflicting at the semantic level if they were fine-tuned on overlapping domains. Routing logic - deciding which adapter to serve a given request - is the operator's responsibility and can silently degrade quality if misconfigured.

Base model quantisation interactions. Combining 4-bit base models (QLoRA style) with multi-adapter serving adds complexity. The adapter weights are in FP16/BF16 while base model activations pass through dequantisation; the SGMV kernel must handle the mixed precision path, and not all serving systems implement this correctly.

Further Reading

• Sheng, Y. et al. (2023). S-LoRA: Serving Thousands of Concurrent LoRA Adapters. arXiv:2311.03285. The primary reference for unified paging and tensor-parallel adapter serving.
• Chen, L. et al. (2023). Punica: Multi-Tenant LoRA Serving. arXiv:2310.18547. Introduces the SGMV kernel that underlies most efficient multi-adapter batching.
• Hu, E. J. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685. The foundational paper; understanding the parameterisation is prerequisite to understanding the serving problem.
• vLLM LoRA documentation: https://docs.vllm.ai/en/latest/features/lora.html. Practical configuration guide for production deployments.