Multi-Tenant Serving and Isolation

Phase 1 of any LLM product: one model, one prompt template, one API key, every tenant gets the same behaviour. Cheap, simple, the right starting point. Phase 2: enterprise customers demand their own behaviour - their tone, their vocabulary, their refusal rules, their internal knowledge. You have three options - prompt-engineer per tenant (cheap, low ceiling), fine-tune a model per tenant (expensive, high ceiling), or serve thousands of LoRA adapters on top of one shared base model (the middle path that S-LoRA and vLLM made practical). The choice is fundamentally about where you put the isolation boundary.

The simple case - shared model, prompt-level isolation

One base model. Every tenant's request goes through the same vLLM instance. Tenant identity is carried in the prompt (system message: "You are an assistant for ACME Corp") and in metadata (for routing, rate limits, audit).

This works for 80% of multi-tenant products and almost all SaaS chat applications. Make it your default and only move off it when a specific tenant pays you enough to justify it.

Per-tenant LoRA adapters

LoRA (Hu et al. 2021) fine-tunes a model by training low-rank update matrices A and B such that the new weight is W' = W + alpha * B @ A where A and B are tiny (rank 8-64 instead of full d-by-d). A LoRA adapter for a 7B base model might be 20-100 MB instead of 14 GB for the full model.

The serving question: can you swap LoRA adapters per request without reloading the base model? Naively, no - you would have to merge the LoRA into the base weights, which takes seconds. With dedicated infrastructure, yes - you keep the base weights in HBM, keep N LoRA adapters in HBM (or paged from CPU), and dispatch each request through the appropriate adapter at the matmul level.

S-LoRA - the paper that made it work at scale

S-LoRA (Sheng et al., arXiv:2311.03285) demonstrated serving thousands of LoRA adapters on a single GPU with small overhead. The core contributions:

1. Unified paging. Both the KV cache and the LoRA adapter weights live in one unified memory pool, allocated in fixed-size blocks. This solves the fragmentation that destroys naive LoRA serving when adapters vary in rank.
2. Heterogeneous batching. A single batch can mix requests against different LoRA adapters. Custom CUDA kernels apply the right adapter per request without breaking batched matmul.
3. Tensor-parallel sharding of both base weights and adapter weights across GPUs.

Reported throughput: up to 4x over the next-best baseline while supporting several orders of magnitude more adapters than HuggingFace PEFT or vLLM's pre-S-LoRA path. The S-LoRA reference implementation has been archived; the ideas were absorbed into vLLM and SGLang.

vLLM's LoRA serving

vLLM supports per-request LoRA via LoRARequest. Adapters are passed at server start with --lora-modules or loaded/unloaded at runtime via API endpoints. Constraints worth knowing:

• All adapters in a deployment share max_lora_rank - set it to your tallest rank to avoid retraining.
• LoRA inference adds 1.5-3x latency overhead vs the base model in single-request mode; this shrinks rapidly with batching.
• Adapter weights stream from CPU to GPU on first use; the second request to the same adapter is fast.

Continuous batching across tenants

Continuous batching (the Orca / vLLM scheduling trick) is what makes shared serving practical: each decoding step processes all in-flight sequences, regardless of which tenant they belong to. The implication for multi-tenancy:

• A tenant with a 10k-token prompt mixes in the same batch as a tenant with a 50-token prompt.
• Per-step throughput depends on the batch composition, not on any single tenant's traffic.
• Long-prompt tenants effectively get a fair share of the bandwidth - no special-casing needed.

The combination of continuous batching + S-LoRA-style per-request LoRA gives you the best of both worlds: shared compute, per-tenant behaviour, no model duplication. This is the modern default for serving fine-tuned models at any non-trivial tenant count.

Data isolation guarantees - what you can and cannot claim

In a shared-model deployment, every tenant's prompt and response transits the same process. You cannot honestly claim cryptographic isolation between tenants on a shared model. What you can claim:

If a tenant requires "no shared inference process," the answer is a dedicated deployment (separate vLLM instance per tenant). That is the cost-vs-isolation trade-off in its purest form.

Noisy neighbour mitigation

The same noisy-neighbour problem from the gateway layer recurs at the serving layer. A tenant submitting a 100k-token prompt holds up the prefill queue for everyone. Mitigations:

• Chunked prefill (vLLM, SGLang, TensorRT-LLM). Slice long prompts into 512-2048 token chunks and interleave them with ongoing decodes. A single long prefill no longer blocks the batch.
• Per-tenant in-flight cap. Limit each tenant to N concurrent sequences in the engine. Excess requests queue at the gateway, where they can be load-shed cleanly.
• Priority lanes. Run two vLLM instances: a "fast lane" sized for small prompts with tight latency SLOs, a "bulk lane" for long-context and batch work. Route by request shape.
• Separate deployment for the worst offender. If one tenant consistently submits 100k-token prompts, give them their own engine.

The cost-vs-isolation trade-off

The right design is rarely one row of this table - it is two or three rows, with tenants assigned by tier. A typical mature deployment runs:

• Free and SMB on shared model + per-tenant prompt.
• Mid-market on shared model + per-tenant LoRA, served from a multi-LoRA vLLM cluster.
• Enterprise on dedicated model instances in their preferred cloud / region.
• Regulated on dedicated GPU pools with attested boot and air-gapped network.

When it falls down

• LoRA adapter explosion. "We have 10,000 tenants and each has a custom adapter" is the point at which adapter storage (CPU RAM or NVMe paging) becomes a real cost, not just a footnote. Plan capacity for adapter weights, not just base weights.
• LoRA training pipeline is the bottleneck. Serving thousands of LoRAs is easy; producing thousands of well-trained LoRAs is hard. Most teams over-invest in serving infrastructure and under-invest in the data and training loop that feeds it.
• Audit and compliance under shared inference. A tenant's auditor wants to know their data never co-mingles. On a shared engine, technically it does (in HBM, briefly, during decode). Document this honestly and offer dedicated deployments for tenants who need a different answer.
• Cache cross-tenant leakage. If you enable prefix caching without scoping the cache key by tenant, tenant A can hit a cache entry written by tenant B's identical prompt. Always scope cache keys by tenant_id.

Further reading

• S-LoRA: Serving Thousands of Concurrent LoRA Adapters - Sheng et al. 2023 - the paper, including unified paging and heterogeneous batching.
• S-LoRA reference implementation - archived but worth reading for the kernel design.
• vLLM LoRA documentation - the production path for per-request LoRA in a continuously-batched server.
• Efficient Memory Management for Large Language Model Serving with PagedAttention - the vLLM paper; continuous batching across tenants is built on this.
• Anyscale: How continuous batching enables 23x throughput in LLM inference - the practitioner-level explanation of why mixed-tenant batching works.