Multi-Tenancy and GPU Sharing

An 80 GB accelerator running a 7B model at low request rates uses a fraction of its memory and a smaller fraction of its compute. Allocating it exclusively to that workload wastes most of what was paid for, and the response is to put several workloads on one device. What that costs depends entirely on which sharing mechanism is used, and they differ far more than they appear to.

Three mechanisms

Time slicing lets multiple processes submit work to one GPU, with the driver context-switching between them. It requires no special hardware support and provides no isolation: memory is not partitioned, so one process can exhaust it and crash the others, and there is no performance guarantee, since a compute-heavy neighbour starves everyone. It suits development and notebook workloads where crashes are cheap.

Multi-Process Service allows kernels from different processes to execute concurrently on the same device rather than time-slicing, which improves utilisation for workloads that individually underuse the GPU. Memory is still shared without hard partitioning, and a fault in one client can affect others.

Multi-Instance GPU partitions the hardware itself. Supported from the A100 generation on the datacentre parts, MIG divides a device into up to seven instances, each with its own dedicated slice of streaming multiprocessors, its own memory partition and its own path to memory. Isolation is at the hardware level, so a crash or a memory exhaustion in one instance does not touch the others, and performance is predictable because the resources are genuinely separate.

The tradeoff is granularity and rigidity. MIG profiles are fixed sizes, reconfiguration requires draining the device, and the partitions cannot be pooled for a job that needs the whole GPU. A cluster with MIG enabled has decided in advance how its capacity is divided.

Where sharing is and is not appropriate

Sharing suits development environments, small-model inference, batch work with no latency target, and any workload whose GPU utilisation is genuinely low. It does not suit latency-sensitive serving, because interference produces tail latency that no amount of averaging hides, and it does not suit large training jobs, which need the whole device and its interconnect.

The measurement that decides this is not average utilisation but the shape of the utilisation over time. A workload at 30 percent average utilisation with brief full-device bursts interferes badly with a neighbour; one at a steady 30 percent shares well.

When it breaks

Memory is the binding constraint more often than compute. Two models that each fit comfortably in memory but whose activation peaks coincide will exhaust the device. Under time slicing this is an out-of-memory crash for whichever process asks second, and it is not attributable to the process that caused it.

Interference is invisible in per-process metrics. Each tenant sees its own throughput fall with no indication why. Diagnosing shared-GPU interference requires device-level metrics correlated across tenants, which most monitoring setups do not collect.

MIG partitions are not fungible. Seven small instances cannot serve one job that needs a full device, so a fleet partitioned for inference cannot absorb a training job without reconfiguration and draining. This is a capacity planning decision that looks like a configuration setting.

Sharing across trust boundaries needs care. Side channels through shared caches and memory have been demonstrated on GPUs, so co-tenanting workloads from different security domains on one device is a decision requiring more than a utilisation argument, and MIG's hardware partitioning is the relevant mechanism where it matters.