Confidential Computing for Model Inference

Encrypt the disk, encrypt the wire, and there is still a window where the prompt exists as plaintext: inside the machine that is answering it. During a forward pass the tokens, the KV cache and the weights sit unencrypted in GPU memory, addressable by anything with enough privilege on the host. A hypervisor can read them. A compromised driver can read them. So can an insider with a debugger attached to the serving process. That gap is why an enterprise legal team can be satisfied by TLS and AES-256 at rest and still refuse to send a contract to a hosted model.

Confidential computing closes the gap by moving the trust boundary inside the silicon. The workload runs in a hardware-enforced enclave whose memory is encrypted with a key the CPU or GPU generates and never exports, and the platform will produce a signed statement about exactly what is running inside. The host operating system, the hypervisor and the cloud operator are all outside the boundary and are all assumed hostile.

Two halves of one boundary

For LLM inference the enclave has to span two devices, and they are protected by different mechanisms.

On the CPU side, the process runs inside a confidential VM: an AMD SEV-SNP guest, an Intel TDX trust domain, or an Arm CCA realm. Each encrypts guest memory with a per-VM key held in a security processor, and each adds a page-ownership structure that stops the hypervisor remapping or replaying guest pages rather than merely reading them (Misono et al., 2024, Confidential VMs Explained: An Empirical Analysis of AMD SEV-SNP and Intel TDX, ACM SIGMETRICS).

On the GPU side, an H100 booted in confidential mode walls off its HBM as a Compute Protected Region. Compute engines run entirely inside it; the DMA engines are the only user-accessible path across the wall, and hardware forces everything they write outward to be encrypted and authenticated with AES-GCM-256 (Dhanuskodi et al., 2023, Creating the First Confidential GPUs, ACM Queue 21(4)).

Between the two sits the part that costs you. PCIe is not a trusted link on Hopper, so data does not move directly from confidential VM memory into protected HBM. It is staged through a shared, unprotected bounce buffer: the CVM encrypts a chunk into shared pages, the GPU DMAs it in and decrypts and authenticates it inside the CPR, and the same dance runs in reverse for results. Every byte crossing the boundary is encrypted twice and copied an extra time.

The cost lands on transfer, not compute

Because the protection inside the CPR is architectural rather than arithmetic, kernels run at full speed. The overhead is a property of the bus, which makes the cost profile counter-intuitive: it shrinks as the model gets bigger, because a larger model does more compute per byte transferred.

A benchmark study of Hopper confidential mode found average overhead below 9%, with most realistic LLM queries under 5% and the penalty approaching zero for large models and long sequences, and attributed the remainder to data transfer rather than in-GPU computation (Zhu et al., 2024, Confidential Computing on NVIDIA Hopper GPUs: A Performance Benchmark Study, arXiv:2409.03992). End-to-end serving numbers are worse than microbenchmarks, as they usually are: measured under Intel TDX at fixed request rates, confidential mode raised average time-to-first-token by 21.8% for Mistral-7B and 27.8% for Qwen3-30B-A3B, with global token throughput down 17.7% and 21.1% (Benchmarking Confidential GPU Inference on NVIDIA H100 under Intel TDX, 2026, arXiv:2607.19353).

Blackwell changes the shape of the problem rather than the size of the constant. NVIDIA describes it as the first TEE-I/O capable GPU, with inline protection over NVLink and AES-GCM engines running at line rate, which removes the bounce buffer when the host supports TEE-I/O as well. Independent measurement puts well-configured Blackwell confidential inference at low single-digit percentage overhead, while noting that stock inference stacks give up 30% to 40% for configuration reasons rather than architectural ones (Benchmarking Confidential Computing Performance on NVIDIA Blackwell GPUs, 2026, arXiv:2608.26575). Treat the vendor's "nearly identical performance" as the ceiling and the stock-configuration number as what you will actually get on the first day.

What is inside the boundary, and what is not

Confidential computing protects data in use against the infrastructure. It does not protect anything against the code you chose to run. If the serving image logs prompts, ships them to an analytics endpoint or writes them to a volume outside the enclave, the enclave dutifully encrypts memory while the application leaks. The guarantee is "the operator cannot read what the enclave is computing", and it is only as strong as the claim about which enclave image is running, which is a separate mechanism entirely (see remote attestation for AI workloads).

It also protects nothing about the output. A model that memorised training data still emits it; an API that answers membership-inference queries still answers them. Encrypted memory and differential privacy solve orthogonal problems, and conflating them is the most common mistake in procurement conversations about "private AI".

When it breaks

The TCB is large and is not yours. Inside the boundary you trust the CPU vendor's microcode and security processor, the GPU firmware, the vGPU stack, the guest kernel and the inference server. A firmware bug is not patchable by you, and a vulnerability there is a vulnerability in the whole guarantee.

Availability and residency are unchanged. The operator cannot read your prompts; the operator can still delete your VM, throttle it, or refuse to run it. Confidentiality is not availability.

Side channels survive memory encryption. Timing, power, interrupt patterns and bus traffic are outside the threat model of every shipping CVM, and several published attacks recover meaningful secrets from them (see side channels against confidential inference).

Capacity is the practical limiter. Confidential GPU instances are a narrower SKU than ordinary GPU instances, in fewer regions. Azure's NCC H100 v5 confidential VMs, built on AMD EPYC with SEV-SNP plus H100 NVL, reached general availability in 2024 in two regions (Microsoft, 2024, General Availability: Azure confidential VMs with NVIDIA H100 Tensor Core GPUs). Check capacity before promising a customer an attested deployment.