Securing Model Serving Infrastructure

In August 2025 Wiz disclosed three bugs in NVIDIA's Triton Inference Server that chained into full takeover by a remote, unauthenticated attacker (Wiz Research, 2025, Breaking NVIDIA Triton: CVE-2025-23319). An oversized request produced an error message that leaked the name of the Python backend's internal shared-memory region (CVE-2025-23320). The public shared-memory registration API accepted that name without checking whether it belonged to the caller (CVE-2025-23319). Read-write access to backend memory then yielded code execution (CVE-2025-23334). The server itself was the vulnerability, fixed in release 25.07.

Where untrusted-weights-and-deserialisation-risk asks whether an artefact can be trusted, this concept asks whether the machine running a trusted model survives hostile requests.

Three planes, three attack surfaces

The request plane takes prompts, tensors and images from users. It is where input validation, rate limits and resource ceilings belong, and where verbose errors leak internals, as the Triton chain showed.

The management plane loads and unloads models, registers shared memory, submits jobs and reports metrics. It is frequently unauthenticated by design because it was built for a trusted cluster. Ray's Jobs API executes submitted code, and for years had no authentication at all; token authentication arrived later as an opt-in. Anyscale disputed CVE-2023-48022 as intended behaviour, while Oligo documented it being exploited in the wild as "ShadowRay" and reported a renewed campaign in November 2025 (Oligo Security, 2024, ShadowRay). The dispute is instructive: a feature and a vulnerability are the same code, and the difference is whether the port is reachable.

The internal plane carries traffic between workers in a multi-node deployment. Oligo's ShadowMQ research found the same pattern copied across projects: ZeroMQ recv_pyobj() deserialising pickle from unauthenticated sockets, which is remote code execution for anyone who can reach the port (Oligo Security, 2025, ShadowMQ). It was assigned CVE-2025-30165 in vLLM's V0 engine and CVE-2025-23254 in TensorRT-LLM, which added HMAC validation. That is the same bug as a pickled checkpoint, moved from disk to the wire.

The common control is boring: management and internal ports bound to private interfaces, mutual TLS or signed messages between workers, and an authenticating gateway in front of the request plane.

Resource ceilings are a security control

An LLM request's cost is set by its length, and length is attacker-controlled. The key-value cache holds, per token,

bytes, where \(L\) is the layer count, \(h_{kv}\) the number of key-value heads, \(d_h\) the head dimension, \(b\) bytes per value, and the 2 covers keys and values. For the Llama 3 8B shape (\(L = 32\), \(h_{kv} = 8\), \(d_h = 128\)) in 16-bit precision that is \(2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072\) bytes, 0.125 MiB per token. A single 128,000-token request pins 16 GiB of accelerator memory. A few concurrent requests at maximum context exhaust a GPU that normally serves hundreds of short ones. Per-tenant caps on input length, max_tokens, concurrency and queue time are denial-of-service controls, not product settings.

Isolation between tenants

Time-slicing gives no memory isolation. NVIDIA's Multi-Instance GPU partitions supported data-centre GPUs into up to seven instances with dedicated memory and fault isolation, a hardware boundary, at the cost of fixed partition sizes.

Two incidents mark the limits. LeftoverLocals (CVE-2023-4969) showed that on affected Apple, AMD, Qualcomm and Imagination GPUs one process could read another's uncleared GPU local memory. Trail of Bits recovered about 181 MB per query from llama.cpp on an AMD Radeon RX 7900 XT, enough to reconstruct responses (Trail of Bits, 2024, LeftoverLocals). NVIDIAScape (CVE-2025-23266, CVSS 9.0) let a three-line Dockerfile setting LD_PRELOAD abuse an NVIDIA Container Toolkit hook and gain root on the host. Wiz estimated it affected 37% of cloud environments. The lesson people draw from the second is that a container sharing a kernel is not a tenant boundary for GPU workloads; a VM or microVM per tenant is.

Confidential computing, and what it does not cover

Confidential inference pairs a CPU trusted execution environment (Intel TDX or AMD SEV-SNP) with a GPU in confidential mode, such as NVIDIA's Hopper generation. Transfers over PCIe are encrypted, and the client verifies an attestation before sending a prompt. The threat it answers is the host operator: a cloud administrator or compromised hypervisor reading prompts and weights. A benchmark on Hopper GPUs measured overhead below 7%, driven mainly by encrypted CPU-GPU data transfer, and nearly zero for larger models and longer sequences (Zhu et al., 2024, Confidential Computing on NVIDIA Hopper GPUs: A Performance Benchmark Study, arXiv:2409.03992).

Whether that is worth adopting is genuinely contested. Advocates see it as the only technical basis for sending regulated data to a third-party model. Sceptics note that TEEs have a long history of side-channel breaks and that attestation proves which code booted, not that the code is safe.

When it breaks

The enclave runs the vulnerable server. Every bug in this concept, from the Triton chain to pickle on the wire, works identically inside a TEE. Confidential computing moves trust away from the operator, not away from the software.

Patching lags the disclosure. Inference containers pin framework versions for reproducibility, which is exactly what keeps a fixed CVE live in production. Rebuild cadence has to be a security SLA.

Logs become the leak. Prompt and response logging for debugging creates a store of the most sensitive data the system sees, usually with weaker controls than the endpoint.

Disputed CVEs stay open by default. When a vendor classes unauthenticated code execution as a feature, any fix tends to arrive as an opt-in setting that scanners do not check, so the real defence is network placement you enforce yourself.