An enterprise AI platform serves many customers on shared inference infrastructure. What isolation is required, and where is the hardest boundary?

The isolation required

• Data isolation in the request path, which is straightforward — requests do not share state.
• Retrieval isolation, which is not. If the platform provides retrieval over customer data, the index must enforce tenancy at query time, and a filter applied in application code is one forgotten clause away from a cross-tenant disclosure. Separate indexes per tenant are stronger and more expensive; enforced filtering in the engine is the workable middle.
• Capacity isolation, so one customer's burst does not starve others. Per-tenant concurrency limits and token-rate quotas, weighted by cost rather than by request count — request counting treats a short prompt and a very long one identically, which is exactly backwards.
• Fine-tuned model isolation, where a customer's adapted model is derived from their data and must never serve another customer. This is a deployment and routing concern with a severe failure mode.

The hardest boundary

Training and improvement. Whether customer data may be used to improve models is a contractual and regulatory question with an architectural consequence: if the answer is no, the data path must make it structurally impossible rather than policy-prohibited, because a policy is not evidence.

That means separate storage, separate access controls, and demonstrable lineage showing which data contributed to which model — which is a substantial engineering commitment and is what enterprise customers actually audit.

The residency dimension

Enterprise customers frequently require that their data — and their inference — stays in a jurisdiction. That turns a global inference fleet into a set of regional ones with independent capacity, and it removes the efficiency of pooling which is where the platform's margin comes from.

The honest architecture makes residency a tier with a price, rather than attempting to offer it uniformly at pooled economics.

The logging trap

Prompts and completions are the most sensitive data in the system, and they are exactly what teams log for debugging. A logging design that captures them by default, retains them broadly, and makes them searchable by engineers is a serious exposure — and it is the default in most implementations.

Redaction, short retention and access control on prompt logs should be designed in from the start, because retrofitting them means auditing every log statement in the system.