A user asks the internal assistant a question and receives content from a document they cannot access. How did this happen and how is it prevented?

What the interviewer is testing

Whether you know that retrieval must be permission-aware at query time, and where the leak paths are.

How it happened

Retrieval was not filtered by the requesting user's permissions. The index contains everything; the query matched a passage in a restricted document; the model summarised it.

The variants:

Index-time permissions only — the index was built with the permissions as they were, and access has since changed. Permissions are dynamic; an index is a snapshot.

Filtering after retrieval rather than during it, where the filter was applied to the final answer but the model had already seen the restricted content and paraphrased it. Content in the context window has already leaked, whatever happens afterwards.

Permissions attached to documents but not to chunks, so derived chunks lost their access metadata.

A summary or derived index built across all documents, which itself has no meaningful permission model — this is the subtle one, because the aggregate artifact leaks without any single document being retrieved.

The prevention

Filter during retrieval, using the requesting user's current permissions as a query constraint against access metadata on every chunk. The restricted content must never enter the context.

Evaluate permissions at query time, not at index time — fetch the user's current entitlements.

Propagate access metadata to every chunk and derived artifact, so it survives the pipeline.

Deny by default: a chunk with no access metadata is not retrievable, rather than being treated as public.

Test it continuously: an automated test that queries as a restricted user for content they should not see, across every access path.

What a strong answer adds

The reason this is the decisive argument against fine-tuning on internal documents: a model that has absorbed the corpus into its weights cannot un-know a document for a particular user, and there is no fix short of a model per permission set. Retrieval-based architectures are the only ones that can enforce enterprise permissions at all.

Common weak answers

Post-filtering the answer. Removing the document from the index, which fixes one instance.