Enterprise Generative Search — Azure and Open Source  ·  View 38 of 41  ·  Assurance

Untrusted Content and Prompt Injection

Six concrete attacks, the control for each, and the residual risk stated rather than implied.

Editable source SVG draw.io All views
Where it enters What it attempts Control Residual risk Instruction inside a document Any indexed source white text, alt text, notes Be obeyed as a system rule ignore prior instructions Fenced evidence slot plus screening at index Novel phrasings evade detection the fence still holds Poisoned public page Crawled web source 40 allowed domains Rank above internal policy and be cited Authority weight, source policy external never outranks internal A legitimate external page can still mislead citation makes it visible Malicious tool argument Model output after reading a document Reach a system or exfiltrate arbitrary SQL or URL Broker allowlist, compiled SQL no free-form statements An allowlisted tool misused within its scope audited per call Exfiltration via a link Answer rendering markdown image or link Send context to an outside host as a URL parameter Links only to cited sources no model-authored URLs None known; the renderer is the control Scope escalation by asking The user turn itself Retrieve what the caller may not open act as another principal Filter is server-composed the query cannot set it Inference from the absence of results accepted and documented Jailbreak in the user turn The user turn itself Bypass safety or the answer format Prompt Shields plus schema validation output must parse Over-refusal annoys real users false refusal rate is a gate Untrusted Content — Six Attacks and What Stops Each The controls are structural, not lexical. A filter that looks for the phrase ignore previous instructions has already lost; the evidence slot is never read as instruction in the first place. v 1.0 · owner Data and AI Global Practice

The decision

  • The controls are structural, not lexical. A filter searching for the phrase ignore previous instructions has already lost — the evidence slot is never read as instruction in the first place.
  • The renderer, not the model, decides what becomes a link. Only cited sources become clickable, which closes exfiltration through a model-authored URL completely.
  • The tool broker checks provenance: a tool request whose origin is a document body is rejected before any policy question is even asked.

Residual risk, stated

  • A legitimate external page can still be wrong and still be cited. Authority weighting and visible citation make that a reviewable outcome rather than an invisible one.
  • Absence of results leaks a little information about what exists. Accepted and documented, because the alternative — uniform fake results — is worse for every honest user.
  • Over-refusal is the cost of these controls, and false refusal rate is gated in view 32 so it cannot creep.

Assurance

  • 220 adversarial cases run on every build, including documents with white text, hostile alt text, hostile slide notes and hostile spreadsheet comments.
  • A red-team exercise twice a year, with findings feeding the tool and source policies in view 40.
◀  Identity and Authorisation All views Sensitive Data and Tenant Isolation  ▶
Solution Architecture v1.0 · Data & AI Global Practice · 2026-09 · 41 views · Microsoft Azure and open source38-untrusted-content-defence