Secrets and Credentials in LLM Context

In a systematic study of prompt extraction, simple text-based attacks approximately recovered 86% of the hidden prompts tested against GPT-4 and 95.2% against Llama-2-chat-70B, and a defence that filtered outputs for overlapping 5-grams was largely bypassed on the largest model by asking for the prompt in an encoded or interleaved form (Zhang, Carlini & Ippolito, 2023, Effective Prompt Extraction from Language Models, arXiv:2307.06865). A system prompt is a document the user will probably read.

This is the premise of an entry added in the 2025 revision of the OWASP Top 10 for LLM Applications, LLM07:2025 System Prompt Leakage, which states that the system prompt should not be considered a secret nor used as a security control (OWASP GenAI Security Project, LLM07:2025). As of September 2026 the 2025 list remains the current edition.

The context window has no access control

Every token in context is available to the model when it generates, and the model's output goes to whoever is on the other end. There is no mechanism by which one span is readable for reasoning but unprintable. An API key in the system prompt, a database connection string in a tool description, or a colleague's salary in a retrieved spreadsheet is disclosed to the user who can ask for it, and through indirect injection to any author of content the model reads, who can instruct it to encode the secret into a link or image URL.

Defences that try to make the model keep a secret are probabilistic, and an attacker gets many attempts. If one attempt succeeds with probability \(p\), then \(m\) independent attempts succeed with probability

At \(p = 0.05\) and \(m = 100\), that is \(1 - 0.95^{100} \approx 0.994\). A control that fails 5% of the time is not a control against a patient adversary. OWASP's related entries point the same way: LLM02:2025 Sensitive Information Disclosure covers the data, and LLM06:2025 Excessive Agency covers what a model holding credentials can do with them.

Credential brokering

The robust design keeps secrets out of context altogether. The model emits an intent, such as a tool call naming an operation and its arguments. A broker outside the model authenticates the end user, resolves a short-lived credential scoped to that user and that operation, performs the call, and returns only the result. The model never sees the token, so no prompt can extract it.

Two properties make this hold. Credentials are bound to the user on whose behalf the call is made, not to a service account with the union of everyone's permissions, so a manipulated model can reach at most what its user already could; the sibling concept on agent permissions and blast radius develops the scoping. And tokens are audience-restricted. The Model Context Protocol's security guidance forbids "token passthrough": an MCP server MUST NOT accept tokens that were not explicitly issued for that server, and scopes should start minimal and be elevated per operation (MCP, Security Best Practices).

Secrets that arrive uninvited

Keeping credentials out of the system prompt handles the secrets you put there. Most leaks come through doors nobody designed.

Users paste them. A developer debugging a failed request pastes the curl command, bearer token included, and the conversation is now stored, logged, possibly used for evaluation and possibly retrieved into a later session's memory.

Indexed tools contain them. GitGuardian's 2025 report counted 23.8 million secrets leaked on public GitHub in 2024, found that 70% of secrets leaked in 2022 were still valid, and detected secrets in 2.4% of corporate Slack channels and 6.1% of Jira tickets it scanned (GitGuardian, 2025, The State of Secrets Sprawl 2025). An assistant that indexes Jira indexes live credentials, and retrieval will surface them for a sufficiently relevant query.

Observability stores them. Tracing that records full prompts and completions copies every secret that passed through context into a system with broader read access than the original source. Secret scanning therefore belongs at three points: ingestion into any index, the log and trace writer, and model output before it is rendered. The detection layers are shared with PII detection and redaction.

When it breaks

Pattern scanners miss generic secrets. Provider keys with fixed prefixes are easy to match; GitGuardian classified 58% of leaked secrets as generic, such as passwords and custom tokens with no recognisable format. Entropy and context heuristics catch more at the price of false positives.

Redaction does not un-leak. Once a secret has reached a provider log, a trace store or a vector index, removing it from one copy leaves the others. Rotation is the only remedy that actually closes the exposure.

Tool output is context too. A tool that returns environment variables, a stack trace with a connection string, or a config file hands the model a secret mid-conversation.

Practitioners disagree about protecting prompts at all. Some teams treat system prompts as intellectual property and add anti-extraction instructions and output filters, which do deter casual copying. The extraction results and OWASP's guidance argue those measures are speed bumps. Both can be true, provided no security property depends on the prompt staying hidden.