System Prompt Design

Three properties separate the system prompt from any other text you send: it is present on every turn, it sits at the front of the token sequence, and it is trained to carry higher priority than what follows. Those properties make it the right home for some content and an expensive place to put the rest.

Priority is a trained behaviour, not an enforced one

Models are post-trained so that instructions from a system prompt outrank instructions appearing in user turns, and both outrank text arriving inside tool results or retrieved documents. OpenAI's instruction-hierarchy work made this explicit, generating training data where lower-privilege instructions conflict with higher-privilege ones and teaching the model to selectively ignore them, reporting substantial robustness gains including on attack types absent from training (Wallace et al., 2024, The Instruction Hierarchy, arXiv:2404.13208).

Treat that as a strong prior, never as a guarantee. The hierarchy is a learned tendency implemented in the same weights that process the attack, so a sufficiently persuasive injection in a retrieved document can still win. Anything whose violation is unacceptable belongs in code outside the model, not in a sentence at the top of the prompt.

What belongs in it

Content that is stable across every request in the deployment: the assistant's role and scope, the output contract, refusal and escalation policy, tool-use conventions, and the handful of domain facts that every answer depends on. Content that varies per request belongs in the user turn, both because it is more natural and because moving it out of the prefix breaks caching.

Structure matters as much as content. Delimited sections with explicit headers survive long-context degradation better than one prose block, because a later instruction referring to "the format defined above" needs a retrievable anchor. Positive instructions outperform prohibitions, since "respond in JSON with keys x and y" specifies a target while "do not write prose" leaves the target unspecified. And a system prompt that contradicts itself, which happens as soon as three teams edit it, resolves unpredictably; the model does not report the conflict.

The costs people underestimate

Cache stability. Prompt caching keys on an exact prefix. A system prompt containing the current timestamp, a session ID, or a randomly ordered tool list invalidates the cache on every request and converts your cheapest tokens into your most expensive ones. Put volatile values after the stable block.

Attention budget. The system prompt competes with the actual task for the same fixed attention mass, and instructions in the middle of a long prompt are used less reliably than those at the edges (Liu et al., 2023, Lost in the Middle, arXiv:2307.03172). A 4,000-token system prompt is not free context; it is 4,000 tokens of competition, paid on every turn.

Per-request cost multiplied by traffic. At a million requests a day, a 500-token system prompt is 500 million input tokens daily. Caching mitigates it and does not erase it.

When it breaks

• Instruction pile-up. Each production incident adds a rule, none are ever removed, and by rule forty the model follows the ones it can see and the ones that do not conflict. Periodically consolidate, and test after removal rather than assuming the rule was load-bearing.
• Silent conflict with fine-tuning. If the model was fine-tuned on data with a different system prompt, the deployed prompt fights the weights. Keep them aligned.
• Leakage. System prompts are extractable often enough that any secret placed in one should be considered public. Keep credentials, internal URLs, and unreleased product names out.
• Format instructions do not replace constrained decoding. "Always return valid JSON" reduces malformed output; it does not make it impossible. Use a grammar or schema when correctness matters.