Prompt Compression

A RAG prompt that stuffs ten retrieved chunks plus a long system instruction in front of every question can run 8,000 to 12,000 tokens. You pay for those tokens on every call, they eat into a finite context window, and prefill latency scales with their count. Prompt compression asks a blunt question: how many of those tokens actually carry information the model needs, and can you drop the rest without the answer getting worse? The answer, surprisingly often, is that most of them are redundant. Natural language is low-entropy; the model can reconstruct meaning from a fraction of the surface tokens.

Two families attack this. Hard (extractive) compression deletes low-information tokens from the text itself, keeping a shorter string of real words. Soft compression trains a model to fold a long instruction into a handful of learned vectors that stand in for it. They differ in what the target model has to know, and in what they cost you.

The economics that drive it

Three separate pressures push toward fewer tokens, and they are worth separating because a compressor helps each differently.

• Cost. API pricing is per token. Halving a 10k-token prefix that fires on every request halves that line of the bill. For a shared system prompt repeated across thousands of users, this is the dominant lever.
• Prefill latency. Time to first token is roughly linear in prompt length (attention prefill is the work of reading the input). Shorter prompts start answering sooner.
• Context-window pressure. When retrieved context plus history threatens to overflow the window, compression buys headroom without dropping documents entirely.

The catch is that compression is not free: something has to decide what to cut, and that something runs before the main model. Whether the saving nets out positive depends on how cheap the compressor is relative to what it removes.

Hard compression: drop the low-information tokens

The insight behind LLMLingua (Jiang et al., 2023) is that a small language model's perplexity is a usable signal for token importance. Tokens the small model finds highly predictable given their context carry little surprise, so removing them costs the large model little. Tokens with high perplexity are load-bearing and should stay. You run a small model (say a 7B, or smaller) over the prompt, score tokens, and keep the surprising ones up to a token budget.

LLMLingua adds two refinements over naive per-token thresholding. A budget controller allocates the compression ratio across parts of the prompt (instruction, demonstrations, question) so the important parts are cut less aggressively. And compression is iterative and segment-wise rather than one independent pass, because dropping one token changes the perplexity of its neighbours; scoring tokens in isolation ignores those dependencies. The paper reports up to 20x compression with little performance loss on GSM8K, BBH, ShareGPT, and Arxiv-March23.

LongLLMLingua (Jiang et al., 2023) adapts this to long-context retrieval, where two extra problems appear: the important information is a small fraction of the prompt, and its position matters because models attend unevenly across a long context (the lost-in-the-middle effect). LongLLMLingua makes compression question-aware, scoring how relevant each document or segment is to the actual question rather than compressing every part uniformly, and it reorders content to move the most relevant material to positions the model attends to best. The reported results include up to a 21.4% improvement on NaturalQuestions at a 4x token reduction with GPT-3.5-Turbo, and a 94.0% cost reduction on the LooGLE benchmark; end-to-end latency drops 1.4x to 2.6x on roughly 10k-token prompts at 2x to 6x compression.

The property that makes hard compression practical: the output is still plain text, so it works against any black-box API model. You never touch the target model's weights.

Soft compression: learn tokens that stand in for the prompt

Gisting (Mu et al., 2023) takes the opposite route. Instead of shortening the text, it trains the model to compress a prompt into a small number of "gist" tokens whose key/value activations summarise the full instruction. The trick is a modified attention mask during instruction finetuning: tokens after the gist positions can attend to the gist tokens but not to the original prompt, which forces the gist activations to carry everything downstream generation needs. Once trained, you cache the gist KV for a given instruction and reuse it, skipping the original prompt entirely.

Gisting reports up to 26x compression of prompts, with up to 40% FLOPs reduction, roughly 4.2% wall-time speedup, and storage savings, at minimal quality loss, tested on LLaMA-7B and FLAN-T5-XXL. The wins and the constraints both follow from the method: because the compressed form is a set of activations rather than text, you must control (and finetune) the model to produce and consume them. Soft compression is for models you own; it does not apply to a closed API.

When it falls down

• Exact-extraction and verbatim tasks. Compression is lossy by construction. If the task is "quote the clause verbatim", "return the order ID", or "reproduce this code exactly", dropping tokens or replacing them with activations destroys the very strings you need. Never compress content whose exact characters matter: code, IDs, legal text, structured data.
• The compressor is not free. Hard compression runs a scoring model over the full prompt on every request. That is real compute and latency spent before the main model starts, and it partly offsets the saving. If the small model is not much cheaper than the tokens it removes, or if the prompt is short, you can end up slower overall.
• Soft compression needs the weights. Gisting requires finetuning and serving control, so it is a non-starter against a closed API. It also ties a compressed instruction to a specific model version; a finetune or model swap invalidates the cached gists.
• Prompt caching often beats compression outright. If the expensive part of your prompt is a fixed prefix reused across requests (a shared system prompt, the same few-shot examples), server-side prompt caching computes that prefix's KV once and reuses it directly, with no lossy edit to the text and no scoring model in the hot path. Before reaching for compression, check whether the prefix is stable and cacheable; caching gives you the latency and cost win with zero quality risk. Compression earns its place where the bulk of the prompt is per-request and non-repeating (freshly retrieved documents, unique user context), which is exactly where caching has nothing to hit.

Further reading

• LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models - the perplexity-based budget controller and iterative token-level compression, up to 20x.
• LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression - question-aware compression and document reordering for long-context RAG.
• Learning to Compress Prompts with Gist Tokens - the soft-compression approach; attention-mask trick, up to 26x with minimal quality loss.