Safety, Security & Governance

Proving What Is Real: Content Credentials, Watermarks, and the Limits of Detection

In October 2025 The Washington Post uploaded an AI-generated video carrying signed Content Credentials to eight social platforms, and not one kept the credential. Watermarks survive that trip but not a determined paraphrase, and passive detectors survive neither. Every layer of the provenance stack fails open in a different way, which is why Europe's new marking rules demand several at once.

In October 2025 The Washington Post uploaded an AI-generated video to eight major social apps. The file carried Content Credentials, the signed provenance manifest defined by the C2PA standard. The result: "No platform that The Post tested kept the Content Credentials data on the video or let users access it." Only YouTube showed any indication that the clip was not real, tucked into the expandable description as "Altered or synthetic content" (Washington Post, 2025, via Ritchin).

A signature proves what it signs, and nothing once it is gone. A watermark hidden in the pixels would have survived the upload, but a cheap regeneration pass scrubs it, and a paraphrase dilutes a text watermark below any sensible threshold. Passive detectors, with neither key nor signature, are easiest of all to fool. As of September 2026 this is also law: Article 50 of the EU AI Act has applied since 2 August 2026, requiring generative AI outputs to be "marked in a machine-readable format and detectable as artificially generated or manipulated."

Why this matters: Provenance is now a legal obligation and an engineering system with measurable failure rates. Anyone shipping generative AI into the EU has to pick marking techniques, set detection thresholds, and explain why a scheme that a $50 attack can defeat still counts as "robust and reliable as far as technically feasible."

TL;DR

  • Provenance systems produce positive evidence only. A valid manifest or a watermark score above \(z = 4\) supports a claim; a missing mark is equally consistent with human work, an unmarked model, or removal.
  • The green-list text watermark uses a one-proportion \(z\)-test with a false positive rate near \(3 \times 10^{-5}\) at \(z = 4\); at \(\gamma = 0.5\), \(\delta = 2\) on OPT-1.3B, 98.4% of 200-token generations were detected (Kirchenbauer et al., 2023).
  • Paraphrase dilutes rather than deletes: DIPPER cut watermark detection from 100% to 57.2% at 1% FPR (Krishna et al., 2023), yet even strong human paraphrases are caught after about 800 tokens at \(10^{-5}\) FPR (Kirchenbauer et al., 2024).
  • SynthID-Text ran on roughly 20 million Gemini responses with thumbs-up and thumbs-down rates differing by 0.01% and 0.02%, both statistically insignificant (Dathathri et al., 2024).
  • Regeneration removed 98% of RivaGAN image watermarks at PSNR above 30 (Zhao et al., 2023), and watermarks robust to every quality-preserving attacker are provably impossible (Zhang et al., 2023).
  • The EU Code of Practice (June 2026) commits signatories to signed metadata and an imperceptible watermark, exempts text under 200 tokens, and the omnibus Regulation (EU) 2026/1744 gives systems already on the market until 2 December 2026.

At a Glance

flowchart LR
    G[Generative model]:::purple --> W["Watermark in content"]:::purple
    G --> M["Signed C2PA manifest"]:::purple
    W --> D{Distribution}:::slate
    M --> D
    D -->|re-encode| S[Manifest stripped]:::rose
    D -->|paraphrase, regenerate| R[Watermark diluted]:::rose
    S --> V[Verifier]:::teal
    R --> V
    V --> P["Mark found: evidence"]:::emerald
    V --> N["No mark: proves nothing"]:::amber

    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff

Two marks leave the generator; distribution attacks each with a different tool, and a missing mark lands in the same bucket as human work.

From Detection Arms Race to Marking Mandate

The first answer to synthetic media was a trained classifier, which works until the next generator lacks the artefacts it learned. OpenAI withdrew its own AI text classifier on 20 July 2023 "due to its low rate of accuracy" (OpenAI, 2023), and Krishna and colleagues showed why the category was brittle: their DIPPER paraphraser dropped DetectGPT's accuracy on GPT-2 XL text from 70.3% to 4.6% at a fixed 1% false positive rate (Krishna et al., NeurIPS 2023). A detector with no secret is a detector the adversary can optimise against.

In February 2021, Adobe, Arm, BBC, Intel, Microsoft and Truepic merged the Content Authenticity Initiative and Project Origin into the Coalition for Content Provenance and Authenticity (Microsoft, 2021). Its specification reached 1.0 in December 2021. Version 2.0 (January 2024) was by its own account "a significant departure": signing restricted to X.509 certificates, plus a default C2PA Trust List for hardware and software signers. Version 2.4 (April 2026) added an AI Disclosure Assertion, a Repository Receipt Assertion, and embedding in HTML and structured text (C2PA Specification 2.4).

Watermarking came from ML labs in a rush during 2023. The green-list LLM watermark appeared in January (Kirchenbauer et al., ICML 2023); Stable Signature rooted a 48-bit key in a diffusion decoder in March (Fernandez et al., ICCV 2023); Tree-Ring hid a key in the Fourier spectrum of the initial noise in May (Wen et al., 2023). Removal attacks followed within weeks, and by November an impossibility result. Regulators did not wait for the argument to settle.

timeline
    title Provenance and watermarking, 2021 to 2026
    2021 : C2PA formed by Adobe, Arm, BBC, Intel, Microsoft, Truepic
         : C2PA specification 1.0 in December
    Early 2023 : Green-list LLM watermark
               : Stable Signature and Tree-Ring for diffusion images
    Late 2023 : Regeneration and purification attacks on images
              : Watermarks in the Sand impossibility result
    2024 : C2PA 2.0 adds the C2PA Trust List
         : EU AI Act in force, SynthID-Text in Nature
    2025 : Pixel 10 signs camera photos at Assurance Level 2
         : Washington Post finds 8 of 8 platforms drop credentials
    Mid 2026 : C2PA 2.4 adds AI Disclosure Assertion
             : Transparency Code of Practice and omnibus 2026/1744
    Late 2026 : Article 50 applies from 2 August
              : Grace period for existing systems ends 2 December

How the Three Layers Actually Work

Metadata lives beside the content and needs no secret to verify; watermarks live inside it and need a key.

The signed manifest

A C2PA manifest bundles assertions (an AI disclosure, a list of edit actions, references to ingredient assets) under a claim that lists each assertion by hash, signed with an X.509 certificate. A hard binding, a cryptographic hash over the asset's bytes, ties the manifest to one exact file: change a pixel and it fails. Each compliant edit appends a manifest naming the previous asset as an ingredient, forming a signed chain.

Verification checks the binding, the signature, and the certificate against a trust list, which is the real security boundary. Cryptography proves that a listed key signed the claim, not that the claim is true; a camera pointed at a monitor produces a validly signed "photograph." Google's Pixel Camera on Pixel 10 achieved Assurance Level 2, "the highest security rating currently defined by the C2PA Conformance Program," by generating keys in hardware-protected storage and proving it with Android Key Attestation before certificates are issued (Google Security Blog, 2025).

The manifest's fatal property is separability. A transcoding platform either re-signs (almost none do) or drops it. C2PA's answer is the soft binding, defined as a content identifier "either (a) not statistically unique, such as a fingerprint, or (b) embedded as an invisible watermark." A verifier holding a stripped file extracts the watermark, looks the identifier up in a manifest repository, and recovers the credential. Provenance metadata falls back on watermarking to survive the real internet.

The green-list text watermark

Text has no imperceptible bits to flip, so the Kirchenbauer scheme hides its signal in the choice among acceptable tokens. At step \(t\), hash the previous token with a secret key, use the result to split vocabulary \(V\) into a green list \(G\) of size \(\gamma|V|\) and a red list \(R\), and add \(\delta\) to every green logit:

\[ \hat p_k^{(t)} = \frac{e^{\ell_k + \delta \cdot \mathbb{1}[k \in G]}}{\sum_{i \in G} e^{\ell_i + \delta} + \sum_{i \in R} e^{\ell_i}} \]

Here \(\ell_k\) is the model's logit for token \(k\). When many tokens are plausible, the bias tips the choice toward green at little quality cost. When one token dominates, as in a memorised quotation or the only valid next character of code, the bias changes nothing. The scheme spends entropy; where there is none, it cannot write.

How much green does that buy? For a perfectly flat distribution, the chance of emitting green is

\[ P(\text{green}) = \frac{\gamma e^{\delta}}{1 + \gamma(e^{\delta} - 1)}, \]

which for \(\gamma = 0.25\), \(\delta = 2\) is \(0.25 \times 7.39 / 2.60 \approx 0.71\) against a chance rate of 0.25. Real distributions are peaked, so the paper scales this by the average spike entropy \(S^\star\) of the model's distributions: \(\mathbb{E}|s|_G \ge \frac{\gamma \alpha T}{1 + (\alpha - 1)\gamma} S^\star\) with \(\alpha = e^\delta\). For OPT-1.3B at \(\gamma = 0.5\), \(\delta = 2\), \(S^\star = 0.807\) and \(T = 200\), the bound is 142.2 green tokens; the empirical mean was 159.5.

Detection needs only the key and the text. Recompute every green list, count green tokens \(|s|_G\) among \(T\) scored tokens, and compare with the binomial null that unwatermarked text hits green at rate \(\gamma\):

\[ z = \frac{|s|_G - \gamma T}{\sqrt{T \gamma (1 - \gamma)}} \]

The numerator is the excess over chance; the denominator is a binomial standard deviation. Rejecting at \(z > 4\) fixes the one-sided false positive rate near \(3 \times 10^{-5}\) analytically, not on a validation set, provided tokens are independent. Repeated (context, token) pairs break that assumption, which the authors flagged and their follow-up measured.

The knob most people miss is the context width \(h\), the number of preceding tokens hashed into the seed. With \(h = 1\), an observer with enough watermarked text can tabulate which tokens follow which and learn the green lists. Wider contexts resist that, but every edited token now corrupts the green lists of the next \(h\) positions, so the mark becomes more fragile to editing. Kirchenbauer's follow-up frames exactly this tension (Kirchenbauer et al., ICLR 2024); the worked example below prices it.

SynthID-Text and tournament sampling

Google DeepMind's production scheme changes the sampler rather than the logits. SynthID-Text seeds \(m\) pseudorandom scoring functions \(g_1, \dots, g_m\) from the last \(H = 4\) tokens plus a key, samples \(2^m\) candidates from the unmodified model distribution, and runs a knockout tournament: pairs compete on \(g_1\), winners on \(g_2\), until one token remains. Experiments generally use \(m = 30\) layers, with a vectorized implementation for efficiency (Dathathri et al., Nature 634, 2024). Since every candidate is a genuine model sample, the scheme can be configured as non-distortionary, and detection scores mean \(g\) values without running the LLM.

In production, across approximately 20 million Gemini responses split between watermarked and unwatermarked models, feedback rates differed by 0.01% and 0.02%, inside the 95% confidence intervals. The authors are equally plain that generative watermarks "are weakened by edits to the text, such as through LLM paraphrasing," and cannot touch open-weight models deployed by others. Christ, Gunn and Zamir go further with a cryptographically undetectable scheme, indistinguishable from unmarked output without the key (Christ et al., COLT 2024). That settles quality in principle, and says nothing about removal.

Image watermarks: three positions

Post-hoc schemes such as DwtDct and RivaGAN add a signal to finished pixels. Decoder-rooted schemes fine-tune the generator so everything it emits carries the mark: Stable Signature detects 90% of images cropped to 10% of their content while flagging one false positive per \(10^6\) images. Semantic schemes mark the generation process: Tree-Ring writes a ring-shaped key into the Fourier transform of the initial noise and detects by DDIM inversion back to noise. Position determines attack surface.

[IMAGE: Three columns labelled post-hoc, decoder-rooted and semantic. Each shows a diffusion pipeline (noise, denoiser, decoder, image) with the key's location highlighted: after the image, in the decoder weights, in the Fourier rings of the initial noise. Caption: "Where the key lives decides how expensive it is to remove."]

Seeing It in Motion

A C2PA manifest is only as durable as its strongest surviving binding.

flowchart TB
    subgraph Asset["Asset as distributed"]
        B[Content bytes]:::blue
        WM["Invisible watermark ID"]:::blue
    end
    subgraph Store["C2PA manifest"]
        A1["Actions and AI disclosure"]:::purple
        A2["Hard binding: hash of bytes"]:::purple
        A3["Soft binding: watermark ID"]:::purple
        C[Claim over assertion hashes]:::purple
        SG["X.509 signature"]:::purple
    end
    TL[C2PA Trust List]:::slate
    REPO[Manifest repository]:::slate
    A2 -->|breaks on any edit| B
    A3 -->|survives re-encoding| WM
    A1 --> C
    A2 --> C
    A3 --> C
    C --> SG
    SG -->|validated against| TL
    WM -->|lookup| REPO

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0

One image through a hostile pipeline, with soft-binding recovery:

sequenceDiagram
    participant Gen as Generator
    participant Repo as Manifest repository
    participant Plat as Social platform
    participant Ver as Verifier
    Gen->>Gen: Embed watermark ID in pixels
    Gen->>Repo: Register signed manifest under ID
    Gen->>Plat: Upload JPEG with embedded manifest
    Plat->>Plat: Transcode and strip metadata
    Plat->>Ver: Serve re-encoded image
    Ver->>Ver: No manifest in file, extract watermark
    Ver->>Repo: Look up manifest by ID
    Repo-->>Ver: Signed manifest for trust list check
    Note over Ver: Failed extraction means unknown, not human

[IMAGE: Two panels. Left: a JPEG drawn as stacked boxes (image data, EXIF, JUMBF manifest box with assertions and signature). Right: the same file after upload, JUMBF box struck through, with a dotted "watermark ID" thread running from the pixels to a repository holding the manifest. Caption: "Hard bindings die with the file; soft bindings let the manifest be found again."]

By the Numbers

Settings differ between papers; compare within rows.

Scheme or attack Setting Before After Source
Green-list watermark OPT-1.3B, \(\gamma=0.5\), \(\delta=2\), \(T=200\), \(z=4\) 98.4% detected no attack Kirchenbauer 2023
Green-list vs. DIPPER GPT-2 XL, 300 tokens, 1% FPR 100.0% 57.2% Krishna 2023
DetectGPT vs. DIPPER GPT-2 XL, 1% FPR 70.3% 4.6% Krishna 2023
KGW vs. random-walk attack Llama-2-7B, 200 tokens mean \(z = 6.24\) mean \(z = 1.63\) Zhang 2023
Human paraphrase FPR \(10^{-5}\) detected detected after ~800 tokens Kirchenbauer 2024
Watermark stealing KGW-family, spoof and scrub "considered safe" over 80% success, under $50 Jovanović 2024
Stable Signature, 10% crop 48-bit key, FPR \(10^{-6}\) n/a 90%+ detected Fernandez 2023
Tree-Ring Rings AUC / TPR@1%FPR, 6 transforms 1.000 / 1.000 0.975 / 0.694 Wen 2023
RivaGAN vs. regeneration PSNR above 30 detected 98% removed Zhao 2023
SynthID-Text in production ~20M Gemini responses baseline feedback within 0.02% (n.s.) Dathathri 2024
Credentials after upload 8 major social apps signed 0 of 8 kept Washington Post 2025

Sources: the primary papers listed below; Krishna figures are Table 1's most diverse "60L, 60O" paraphrase; Jovanović's cost uses January 2024 API prices; Wen's six transforms are rotation, JPEG, crop and scale, blur, noise and colour jitter, with no regeneration; the platform row is the Washington Post test via Ritchin, 2025.

Read top to bottom, every scheme falls. Read along the Kirchenbauer rows, attacks dilute a signal that keeps accumulating with length: failure is a function of how much the adversary rewrote and how much text you see.

[IMAGE: Line chart, x-axis "tokens observed (0 to 1,000)", y-axis "mean z-score", dashed threshold at z = 4. Three curves: unattacked rising steeply, machine-paraphrased at about half the slope, human-paraphrased slowest and crossing near 800 tokens. Caption: "Paraphrase changes the slope, not the sign."]

A Concrete Example

A newsroom receives a 200-token op-ed and suspects it came from a model whose provider publishes a green-list detector with \(\gamma = 0.25\), \(h = 1\) and a threshold of \(z > 4\). The test on paper:

Step 1: the null. If the text was not generated with this key, each of \(T = 200\) tokens is green with probability 0.25:

\[ \mu_0 = \gamma T = 50, \qquad \sigma_0 = \sqrt{T\gamma(1-\gamma)} = \sqrt{37.5} \approx 6.12. \]

Step 2: the threshold. \(z > 4\) means \(|s|_G > 50 + 4 \times 6.12 = 74.5\), so 75 or more green tokens, at a one-sided tail probability of \(3.2 \times 10^{-5}\).

Step 3: count. The detector finds 92 green tokens, a green rate of 0.46 (illustrative, and under the 0.71 flat-distribution ceiling):

\[ z = \frac{92 - 50}{6.12} \approx 6.86, \qquad p \approx 3.5 \times 10^{-12}. \]

Step 4: a light rewrite. Suppose the submitter first swapped 20% of tokens (40) for synonyms. With \(h = 1\), a swap disturbs its own position and the next one's green list, so the disturbed fraction is \(1 - 0.8^{h+1} = 0.36\): about 72 positions revert to rate 0.25 while 128 keep 0.46.

\[ |s|_G \approx 128(0.46) + 72(0.25) = 58.9 + 18.0 = 76.9, \qquad z \approx 4.39. \]

Still flagged, barely. At 30% substitution the count falls to about 70.6 and \(z \approx 3.36\): the essay passes as clean.

Step 5: the same rewrite, wider context. Let the provider use \(h = 4\), as SynthID-Text's seed does, to resist stealing. The 20% rewrite now disturbs \(1 - 0.8^5 = 0.67\) of positions, about 134:

\[ |s|_G \approx 66(0.46) + 134(0.25) = 30.4 + 33.6 = 64.0, \qquad z \approx 2.25. \]

The edit that left the \(h = 1\) mark detectable erases the \(h = 4\) one. Stealing resistance was bought with edit fragility.

Step 6: how much text is enough? Solving \(z = 4\) for \(T\) at green rate \(q\) gives \(T = \left(4\sqrt{\gamma(1-\gamma)}/(q - \gamma)\right)^2\). Unattacked (\(q = 0.46\)): about 68 tokens. After the \(h = 1\) rewrite (\(q \approx 0.385\)): about 166. Dilution multiplies the length needed.

Step 7: the false positive budget. Run on a million human-written submissions a year at \(z > 4\), the test produces about 32 false flags. The Code's 200-token floor looks sensible after this arithmetic.

[IMAGE: Three 20 by 10 grids of squares, green or grey: the original essay (92 green); after a 20% rewrite with h = 1 (swapped squares outlined red, right neighbours hatched); and with h = 4 (hatching extending four squares right). A z gauge beside each reads 6.86, 4.39 and 2.25. Caption: "Wider hashing windows spread every edit across more positions."]

Where It Breaks

The impossibility result, and what it rules out

Zhang et al. prove that strong watermarking is impossible against an attacker with a quality oracle (can judge whether a candidate is still a good response) and a perturbation oracle (makes small quality-preserving edits that mix well over good outputs) (Zhang et al., ICML 2024). A low-false-positive watermark marks only a small fraction of good outputs, so a long enough random walk among good outputs leaves the marked set. Against KGW on Llama-2-7B, mean \(z\) fell from 6.24 to 1.63, with GPT-4 judging the quality change small.

The disagreement with the watermarking camp is narrower than it looks: Zhang et al. rule out marks that no attacker can remove, Kirchenbauer et al. argue for marks that are expensive to remove at a given length, and both hold. Regulation has to be built on that cost, not on a guarantee.

Spoofing damages the provider

Spoofing makes harmful content test positive for a provider's key. Jovanović, Staab and Vechev approximated green-list rules from API queries and then spoofed and scrubbed schemes "previously considered safe" for under $50, at average success above 80% (Jovanović et al., ICML 2024). For images, Saberi et al. built from black-box access a noise pattern that makes real images read as watermarked (Saberi et al., 2023). A public detector can be both evaded and framed.

Regeneration defeats pixel-space marks

Zhao et al. noise an image's latent and reconstruct it with a diffusion model, provably removing marks whose perturbation is small in \(\ell_2\) (Zhao et al., NeurIPS 2024). Saberi et al. formalise the consequence: for low-perturbation marks, diffusion purification forces a trade-off between evasion and spoofing error, so no threshold keeps both small. Tree-Ring resists purification better, but the WAVES benchmark found "a single regeneration such as Regen-Diff and Regen-VAE can significantly harm" its TPR at 0.1% FPR with reasonable image quality (An et al., ICML 2024). Tree-Ring's own 0.975 AUC was measured against six classical transforms, a different adversary.

Low entropy, and stripping as the default

Code, JSON and short replies carry little entropy, and memorised passages none; Kirchenbauer's undetected cases "typically involve data memorization." Hence the Code's 200-token line. Metadata fails just as mundanely: nobody attacked the Washington Post video; ordinary transcoding removed its credentials.

The analogue hole

If the signer lies, the key leaks, or a trusted camera photographs a screen, the signature is valid and the claim false. Hardware keys make theft harder; nothing makes a photographed screen detectable from its signature.

[IMAGE: A 2 by 2 matrix titled "What a verifier can conclude." Rows: manifest valid / absent. Columns: watermark detected / not. Cells: "Strong claim, check the signer"; "Claim stands, mark may be scrubbed"; "Likely keyed model, history unknown"; "Unknown: human, other model, or stripped." The last cell is grey and drawn largest. Caption: "The most common outcome is the uninformative one."]

Alternative Designs

Design How it works Key advantage Key limitation Best when
Signed metadata (C2PA) X.509-signed manifest hashed to the bytes Exact, public, any length Stripped by re-encoding; attests to signer, not truth Controlled capture and editing pipelines
Generative text watermark Keyed bias or tournament in the sampler Survives copy-paste; calibrated FPR Needs length and entropy; diluted by paraphrase Long-form output from a provider that controls sampling
Decoder-rooted image mark Generator fine-tuned to emit a key Cannot be switched off at inference Regeneration removes it; leaks with open weights Closed image APIs
Semantic image mark (Tree-Ring) Key in the initial noise spectrum Robust to classical transforms Needs diffusion inversion; regeneration harms it Diffusion APIs with detection compute
Fingerprinting or logging Store hashes or outputs, match on query Nothing embedded; no quality cost Privacy cost; the Code says insufficient alone Audio, video, enterprise audit
Retrieval over outputs Semantic search of the provider's output log Caught 97.3% of PG19 paraphrases at 1% FPR Must retain every output Providers able to keep large logs
Passive detection Classifier on statistical artefacts Works on any source Unkeyed, easily optimised against Triage only, never sole evidence

Retrieval is underrated: paraphrase preserves meaning, which is what semantic search matches (Krishna et al., 2023). Each row fails against a different adversary, which is the case for stacking them.

How It Is Used in Practice

What Article 50 now requires

Article 50(2) of Regulation (EU) 2024/1689 obliges providers of systems "generating synthetic audio, image, video or text content" to mark outputs with solutions that are "effective, interoperable, robust and reliable as far as this is technically feasible" (AI Act, Art. 50). Breaches fall under Article 99(4): fines up to EUR 15 million or 3% of worldwide annual turnover, whichever is higher.

Regulation (EU) 2026/1744, signed 8 July 2026 and published in the Official Journal on 24 July, inserted Article 111(4): providers of systems "placed on the market before 2 August 2026 shall take the necessary steps in order to comply with Article 50(2) by 2 December 2026" (Regulation (EU) 2026/1744). The 2 August 2026 application date itself did not move; new systems complied from day one.

The Code of Practice on Transparency of AI-generated Content, published in June 2026 and confirmed adequate by the Commission and AI Board in July, turns the obligation into architecture (European Commission, 2026). Signatories commit to signed metadata and an imperceptible watermark, "with the exception of very short text," currently text under 200 tokens. Fingerprinting or logging is optional and "not considered sufficient" alone. Free-form text detection may be restricted to verified experts given its "lower level of reliability," and signatories under 1,000,000 monthly users may charge a proportionate detection fee. The Code encodes a layered, positive-evidence design, and admits text is the weak layer.

What providers actually build

A provider serving the EU typically runs four services. A signing service holds C2PA keys in an HSM or secure element and stamps an AI disclosure and soft-binding ID into each media manifest. A sampler hook applies the text watermark and must coexist with batching and speculative decoding; SynthID's paper offers a variant that preserves detectability at some latency cost. A detection endpoint reports which layer produced a verdict and is rate-limited, because an unmetered detector is a free oracle for watermark stealing. A repository maps watermark IDs to manifests so stripped content can be re-credentialed. Key rotation that keeps old content verifiable, and thresholds set against real query volume, are the ongoing chores.

[IMAGE: A provider's compliance stack: an inference cluster with a "watermark sampler" box, a signing service with an HSM icon, a manifest repository, and a public detection API behind a rate limiter with an "expert access" gate for text. Caption: "Article 50 compliance is four services, not one feature."]

Insights Worth Remembering

  1. Provenance is a one-sided test. Every mechanism supports "this was made by X" and none supports "this was not." Reading absence of a mark as human authorship repeats the error of unkeyed detectors, which misclassified non-native English writing as AI-generated.

  2. Metadata and watermarks fail against different adversaries. Manifests die in transcoding and survive paraphrase; watermarks survive transcoding and die in paraphrase or regeneration. That is why C2PA's soft binding and the EU Code pair them.

  3. Robustness is a price, not a property. A patient attacker with a quality oracle always wins, so a useful watermark specification states attack cost at a given length.

  4. Length is the master variable for text. Power grows with \(\sqrt{T}\); a 20% rewrite more than doubled the tokens needed in the worked example.

  5. Wider hashing contexts trade stealing resistance for edit fragility. Moving from \(h = 1\) to \(h = 4\) turned \(z = 4.39\) into \(z = 2.25\) for the same edit.

  6. A public detector is also a spoofing oracle, and the trust list is the real boundary. Rate limits, expert-only access and key issuance policy are security controls, not bureaucracy.

Open Questions

Can a text watermark be both hard to steal and robust to editing? Measured: \(h = 1\) schemes are learnable from queries, and wider contexts are edit-fragile. Whether semantic watermarks keyed to meaning can keep a calibrated false positive rate is unresolved.

Will platforms preserve or re-sign manifests? Measured: zero of eight kept credentials in October 2025. The AI Act's marking duty falls on providers of generative systems, not on the platforms that re-encode their output.

What will regulators accept as "technically feasible"? Whether authorities will accept a text watermark that a $50 attack can scrub, provided the provider followed the Code, is untested as of September 2026.

Can open-weight models be marked at all? A user who controls the sampler can skip a text watermark, and decoder-rooted marks ship inside weights that can be fine-tuned. Whether any scheme survives that user is open, and it is a structural gap in a cooperation-based regime.

Do errors fall evenly across writers? Low-entropy or non-native prose gives a watermark less to work with, and passive detectors have been measured misclassifying non-native English writing (Liang et al., 2023).

Sources and Further Reading

  1. Kirchenbauer, J., Geiping, J., Wen, Y., et al. (2023). "A Watermark for Large Language Models." ICML 2023. arXiv:2301.10226

  2. Kirchenbauer, J., Geiping, J., Wen, Y., et al. (2024). "On the Reliability of Watermarks for Large Language Models." ICLR 2024. arXiv:2306.04634

  3. Dathathri, S., See, A., Ghaisas, S., et al. (2024). "Scalable watermarking for identifying large language model outputs." Nature, 634, 818–823. doi:10.1038/s41586-024-08025-4

  4. Christ, M., Gunn, S., & Zamir, O. (2024). "Undetectable Watermarks for Language Models." COLT 2024, PMLR, 1125–1139. arXiv:2306.09194

  5. Krishna, K., Song, Y., Karpinska, M., et al. (2023). "Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense." NeurIPS 2023. arXiv:2303.13408

  6. Zhang, H., Edelman, B. L., Francati, D., et al. (2024). "Watermarks in the Sand: Impossibility of Strong Watermarking for Generative Models." ICML 2024. arXiv:2311.04378

  7. Jovanović, N., Staab, R., & Vechev, M. (2024). "Watermark Stealing in Large Language Models." ICML 2024. arXiv:2402.19361

  8. Fernandez, P., Couairon, G., Jégou, H., et al. (2023). "The Stable Signature: Rooting Watermarks in Latent Diffusion Models." ICCV 2023. arXiv:2303.15435

  9. Wen, Y., Kirchenbauer, J., Geiping, J., & Goldstein, T. (2023). "Tree-Ring Watermarks: Fingerprints for Diffusion Images that are Invisible and Robust." arXiv:2305.20030

  10. Zhao, X., Zhang, K., Su, Z., et al. (2024). "Invisible Image Watermarks Are Provably Removable Using Generative AI." NeurIPS 2024. arXiv:2306.01953

  11. Saberi, M., Sadasivan, V. S., Rezaei, K., et al. (2023). "Robustness of AI-Image Detectors: Fundamental Limits and Practical Attacks." arXiv:2310.00076

  12. An, B., Ding, M., Rabbani, T., et al. (2024). "WAVES: Benchmarking the Robustness of Image Watermarks." ICML 2024. arXiv:2401.08573

  13. Liang, W., Yuksekgonul, M., Mao, Y., et al. (2023). "GPT detectors are biased against non-native English writers." Patterns. arXiv:2304.02819

  14. Coalition for Content Provenance and Authenticity (2026). "Content Credentials: C2PA Technical Specification," version 2.4, with version history from 1.0 (December 2021). spec.c2pa.org. Founding announcement: Microsoft, 22 February 2021

  15. Regulation (EU) 2024/1689 (Artificial Intelligence Act), Articles 50 and 99 (text), as amended by Regulation (EU) 2026/1744, new Article 111(4) (EUR-Lex).

  16. European Commission (2026). Code of Practice on Transparency of AI-generated Content (June 2026) and signing guidance. digital-strategy.ec.europa.eu

  17. The Washington Post (22 October 2025). "Tests show top social platforms don't disclose markers on AI videos." washingtonpost.com; quoted by F. Ritchin

  18. Google Online Security Blog (10 September 2025). "How Pixel and Android are bringing a new level of trust to your images with C2PA Content Credentials." security.googleblog.com

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.