Subscribe
AI
7 min read

What Is an AI Text Watermark? How Researchers Make LLM Output Detectable on Purpose

For most of 2023, the honest advice about spotting AI-written text was: do not trust any tool that claims it can. OpenAI launched an AI text classifier on January 31, 2023 and quietly withdrew it on July 20, 2023, citing a low rate of accuracy. By the company's own numbers the classifier correctly flagged 26% of AI-written text as likely AI-written while incorrectly labeling 9% of human writing as machine-made (Ars Technica, OpenAI). The failure mode got uglier under scrutiny. A study by Weixin

What Is an AI Text Watermark? How Researchers Make LLM Output Detectable on Purpose

For most of 2023, the honest advice about spotting AI-written text was: do not trust any tool that claims it can. OpenAI launched an AI text classifier on January 31, 2023 and quietly withdrew it on July 20, 2023, citing a low rate of accuracy. By the company's own numbers the classifier correctly flagged 26% of AI-written text as likely AI-written while incorrectly labeling 9% of human writing as machine-made (Ars Technica, OpenAI).

The failure mode got uglier under scrutiny. A study by Weixin Liang and colleagues at Stanford, published in the journal Patterns, ran seven popular detectors on 91 TOEFL essays written by non-native English speakers. The detectors flagged 61.22% of them as AI-generated, while grading essays by US eighth-graders almost perfectly. Eighteen of the 91 human essays were unanimously declared machine-written by all seven tools (Stanford HAI, Patterns00130-7)).

The root problem is that ordinary human writing has no fixed statistical signature to detect. Predictable, plain prose, which is exactly what second-language writing often looks like by the metrics detectors use, reads as AI to them. Since passive detection has nothing reliable to find, a different school of research asked the opposite question: what if we make AI text detectable on purpose? That idea is called watermarking, and by late 2026 it is the only approach with a real product deployment and, in the European Union, a regulatory mandate behind it.

The core trick: bias the coin flip

A language model generates text by computing, at each step, a probability distribution over the vocabulary and sampling from it. In 2023 John Kirchenbauer and colleagues at the University of Maryland proposed nudging that sampling step so the resulting text carries a statistical signal that is invisible to readers but testable afterward (ICML 2023 paper, open-source implementation).

The scheme works like this. Before each token is sampled, a secret key plus the preceding tokens are hashed into a random seed. That seed splits the vocabulary into a "green list" and a "red list", with the green share set by a parameter gamma, usually 0.5. The model's logits for green tokens get a small additive bump, delta, typically between 0.5 and 2.0, before softmax. The bump is gentle: the model still mostly picks words it considered likely, but whenever it faces a close call between two plausible continuations, the green one wins more often than chance. The paper also introduced a "soft red list" refinement that skips the bias when one next token dominates the distribution, protecting the parts of text where there is no real choice to make.

Detection is then a statistics problem, not a vibe check. Anyone with the secret key can replay the hashing over a suspect text, sort each token into green or red, and count. Genuine human text, or text from an unwatermarked model, lands near 50% green. Watermarked text comes out "too green," and the deviation gives a z-score and a p-value, interpretable like any hypothesis test. Kirchenbauer's team reported that the test separates watermarked from human text reliably on passages of around 200 tokens, while the implementation's own notes flag the practical limits: short texts lack statistical power, and repeated phrases distort the test because the pseudo-random lists stop being independent (Arize interview, implementation notes).

One property of the scheme is worth keeping in mind, because everything that follows is a consequence of it. The watermark rides on uncertainty. Where the next word is nearly forced, like the rest of a famous quotation, there is nothing to bias, so the signal only accumulates in the low-confidence, many-choices stretches of text. That is also why watermark strength is always traded against text quality: bigger bumps mean easier detection and more distorted writing.

A grid of word tiles split into green and red lists, with a hand tipping the choice toward the green side

Watermarks that do not bend the distribution

Researchers then asked whether the quality trade-off could be avoided by design. Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto and Percy Liang developed so-called distortion-free schemes that leave the model's output distribution mathematically unchanged, embedding the watermark into the randomness of the sampling process itself, via inverse-transform and exponential-minimum sampling, rather than into the logits (TMLR 2024, arXiv). Their numbers show both the promise and the limit: on OPT-1.3B and LLaMA-7B the watermark stayed detectable at p at most 0.01 from 35 tokens even after 40% to 50% of tokens were randomly edited, but on the instruction-tuned Alpaca-7B, whose low-entropy responses offer fewer free choices, only about 25% of typical 100-token answers were detectable at that confidence.

What actually shipped: SynthID-Text

The first large-scale public deployment of an LLM text watermark is Google DeepMind's SynthID-Text, described in Nature in October 2024 (Nature, IEEE Spectrum). Instead of a green-red split, it assigns pseudorandom scores to candidate tokens via keyed functions and picks through a tournament-style sampling step. DeepMind reported an A/B test across 20 million Gemini prompts where user satisfaction with watermarked and unwatermarked answers was statistically indistinguishable, and open-sourced a reference implementation developers can run as a logits processor on their own models (DeepMind SynthID page, GitHub).

The deployment boundaries matter. Google has said the watermark applies to the Gemini app and web experience, not to API outputs, so text generated through the API currently carries no mark (developer forum answer). And the public SynthID Detector checks images, video and audio; text verification remains restricted to Google and licensees holding the key. Research from ETH Zurich's Secure, Systems and Platforms group adds a wrinkle in the other direction: even when the key is secret, an observer can tell from black-box queries alone that a model is serving watermarked text (ETH SRI lab).

Why the signal stays fragile

Watermarking moved the detection problem from pattern-matching to hypothesis testing, but it did not make it airtight. Three lines of attack define the frontier.

The first is paraphrase. Sadasivan and colleagues showed that recursively rewriting AI text with other language models, one small meaning-preserving edit at a time, can defeat watermark detectors, neural detectors and retrieval-based detectors alike on passages of around 300 tokens, while only mildly degrading quality (arXiv). Follow-up work on SynthID-Text confirms it is vulnerable to paraphrasing, back-translation and copy-paste edits (arXiv).

The second is a structural impossibility result. In work picked up at ICML, researchers proved that under two natural assumptions, an attacker with a quality oracle that can judge whether text still reads well, and a perturbation oracle that can suggest edits likely to preserve quality, can strip any low-false-positive watermark without measurable quality loss. The attacker needs only black-box access to a model no stronger than the watermarked one (ICML, PMLR, IACR eprint). Intuitively: if the watermark only selects among high-quality outputs, then wandering through the space of high-quality outputs eventually lands you on one that no longer looks marked.

The third is the detection service itself. A public detector acts as an oracle, leaking token by token which choices carry the mark, which enables both removal and spoofing attacks that frame human text as AI-generated. The same line of work shows watermarking schemes are not equally vulnerable: SynthID's design cuts spoofing success rates substantially compared with plain red-green schemes (CMU CSD blog, ETH SRI lab).

The honest summary is that watermarking degrades, rather than defeats, the attacker. Paraphrasing costs the attacker effort and quality, and the trade-off between detectability, robustness and text quality is real, with each axis bounded by the others (NeurIPS 2024).

A printed row of green dots passing through a rewriting roller and emerging faded

The law arrived before the perfect detector

Which brings up the reason this research suddenly has a budget line. EU AI Act Article 50 applies from August 2, 2026: providers of systems generating synthetic text, audio, image or video must mark outputs in a machine-readable format, effective, interoperable, robust and reliable as far as technically feasible, with systems already on market getting until December 2, 2026 to comply (European Commission guidelines, Stibbe analysis). Watermarking is one permitted technique alongside metadata, cryptographic provenance and logging, but the accompanying Code of Practice goes further for free-form text: above 200 tokens, an imperceptible watermark is effectively mandatory, precisely because metadata stripped with the formatting is not enough (Orrick analysis). Regulators have, in effect, mandated a technology whose research literature concedes it cannot be unbreakable.

What it means when you read the output

An AI text watermark is a provenance audit tool, not a lie detector. It answers a narrow question, did model X with key K generate this passage, with statistical confidence, and it only helps you if the provider holds up its end: the model watermarks at generation, the detector stays private, and the text survives transit unedited. A missing mark is not evidence of a human author, and a clean z-score still does not prove a specific passage escaped paraphrase. Three years after the classifier shutdown, that remains the actual state of the art: not detecting AI text, but making willing providers able to vouch for theirs.

Sources

AILLMEducation