AI Watermarking in vLLM Costs No Throughput. Four Settings Decide Whether Your Output Carries It.

vLLM 0.30.0, released on 22 September, can watermark the text it generates, and the project's blog post two days later makes a strong case for AI watermarking on a self-hosted stack: the mark does not change the model's output distribution, and on one H100 serving Qwen3.5-27B with three-token MTP speculation the mean matched throughput change ranged from minus 1.1 to plus 2.0 percent across batch sizes from 1 to 256. The measurement is careful, with medians across eight keys and confidence intervals per batch size. What it does not tell you is how much of your own traffic ends up marked, and the shipped code answers that differently from the headline.

The claim, stated fairly

The scheme is Gumbel-max sampling with a keyed pseudorandom function. Instead of fresh random noise at each step, vLLM derives the noise from a secret key, the previous four tokens, and each candidate token, then takes the argmax. Sampling this way produces the same distribution as ordinary sampling, so quality should not move, and a detector holding the key can recompute the noise afterward and sum per-token scores into a p-value. Detection needs the key and the tokenizer, not the model weights. The docs credit the SynthID-Text work, published in Nature in October 2024, for both the non-distortion argument and the speculative-decoding protocol; SynthID-Text itself is listed as planned, not implemented.

The quality claim has a regression gate: the repository runs the full 1,319-question GSM8K set against a watermarked Qwen3.5-4B at temperature 0.7 and fails if accuracy drops more than 0.015 below 0.885, a threshold its config says was calibrated from 13 unwatermarked runs at three standard deviations.

Where the measurement stops

The blog's own detection chart holds the first boundary. At a 1 percent false-positive rate, creative writing approaches full detection by about 100 scored tokens. MBPP code output reaches roughly 69 percent at 400 tokens when you test one candidate key, and about 49 and 43 percent when you test 10 or 100. The docs tell you to test every tokenizer, key and configuration you have ever served, with a multiple-testing correction, so a team that has rotated keys or changed models is on the lower curves. This is the same entropy limit the Claude watermark post found in the SynthID-Text paper: low-entropy text carries little signal. vLLM inherits it.

Temperature 0 is never marked. Gumbel-max needs stochastic sampling. The sampler checks each request, and a greedy one goes through ordinary sampling with a warning logged once per worker. A pipeline pinned to temperature 0 for reproducible extraction or tool calls produces unwatermarked text with watermarking switched on.

Any caller can switch it off. Watermarking is off unless the server starts with a watermark config, and on for every request once it does. But the per-request watermarking field defaults to true and can be set to false by the client, through the OpenAI-compatible endpoints and the Rust frontend alike. The docs say plainly that deployments requiring the mark must restrict that field to trusted callers or strip it at ingress. Issue #58191, filed the day of the release, reports this as a bug; PR #58202 adds an enforce option, off by default, and is still open. Its test shows the opt-out request returning different text with an HTTP 200 and no warning.

Speculative decoding skips the repetition guard. Keyed noise repeats when a four-token context repeats, which can push generation into loops. vLLM counters it by sampling normally at repeated contexts, and the config logs a warning if you turn that off. The docs state the guard is not applied to accepted draft, recovery or bonus tokens. The benchmark used MTP speculation; the fix, PR #56807, is open.

The dual-key detector disagrees with the generator. Of the two shipped algorithms, only dual_key_gumbel supports speculative decoding natively. In v0.30.0 the config and the watermarker default its alpha to 0.1, and the detector class defaults to 0.2. Alpha sets the detector's score weights and variance, so a default detector scores default output with the wrong weights. The reference HTTP detection server only builds the single-key detector. The feature's author said in issue #57839 that the server is a minimal example and not production-ready, and another contributor added that an open detection endpoint lets an adversary strip or forge the mark without the key. The docs back that up: the default PRF, Philox, is not cryptographic and offers no forgery resistance.

What survives production

The throughput result survives. So does distortion-freedom per token, with the regression gate behind it. What does not survive is the idea that one flag at startup gives you watermarked output. It gives you a watermarked sampling path, and the share of traffic that goes down that path is set by four things outside that flag: request temperature, whether clients can send the opt-out field, whether speculative decoding is on, and how long and open-ended the outputs are.

This is the monitoring gap the PoC-to-production gap warns about. Silent degradation is the default failure mode. Nothing here fails loudly: a greedy request logs one warning per worker, an opt-out logs nothing, and a mismatched detector alpha returns a p-value with no warning at all.

The central claim, which you can falsify: on vLLM 0.30.0, the share of a deployment's output that a detector flags depends on those four settings, not on the watermark flag, and for temperature-0 traffic it is zero. What would prove it wrong is a detection rate on your own logged outputs, at a 1 percent false-positive rate, that matches the creative-writing curve regardless of temperature and output type.

What to check before you rely on it

1. Count the share of requests at temperature 0 in your logs. That share is unwatermarked.
2. Strip or reject the watermarking field at your gateway until an enforce option ships in a release you run.
3. If you use speculative decoding, pass alpha to the dual-key detector explicitly instead of trusting the default.
4. Measure your false-positive rate on real unwatermarked traffic with your deployed key, as the docs require, before treating is_watermarked as a verdict.
5. Score 200 logged outputs per workload and record the flagged share next to the median output length.

The detector call for step 5 is short:

from transformers import AutoTokenizer
from vllm.v1.watermarking import GumbelWatermarkDetector

tok = AutoTokenizer.from_pretrained(MODEL)
ids = tok.encode(text, add_special_tokens=False)
result = GumbelWatermarkDetector(key=KEY, prf='philox').detect(ids)
print(result.p_value, result.is_watermarked)

If the flagged share for a workload is under half, that workload needs a provenance record kept at generation time, not a detector run afterward.