To Block AI Bots, Longest Match Beats File Order

Cloudflare announced Bot Preference Sync on 21 August 2026. The robots.txt your site serves is now generated from the AI bot policy you already set in the dashboard, so the file that states your preference and the rule that enforces it stop drifting apart. It is available from Free through Enterprise, and the announcement describes it as on by default for new customers, with existing customers on the older managed robots.txt feature prompted to review and confirm their settings during the transition.

The pitch is alignment. The question the pitch raises is what alignment means when the served file has two authors, and that answer is in the docs and in RFC 9309 rather than in the announcement.

What the generated file actually contains

The managed robots.txt documentation names the eight crawler tokens that get Disallow: / — Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot, and meta-externalagent. For the wildcard user-agent it writes Allow: / together with a content signal line declaring search=yes, ai-train=no, use=reference.

Sit with that wildcard group for a second. It allows everything and declares that training is not permitted. Those are not contradictory in the vocabulary the docs set up — the signal describes permitted use, the Allow describes permitted fetching — but they land in the same group, and only one of the two is a field the standard defines. RFC 9309 section 2.2.4 lets a crawler interpret records outside the protocol, provided that doing so does not interfere with parsing the records the protocol does define. A parser that handles only defined fields reads Allow: / and stops. The content signal is a claim about permitted use, enforceable by contract or by law if at all, never by the parser.

Both the announcement and the docs say the managed block is prepended to an existing origin robots.txt and returned as a single response. Neither says what prepending means to a crawler.

Why prepending does not block AI bots you already allowed

Prepend sounds like precedence. Under RFC 9309 it is not.

Section 2.2.1 requires a crawler to match its product token case-insensitively, and when more than one group carries the same user-agent value, to combine those groups into one. Section 2.2.2 then says the most specific match must be used, where most specific means the most octets, with ties resolved in favour of the allow rule. Neither section mentions file order.

So picture a site whose own robots.txt already carried this, from back when it wanted its documentation reachable by an AI answer engine:

User-agent: GPTBot
Allow: /docs/

Turn the sync on. The served file now has two GPTBot groups: Cloudflare's Disallow: /, and yours. A conforming crawler merges them, then evaluates a request for /docs/guide. The path /docs/ is six octets, / is one, so the allow wins. Everything outside /docs/ is disallowed and /docs/ is served exactly as before, while the dashboard reports that training is blocked.

The mechanism runs the other way too, which is why this is a narrow problem rather than a broken feature. Cloudflare's wildcard Allow: / merges with your wildcard group as well, and it loses to a Disallow: /admin/ you wrote years ago for the same reason — seven octets beats one. Your existing exclusions survive. What does not survive is the assumption that a prepended Disallow: / closes a door you had deliberately propped open for one of those eight tokens.

What changes if you were already running the old feature

The migration is a decision rather than a silent rewrite: the announcement says customers on the legacy managed robots.txt behaviour get prompted to confirm. The vocabulary is the part to re-read. Search and Agent access has three settings — allow, block on pages that serve ads, block everywhere — and Training has two, allow or disallow. A publisher who indicates during onboarding that they monetise pages carrying ads gets Training set to disallow by default.

Block on pages that serve ads is the setting that will generate support tickets, because it makes your stated crawl policy a function of which pages carry ads, which is a marketing decision rather than an infrastructure one.

The verdict, and the number to watch

Turn it on, or confirm it if the prompt has already found you. A generated file that tracks the dashboard beats a hand-maintained one that does not, and Cloudflare's own docs are direct about the ceiling: compliance is voluntary, robots.txt does not technically prevent access, and AI Crawl Control is where enforcement lives.

Then stop treating the toggle as the control. This is the gap described in why enterprise AI doesn't ship wearing different clothes — the configured intention is the demo, and the enforced, measured path is the last mile.

Three things to do this week:

1. Run curl -s https://yourdomain.example/robots.txt | grep -in user-agent and count the groups. Two entries for the same token means merged rules, and your own group needs reading for any path narrower than /.
2. For each of the eight tokens, decide whether your answer is a preference or a rule. A preference belongs in robots.txt. A rule belongs in AI Crawl Control or the WAF.
3. Watch the robots.txt violations table in AI Crawl Control, which counts requests to disallowed paths per crawler over a window you choose. Threshold: any non-zero count for a token you disallowed, sustained across seven days, means that operator is not honouring the declaration and the token needs an enforced block instead. Standing measurement is exactly the line item a proof of concept skips, which getting from PoC to production covers at length.

Two smaller notes. Section 2.4 says a crawler should not use a cached robots.txt for longer than 24 hours, so expect up to a day between flipping the toggle and the new file being read. And section 2.5 puts the parsing floor at 500 kibibytes, so the prepended block costs you nothing you will ever measure.

What would show this reading is wrong: publish a group of User-agent: GPTBot with Allow: /docs/, enable the sync, and fetch your own /robots.txt. If the response carries a single GPTBot group, Cloudflare is replacing rather than prepending, and the merge question never arises.