Cloudflare shipped a robots.txt setting on September 15 that blocks AI training without blocking Googlebot, Bingbot or Applebot. It's called Disallow AI Training, and it's available on every Cloudflare plan, including the free tier.

Until now, site owners faced a bundled problem. The three search crawlers also feed AI training, so blocking one meant losing your search listings. Cloudflare says fewer than 1% of sites on its network block search bots. Meanwhile, 17% have turned on some form of training block. That gap is exactly what this setting addresses.

The move follows a July announcement. Cloudflare said then that from September 15, any site blocking AI training would also block the three search crawlers. Talks with Google, Microsoft and Apple followed, and each company has now added a training opt-out or promised one by a set date.

How does Disallow AI Training actually work?

When you enable the setting, Cloudflare's Bot Preference Sync writes two rules into your robots.txt: one disallowing Google-Extended and one disallowing Applebot-Extended. Those tokens serve AI training only, so search indexing stays untouched.

Amazon, Anthropic, Meta and OpenAI already run separate crawlers for training and search. Cloudflare blocks their training crawlers outright. Your pages should still be readable by ChatGPT search and similar tools.

Googlebot, Bingbot and Applebot stay allowed because Cloudflare now labels their operators "Accountable." That label requires four commitments:

• An opt-out from AI training via robots.txt or a similar standard
• An opt-out from AI summaries
• URL-level visibility into which pages were used for training
• Assurance that a training opt-out won't affect search rankings

Google says its URL-level reporting arrives in the coming weeks. Apple's is planned for next year. Microsoft lags furthest: Bingbot won't read the robots.txt rule until early 2027. For now, the only way to keep Bing from training on a page is a NOARCHIVE meta tag.

What does the early data show?

SeenSure tested 1,046 websites against eight AI crawlers on September 14, then again on September 16. Of the sample, 746 sites use Cloudflare, and more of them refused training crawlers after the change:

• GPTBot refusals rose from 18.9% to 22.0%
• ClaudeBot refusals rose from 19.9% to 22.8%

The surprise sits elsewhere. Cloudflare's announcement never mentioned AI search crawlers or agents, yet refusals of those fell by about 13 points each. OAI-SearchBot, which feeds ChatGPT search, was refused by 16.9% of Cloudflare sites before the change and 3.0% after. The 300 non-Cloudflare sites in the sample stayed flat.

If the pattern holds, ChatGPT search and Perplexity can now read many Cloudflare-hosted pages they were denied the week before. "Search moved the most of any category," SEO consultant Aleyda Solis wrote on X when she shared the findings.

Does opting out of training remove you from AI answers?

No. The setting governs whether your content trains models, not whether you appear in AI answers. Site owners now face three separate choices — search indexing, AI training and AI answers — and Cloudflare's control covers only the middle one. Cloudflare says AI summary controls come next year.

Opting out isn't automatic advice either. For sites that don't run ads, Cloudflare's recommended default leaves training set to Allow. A brand that wants AI tools to know and mention it could benefit from appearing in training data, though no AI company has quantified how much that matters. Cloudflare recommends Disallow AI Training for publishers monetizing content through ads.

One caveat on enforcement. A robots.txt rule is a request, and a crawler can ignore it. Cloudflare can detect and block crawlers that do. But with Google, Apple and Microsoft, it relies on commitments made to a company that sells bot-blocking products and wrote the "Accountable" standard itself. Cloudflare says it will track their progress publicly on Cloudflare Radar.

How do you check your own setup?

Semrush's Site Audit includes a Blocked from AI Search check that reads robots.txt and lists which AI crawlers are blocked on which pages. Google-Extended is on its list, so it will show as blocked once Disallow AI Training is on. Googlebot, OAI-SearchBot and the other search crawlers shouldn't appear blocked.

Site Audit reads robots.txt only. It won't catch a block made at Cloudflare's firewall. Your server logs will, so confirm the search crawlers you want still receive 200 status codes.

For measuring downstream impact, the AI Visibility Toolkit tracks brand and page appearances across ChatGPT, Google AI Mode, AI Overviews and Gemini. Its Cited Pages tab lists which URLs get cited and by how many prompts. Record your numbers before flipping the setting, then compare a few weeks later — that comparison is the only way to know whether the training opt-out shifted your AI answer visibility at all.