Google, Microsoft and OpenAI Commit to Shared AI Training Opt-Out Signal

Google, Microsoft and OpenAI announced on Friday that they will honor a single shared robots.txt line for AI training preferences, letting a site state one opt-out to every participating provider instead of writing rules for each company's crawler. None of the three supports the line today.

The signal comes from an Internet Draft that Anthropic's Kevin Kelley submitted to the AI Preferences working group at the IETF on the same date. The IETF's document tracker lists it as an individual draft, which does not represent official IETF standardization activity. In a post to the working group's email list, Kelley said Anthropic has started development to provide the feature.

How does the shared line work?

Kelley's draft adds no new directive to robots.txt. A site writes an ordinary rule group under what the draft calls a policy token, __usage_ai-training, using the familiar Allow and Disallow directives. The draft gives an example that opts out of AI training site-wide while keeping the /press-kit/ folder open:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

The token draws its meaning from the working group's vocabulary draft, which defines AI Training as using content to modify a generative AI model's learned parameters. Models that only perform non-generative tasks, such as classification or ranking, fall outside that definition.

Existing robots.txt rules will not count as a shared preference. A wildcard group (User-Agent: *) can never match the token, and a file without a __usage_ai-training group signals no preference — which Kelley says is true of every robots.txt currently published. The token also does not control crawling.

On Oct. 22, this publication covered a different working group draft, co-written by Google's Gary Illyes, that would add a Content-Usage rule to robots.txt. Kelley wrote that the group set that draft aside to make progress on the vocabulary first.

What does Google's implementation look like?

Google's post, written by John Mueller, says that when a site opts out via the shared line and does not specify Google-Extended, Google plans to interpret that "as if the publisher had disallowed Google-Extended." Google aims to support the control in 2027; its crawler documentation says early 2027.

That matters because Google-Extended governs two things at once: whether content can train future Gemini models and whether it can be used for grounding in Gemini Apps and on Vertex AI. Grounding, in Google's description, means supplying content from the search index to the model when it answers. Sites that specifically allow Google-Extended become eligible to appear in Gemini Apps responses through grounding.

Google's documentation confirms Google-Extended does not affect inclusion in Google Search. AI Overviews and AI Mode carry a separate opt-out, the Search generative AI control in Search Console, which rolled out worldwide in August.

How does Microsoft's approach differ?

Microsoft's post, by Bing principal product manager Krishna Madhavan, adopts the shared line for training and adds a separate AI Grounding preference set only at the page level. Microsoft also allows a meta tag or HTTP header to set the training preference for a single page, overriding robots.txt for that page.

Grounding defaults to Allow, and Disallow keeps a page out. A third option, DisallowExtended, permits grounding in Microsoft's own tools like Copilot and Bing's AI Mode but blocks grounding through Microsoft-provided grounding services used by other businesses, known as Web IQ.

Microsoft said either restriction will limit a brand's visibility in AI answers and will have no effect on listings in Bing search results.

What must NoArchive and NoCache users do?

Microsoft is the only company replacing controls sites already use. It will retire NoArchive and NoCache as training and grounding signals and begin honoring the new options on February 1, 2027, as it crawls. It stops recognizing the older tags after March 1, 2027.

Sites using the old tags that want to stay out of grounding must record a temporary preference in Bing Webmaster Tools. That setting opens Dec. 15 and is available only to sites still using those tags on that date. Microsoft's FAQ says pages without a recorded preference by March 1 default to allowing grounding until Microsoft recrawls them. Once Microsoft detects either new preference on a site, it will disregard any remaining NoArchive or NoCache tags there.

Where do OpenAI and Anthropic stand?

OpenAI aims to honor the signal in the next few months, and its current robots.txt policy remains in effect. Neither OpenAI's blog post nor its crawler documentation, checked Oct. 30, says what happens when a shared line rule and a GPTBot rule conflict.

As of Oct. 30, Anthropic's newsroom had no posting on the signal. Kelley's mailing list message says implementing support across all Anthropic services will take time. Anthropic's crawler-help page, also checked Oct. 30, lists only three crawl control options: ClaudeBot, Claude-User and Claude-SearchBot.

Why does this matter for publishers?

One robots.txt group could eventually replace a stack of per-company training rules. But the shared opt-out will not reach the same uses at Google and Microsoft. Microsoft splits training and grounding into separate settings, while Google bundles both under Google-Extended — its crawler documentation describes no way to block Gemini training while still allowing Gemini Apps grounding.

Google, OpenAI and Kelley all confirm existing controls will keep working, so sites relying on NoArchive or NoCache to limit Copilot's use of their content face a migration task before March 1, 2027. OpenAI's post encourages other AI providers to honor the signal, and Kelley wrote that he hopes others will join. Google and OpenAI say they will share more details as support arrives; Kelley's draft expires May 3, 2027 unless a new version replaces it, and he plans to propose his design as a change to the Content-Usage draft.