Google Commits to IETF Robots.txt AI Training Signal for 2027

Google says it will implement a shared robots.txt AI training control in 2027, following a draft proposal from the AIPREF working group at the IETF. The announcement mirrors a move Bing disclosed earlier, and it targets one persistent publisher complaint: managing separate opt-out instructions for every AI developer.

Google framed the change in a statement: "This will help websites express their AI training preferences across multiple AI providers in a single step and is based on work from the AIPREF working group at the Internet Engineering Task Force (IETF)."

That sentence carries two claims. One is concrete: a single-step mechanism. The other is a promise about an unfinished standard, and the IETF has not finalized it.

What does the new control actually do?

The mechanism, still a work in progress, lets websites state their AI training preferences once. Instead of listing each crawler individually, a site owner would place a shared "policy token" in robots.txt. That token would set broad preferences across every participating AI provider at once.

The benefit is scale. One instruction covers many companies. The catch is participation — the token only binds providers that choose to honor it. Google named no peers beyond itself and Bing's parallel announcement.

Google also positioned the move as a response to outside pressure. The company said it is "moving forward with this initiative alongside peer AI providers as part of a shared commitment to foster an open, creator-first web ecosystem, and following feedback from publishers, civil society and regulators calling for an easier way for websites to express their AI training preferences."

How would Google interpret the signal?

John Mueller, Google Search Advocate, spelled out the interpretation rule in a public post:

"When a website uses this policy token to opt out of training in its robots.txt file (and Google-Extended is not explicitly specified), Google plans to interpret that as if the publisher had disallowed Google-Extended. Other AI providers can handle it similarly, making it easy for website owners to opt out across all providers, or to pick and choose."

Note the conditional. If a site explicitly specifies Google-Extended, that instruction takes precedence. The token is a broad default, not a replacement for company-specific controls.

Google confirmed existing controls will keep functioning alongside the new joint signal. Site owners can set an industry-wide preference and still tailor instructions to individual developers through mechanisms like Google-Extended. Google added one incentive: websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.

Publishers should also remember the separate Search Console opt-out method for Search generative AI. That control remains distinct from the robots.txt token.

Is this a standard yet? No.

The proposal is a draft. Google itself said the new mechanism brings website owners "the benefit of the AI training definition developed at the IETF while multistakeholder discussions continue on the rest of the AIPREF vocabulary."

In plain terms: the definition of AI training is settled enough to build on. The rest of the AIPREF vocabulary is not.

Google argued the draft reflects sustained feedback from publishers, civil society groups, regulators and other stakeholders who have called for industry-wide controls on AI training. The company said support across major AI providers would enable a future standard that is "cross-industry, robust, and durable" — three adjectives attached to a document that could still change in the IETF process.

The announcement also carries a regulatory anchor. Google said the proposal is consistent with the EU Code of Practice for General Purpose AI Model Providers under the EU AI Act, which recognizes robots.txt as a primary method for web publishers to express machine-readable preferences. The company committed to "continue to work with these stakeholders to refine and standardize this through the IETF working group."

What publishers should watch

The timeline is long. Google aims to support the control, alongside other AI providers, in 2027, and promised updates on progress.

Between now and then, three things matter:

• Whether the AIPREF working group finalizes the token's specification.
• Which AI providers beyond Google and Bing commit to honoring it.
• How the signal interacts with per-company controls like Google-Extended and the Gemini grounding eligibility rule.

Takeaway: publishers gain a promised single-step opt-out, but the promise matures in 2027 and depends on an unfinished IETF standard and voluntary adoption by competitors Google has not yet named.