One Robots.txt Line Will Soon Tell Google, Microsoft, and OpenAI No to Training

On October 30, Google, Microsoft, and OpenAI each announced plans to honor one shared robots.txt line for AI training preferences. Add a group for the policy token __usage_ai-training, disallow the paths you want kept out of training, and every participating company reads the same preference alongside its own crawler rules. Google's Search Central post frames it as one step that replaces separate instructions per AI developer.

My position: add the training opt-out now, but refuse Google's bundle. Pair the shared line with an explicit Google-Extended allow, because opting out of training should not cost a site the Gemini Apps grounding visibility Google has tied to it.

Three limits qualify that announcement. Nobody honors the line today, it answers only the training question, and at Google it also removes Gemini Apps grounding unless the site speaks for itself. Settle three separate decisions before you paste anything, because the new line settles only one of them.

What each company has said so far

Google aims to support the shared control in 2027, with its crawler documentation narrowing that to early 2027. When a site opts out through the shared token and does not explicitly mention Google-Extended, Google plans to read that as a Google-Extended disallow. A site that specifically allows Google-Extended stays eligible to appear in Gemini Apps responses through grounding.

Microsoft's Bing announcement adopts the shared line for training, adds a separate page-level grounding preference, and honors both from February 1, 2027. A page-level training preference set with a meta tag or HTTP header overrides robots.txt for that page.

OpenAI aims to honor the signal in the coming months, and its current robots.txt policy stays in effect until then. What happens when a shared-line rule conflicts with a GPTBot rule is unstated in OpenAI's post and crawler documentation. Anthropic has published no announcement of its own, though the draft was written by Kevin Kelley of Anthropic, and Kelley told the working group's mailing list Anthropic has started developing support.

The mechanism comes from an Internet-Draft filed October 30 with the IETF's AI Preferences working group. It is an individual submission, not an endorsed standard, and it expires May 3, 2027, unless a new version replaces it. Its definition is narrow on purpose: it covers changing model parameters, not retrieving content to answer, which is why grounding needs its own decision.

A file with no shared group says nothing about training

A wildcard group, User-Agent: *, must not be read as matching the policy token. A file with no __usage_ai-training group expresses no preference at all, whatever its other rules say, which Kelley states is true of every robots.txt file published today.

Nearly every site sits there right now: silence in the file is not a refusal. If you want a preference on record for the day support arrives, you have to write the group. Adding it now is safe: parsers that do not recognize the token ignore the group, and a file written early is already in place when each company starts reading the line.

A usage preference is independent from a crawl rule: the token does not tell any bot to stay off your pages, as I covered when I wrote about how ChatGPT finds unlinked pages (ChatGPT Found Your Hidden Pages).

At Google, refusing training also refuses Gemini grounding

Google-Extended covers two uses at once: training future Gemini models, and grounding in Gemini Apps and on Vertex AI. Grounding means supplying content from the search index to the model when it answers. A shared opt-out therefore also removes Gemini Apps grounding, unless the site explicitly allows Google-Extended. Google's documentation describes no way to refuse training while keeping Gemini Apps grounding.

Training affects future models. Grounding affects answers now. Microsoft lets a site refuse training while still allowing grounding. At Google, one line gives the same answer to both, so a publisher that gets traffic from Gemini Apps answers loses that grounding visibility when it refuses training.

That bundle is what I argue against. Training and grounding are different uses with different costs, and Microsoft proved they can be separated. The explicit Google-Extended allow separates them at Google today: the shared line carries the training refusal, and the allow keeps the grounding eligibility. Leaving Google-Extended unmentioned is the worst option, because Google reads silence as a disallow and takes both.

Search AI answers need a separate opt-out

The shared line does not remove a site from AI answers. AI Overviews and AI Mode have a separate opt-out, the Search generative AI control in Search Console, and the shared line does not change it. Google-Extended does not affect inclusion in Google Search at all.

Microsoft has its own control. Grounding is set per page with a meta tag or HTTP header, with three settings: Allow, the default; Disallow, which keeps the page out of AI answers; and DisallowExtended, which permits grounding in Microsoft's own products but not in third-party grounding services. Microsoft says either restriction limits visibility in AI answers, and neither changes eligibility for ordinary Bing results.

Microsoft's deadline applies even to sites that do nothing

Microsoft's announcement is the only one that retires controls sites already use. After March 1, 2027, Bing stops reading NoArchive and NoCache as training or grounding signals. Sites that rely on those tags to limit Copilot must publish the new preferences before that date.

A temporary grounding preference in Bing Webmaster Tools opens December 15, only for sites still using the old tags on that date. Pages with no recorded preference by March 1 default to allowing grounding until Microsoft recrawls them, and once Bing detects a new preference on a site, it disregards the old tags there.

The worksheet I run with clients

The October 27 audit had three decisions: crawl access, Search AI surfacing, and training (Your Robots.txt Does Not Stop AI Training). The October 30 announcement splits the third one. This is the worksheet I now run with the clients I manage at First Rank:

1. Decide training, per path. The whole site, or carve-outs such as a press kit. Write the shared group now, and pair it with an explicit Google-Extended allow so the training refusal does not become a grounding refusal at Google. Keep the GPTBot rules in place until OpenAI confirms it reads the shared line.
2. Decide grounding separately, per company. At Google, the explicit allow keeps Gemini Apps grounding; silence or an unpaired opt-out removes it. At Microsoft, set the page-level grounding tag on the money pages, not on the site root alone.
3. Decide Search AI separately. Google's Search generative AI control in Search Console governs AI Overviews and AI Mode, and the shared line leaves it untouched.

Crawl access stays as it was: the token does not change crawler groups.

Put five dates on the calendar: December 15 for the Bing bridge check, February 1 for Bing's live date, March 1 for the NoArchive and NoCache retirement, early 2027 for Google, and May 3, 2027, when the draft expires.

My Monday panel from the October 27 audit runs ten money questions three times each across ChatGPT, Gemini, Perplexity, and Claude, logged in and logged out. That panel gains a column for Gemini Apps and Copilot grounding as support arrives: a grounding change shows up as citation loss, not ranking loss, so a panel that only tracks positions will miss it.

Which of the three have you decided for your own site: training, grounding, or Search AI?