Your Robots.txt Does Not Stop AI Training

A client asked me last month whether blocking GPTBot in robots.txt would keep their pricing pages out of AI training. The checklist line they quoted me read: "Block GPTBot to opt out of AI training." It was wrong, and the fix they were about to make would have protected nothing they wanted to protect.

On October 21, 2026, Google stated it plainly. In the second episode of its Search Central series The Non-Canonical Cabinet, Gary Illyes called robots.txt an "access preference protocol" and said it "doesn't dictate what a system does with the data after it has already crawled it." Search Engine Journal covered the episode the next day. Robots.txt decides who may knock on the door, not what happens to the content once a crawler has it.

Google's parser only speaks four words

Google's robots.txt specification supports four elements: User-Agent, Allow, Disallow, and Sitemap. Everything else in the file is ignored silently by Google's parser, with no warning in the file itself.

The episode made the point with a fictional directive, No-AI-Chicken-Training, shown on screen to demonstrate that a parser discards lines it does not recognize. An invented line like "NoAI" does exactly as much in Google's parser as the chicken directive, which is nothing. A site owner adds such a line, checks that the file loads, and walks away believing a decision was made.

A Disallow rule does not remove a page from the index either, since a blocked crawler never reads its noindex tag, so links from other sites can still put the URL in the index, as John Mueller explained in September 2024.

Three decisions, not one file

Decision one is crawl access, and it works per crawler. To keep one AI crawler out while letting others in, you name that crawler's user-agent token explicitly. GPTBot collects training data for OpenAI, OAI-SearchBot builds ChatGPT's search index, and ChatGPT-User fetches a page when a person asks about it. Blocking the first leaves the other two free to fetch, which is why a site that blocks GPTBot can still appear in ChatGPT answers. The same split I covered in ChatGPT Found Your Hidden Pages. OpenAI says robots.txt rules may not apply to user-triggered ChatGPT-User fetches, so a ChatGPT-User block is not a guaranteed shut-off the way a GPTBot or OAI-SearchBot Disallow is.

Decision two is surfacing in Google's AI answers, and it has nothing to do with robots.txt. The control rolled out worldwide on August 31, 2026; October 21 only consolidated it onto the "Control what you share" page. The Search Console control is site-wide: include, exclude, or inherit from a parent property. Exclude removes your content as an input for AI Overviews, AI Mode, and generative features in Discover, and Google says it is not a ranking signal for the rest of Search. Most content is excluded within 1-2 days of the control going live, but Google warns some content takes longer.

For finer control, the snippet rules work at page and passage level. Nosnippet keeps a page's text from being used as a direct input for AI Overviews and AI Mode, and data-nosnippet does the same for a marked passage, such as a price table wrapped in a div, span, or section element. Nosnippet is not a free AI-only switch: it also removes the ordinary text snippet. The page must also be crawlable for Google to read either rule.

Decision three is training, and for Google it lives in a token that is not a crawler at all. Google-Extended has no user agent of its own. It tells Google whether content it crawls may train future Gemini models or ground Gemini Apps and Vertex AI. Google-Extended does not affect Search inclusion or ranking, and it does not stop fetching: Googlebot still crawls. It limits future Gemini training and grounding use, including training of the models behind Search AI responses, and it does not recall content already used. Neither reaches already-collected content, copies on other sites, or third-party datasets.

Most top sites' homepages allow training crawlers

SEOScoreAPI tested 15 AI crawler tokens against the homepage of the Tranco top 10,000 sites on October 17, 2026. Of the 6,894 sites that gave a readable answer, 84.2 percent blocked no AI training crawler at all, only 4.2 percent blocked all nine training crawlers tested, and 6.0 percent blocked at least one training crawler while allowing every AI search and user-triggered crawler. Homepage only: sites with no file count as allowing, and robots.txt states policy, not enforced blocking.

The Meta twist, delayed to November 20

The same week gave a second reason to stop treating one document as the whole answer. Meta's upcoming commercial terms, originally scheduled for October 30 and now delayed to November 20, 2026 after publisher complaints, originally said Meta could use content from domains associated with business accounts to "provide, develop, improve, research, train, and maintain the integrity of" its products. After Nieman Lab's legal experts reviewed the wording, Meta removed "train" and narrowed the purpose to advertising and business products, as TechCrunch reported on October 17. The crawl permission remains, and TechCrunch's coverage notes that "develop" is still broad enough to cover training. Meta says the revision "does not change how Meta uses web crawlers," but that assurance sits in a statement, not in the terms.

The audit I run

Here is the sequence I use, published from dalindegraff.com. First, read the live robots.txt and list every AI token it names, grouped by job: training, search index, user-triggered fetch. Verify any token you cannot place in one of those groups. Second, check the Search Console property for the Search generative AI control, including whether a smaller property inherits a parent's setting. An inherited exclude is the surprise I check first. Third, decide surfacing at the right grain: site-wide switch, page-level nosnippet, or passage-level data-nosnippet. Fourth, set the training tokens separately.

Then measure. Google measures impressions, not clicks or queries, so it will not tell you whether a citation moved traffic. My Monday routine is a fixed panel of ten money questions run across ChatGPT, Gemini, Perplexity, and Claude, logged in and logged out, three samples each, recording which pages get cited. That panel shows whether a control change moved a citation or only moved a setting. It is the same GEO discipline as a First Rank campaign that lifted LLM search traffic 98 percent in 30 days.

A Winnipeg therapy clinic I work with grew calls 300 percent through local search visibility, which a careless block-all-AI rule would put at risk. The same identity-first thinking runs through How to Win Google's Official-Site Box: a blocklist that makes you unrecognizable is not protection.

Two traps close the audit. The first is the invented directive: if a line is not one of the four Google supports, it does nothing in Google's parser. Keep other signals only if you know which system reads them; do not delete them blindly. The second is the stale list: re-check the token list every quarter, because a crawler launched after your last edit is allowed by default. Until a usage rule exists in the standard, the advice is narrower than the checklist version: robots.txt decides who may fetch.

If you run the audit, start with one page: open your robots.txt and check each line protecting you against the four words the parser reads. Which of your lines survives that test?