ChatGPT Found Your Hidden Pages

On October 15, Cloudflare and OpenAI shared early results from a research pilot they started in July 2026. Signals from Cloudflare's network helped OpenAI's search crawler discover and index hundreds of millions of new web pages for ChatGPT search, according to the Cloudflare blog post announcing the results. The pilot covered a participating cohort of about 1 percent of Cloudflare's total network, per Search Engine Journal's report on the same announcement.

The pilot widened discovery. ChatGPT can now find pages that link-based crawling never reached.

The pages link-based crawling never reached

A traditional crawler finds pages by following links. A page with few or no inbound links can sit unpublished in practice, however useful it is. Cloudflare's examples of the kind of page that matters here: the manual for one specific washing machine model, current passport requirements and fees, a local restaurant's latest menu and opening hours. The post puts it plainly: these pages can be valuable even if few other sites link to them.

The same pattern runs through local SEO. The pages that answer a customer's exact question are often the least linked pages on a site: a service page for one treatment, one fee schedule, one set of hours. Nobody links to a clinic's fee page from a blog roundup. A customer still asks ChatGPT about it.

I learned that on a Winnipeg therapy clinic campaign where calls rose 300 percent. The pages doing the work were specific service pages answering specific local questions, in plain language. I wrote about the same pattern from the other side in my post on winning Google's official-site box: when an answer engine goes looking for one fact, it wants the page that states that fact. The Cloudflare pilot means more of those pages are now in ChatGPT's index.

Three things the pilot did not change

First, there is no toggle to join. The pilot ran on a participating cohort, and neither announcement describes a setting a site owner can flip to opt in. What you control is whether those pages are linked, readable, and quotable when a crawler arrives.

Second, a signal is not permission. Cloudflare states it directly in the post: a signal about a page does not grant permission to access it. Your robots.txt and your Cloudflare settings still decide access, and the research uses web metadata for search and discovery rather than sharing page content for model training. Discovery got easier, and your existing access controls still decide what a crawler can open.

Third, indexed pages still have to earn citation and traffic. Search Engine Journal notes the October 15 post reports no figures for how often the newly indexed pages were cited in ChatGPT answers or how many visits they sent. OpenAI independently evaluates and ranks the pages it indexes, so a page can be found, indexed, and still never appear in an answer.

The 6-step audit I run on pages ChatGPT can now find

I run this check on client sites at First Rank before I chase any new AI feature, because new discovery surfaces expose the same crawl problems.

1. Split your OpenAI bot rules in two. OpenAI separates search from training into two robots.txt directives, as Search Engine Journal summarizes from OpenAI's crawler documentation. OAI-SearchBot is the bot that feeds ChatGPT search results. GPTBot collects content that may be used for training. Blocking OAI-SearchBot keeps your pages out of ChatGPT search results, though the site may still appear as a navigational link. Allowing OAI-SearchBot while blocking GPTBot keeps your site searchable in ChatGPT while telling OpenAI not to use the crawled content for training. Read your robots.txt and confirm you made that choice on purpose.

2. Check Cloudflare AI Crawl Control. If your site runs on Cloudflare, open AI Crawl Control and look at which AI services are hitting your site and whether each is set to Allow or Block. Check both places: the more restrictive setting wins. An AI Crawl Control block stops the bot even if robots.txt allows it, and a robots.txt block still applies even if AI Crawl Control allows it.

3. Find your orphan pages. Export your sitemap, then list the pages with no internal links pointing at them. Those are your link-invisible pages: the fee schedule, the single-service page, the updated hours page. If a page matters to a customer question, give it at least one internal link from a page that already gets crawled.

4. Kill the three crawl traps Cloudflare named. Bots repeatedly request files that do not exist, which can make a missing page look interesting. Some sites redirect almost any address to their homepage, which turns every typo into a fake page. Other addresses serve copies of the same content. Check your 404 log, your catch-all redirect rule, and your duplicate URLs. Fix the redirect rule first. A homepage redirect on an unknown address tells every crawler your site has one page with a thousand addresses.

5. Put the fact in plain text on the page. If the price lives in a script, the fee schedule lives in a PDF, and the hours live in an image, an answer engine that reaches the page still has nothing it can quote. Put the one fact the page exists to answer in readable text near the top: the price, the fee, the hours, the requirement. ChatGPT cites the page that states the fact in usable form, as I found in my docs-over-blogs post.

6. Measure whether discovery turned into anything. I track this with the fixed prompt panel I use across client work: the same 10 money questions, asked across ChatGPT, Gemini, Perplexity, and Claude, logged in and logged out, every Monday, with cited sources recorded each time. I log the panel results alongside each post at dalindegraff.com. Pair that with your GA4 AI referral channel and watch for the step 3 pages showing up as cited sources. One caution from my own recent work: ChatGPT's UTM tags broke with the GPT-6 rollout in October, so a ChatGPT visit can land in Direct and look like no traffic at all. SEJ reported a fix beginning to roll out Oct 15, so check your own data rather than assuming the break is still total. I covered that measurement break in my post on the missing UTM tags. If you only watch one channel, you will conclude the pilot sent you nothing when it may have sent you visits you mislabeled.

What to do this week

Start with steps 1 and 2. They decide whether any of this applies to you at all. Then run step 3 and pick the five orphan pages that answer real customer questions. Link them, fix their facts into plain text, and add them to your Monday prompt panel.

Which page on your site answers your best customer question but has no links pointing at it? That page is your starting point.