Cloudflare Signals Help ChatGPT Index Hundreds of Millions of New Pages

A research pilot between Cloudflare and OpenAI has helped OpenAI discover and index hundreds of millions of new web pages — and it ran on data from just 1% of Cloudflare's total network. The two companies shared those early results on October 15 in a Cloudflare blog post coauthored by two OpenAI staff members, roughly three months after the pilot was announced on July 8.

The scale of the discovery figure is the story. One percent of Cloudflare's network — one of the largest on the internet — feeding hundreds of millions of previously unseen pages into OpenAI's index signals a new route by which content enters ChatGPT's search results, one that depends less on the traditional link graph and more on infrastructure-level signals.

What the post does not report matters just as much. It gives no data on how often the newly found pages were cited in ChatGPT or how many visits they sent. Finding new pages "can initially increase crawling as those pages are evaluated and indexed," the companies acknowledge. No figures on crawl volume or server load appear either.

What do Cloudflare's signals actually do?

Cloudflare's network signals help OpenAI's search crawler identify pages with value. The blog post names concrete examples:

• the user guide for a particular washing machine;
• current information on passport requirements and fees;
• a restaurant's menu and opening hours.

These pages, the companies write, "can be valuable even if few other sites link to them" — a direct challenge to the link-based discovery model that has governed crawling since the early days of web search.

The July 8 announcement detailed which signals participate: how recently a page was updated, the quality of traffic reaching a website, and what changes were actually made to a page. That data comes from participating websites, though neither announcement identifies who those participants are or whether the newly discovered pages sit on their sites.

The signals also work in the other direction — trimming waste. The October 15 post says they help OpenAI's crawlers avoid unnecessary requests. Bots repeatedly fetch files that no longer exist on a server; some sites redirect nearly any address to the homepage; others serve duplicate content from multiple URLs. Change signals tell the crawler whether revisiting a page would yield anything of value. The stated goal is reducing unnecessary load for Cloudflare customers.

The companies are additionally testing signals that flag potentially harmful pages. Their example: a fake login form an attacker planted on a single page of an otherwise legitimate site, where the aim is to flag that page rather than the entire domain. Early results suggest the signals can help identify pages to keep out of the index.

Does a signal grant access?

No — and the post says so explicitly. "A signal about a page doesn't grant permission to access it," the companies state. Crawler settings still decide.

OpenAI's crawler documentation separates search from training via two robots.txt directives. OAI-SearchBot gathers information for search results; GPTBot collects content that may be used for model training. Block OAI-SearchBot and a site will not appear in ChatGPT's search results, though it may still surface as a navigational link. Allow both, and OpenAI may use data from a single crawl for both purposes.

The October 15 post also clarifies that researchers are using metadata from the web to improve search and discovery, not sharing page content for model training.

For practical control, the post points to Cloudflare's AI Crawl Control, available on every plan. It shows which AI services are accessing a site and offers an Allow or Block toggle for each crawler. OpenAI's crawler activity also appears in standard server logs.

What does this change for site owners?

The honest answer: visibility without any citation guarantee. Once a page enters OpenAI's index, ChatGPT has another source from which to pull information — but the companies state that OpenAI independently evaluates and ranks the pages it indexes, so being discovered through the pilot does not mean being cited.

Publishers wanting search exposure without contributing to training can allow OAI-SearchBot while blocking GPTBot — a configuration OpenAI itself presents as an example on its crawler page.

Measurement remains a weak point. Tracking ChatGPT visits depends on a UTM tag or a referrer, and earlier this month some ChatGPT links carried neither; an analyst said a fix began rolling out on October 15.

What comes next?

The companies' stated next steps are improving how an AI selects a web page, testing faster and more reliable indicators of content change, and measuring outcomes once the AI has found new content. Cloudflare also says it is working with OpenAI to determine how improved content discovery could translate into business outcomes.

According to Cloudflare's July 1 blog post announcing the underlying research, broad access to the network-signals capability is intended at some point in the remainder of 2026 — though the October 15 announcement gave no specific date. Until then, publishers watching for OpenAI's crawlers will find the clearest evidence in server logs and AI Crawl Control.