Google Search Signals: 503s, AI Crawlers, URLs and Images

By Nathan Brooks

A Cloudflare-OpenAI pilot helped OpenAI discover and index hundreds of millions of new pages for ChatGPT Search, the companies said Oct. 15, through a group equal to roughly 1% of Cloudflare’s network.

What did the Cloudflare-OpenAI pilot find?

Cloudflare and OpenAI described a research pilot that fed signals to OpenAI’s crawler. The signals identified candidate pages, skipped nonexistent pages and flagged updates. Their Oct. 15 post had two OpenAI employees as co-authors.

The companies said discovery could initially increase crawling. They disclosed no citation frequency, visit totals or share of new pages used in ChatGPT Search answers.

OpenAI wrote: “A signal about a page doesn’t grant permission to access it.” Discovery, access and citation remain separate. OpenAI ranks pages independently, so finding one does not secure a citation.

OpenAI’s crawler documentation splits search from training. Sites can allow OAI-SearchBot and disallow GPTBot in robots.txt. The report documents discovery at scale, but reports neither citation impact nor referral traffic.

When should a site return a 503?

On Oct. 21, Google Search Central said sites taking servers offline for PHP patches should return an HTTP 503. The post came the day PHP released emergency fixes for two critical flaws.

Google said the status code “ensures already indexed URLs are preserved in Search while you work.” It advised against general 404 or 200 error pages during the patch.

John Mueller said on Bluesky that the response need not run through PHP. A CDN routing rule or temporary server that DNS points to can serve it while PHP remains unavailable.

Google’s crawl-rate documentation limits 503 responses to one or two days. A URL can leave the index if it returns 503 for multiple days.

For any PHP-based site, the response must come from an entity that stays available. Returning 503 to robots.txt requests can block all crawling, according to Google’s guide to pausing a site.

Does robots.txt control AI training?

Gary Illyes said in an Oct. 21 Search Central video that RFC 9309 directives govern which compliant crawlers may crawl. They do not control how a system uses content after retrieval.

Illyes characterized robots.txt as an access preference protocol. To keep out one AI crawler, a site must target that crawler’s user-agent token.

Google’s common-crawler documentation says Google-Extended controls whether Google may use crawled content to train future Gemini models or ground Gemini Apps and Vertex AI. It does not affect inclusion in Google Search.

Google-Extended is not a separate crawler and has no separate HTTP user-agent string.

A Disallow rule under a crawler token keeps compliant crawlers from listed paths. An IETF working draft co-written by Illyes would add Content-Usage preferences, but remains unapproved.

A standard training preference separate from crawl access therefore exists only in draft form.

How should sites handle raw and encoded brackets?

On Oct. 19, John Mueller said on Bluesky that raw square brackets and their percent-encoded forms, %5B and %5D, create different URLs.

He acknowledged that he previously would have treated them as equivalent and may have supplied the wrong answer.

RFC 3986 reserves brackets and plus signs. If a site serves identical content for both forms, Mueller said Google would crawl each URL, eliminate duplicates and select one version.

Mueller advised sites to pick one format for links, canonical tags, sitemap entries and structured data. Google’s URL documentation also requires reserved characters to be percent encoded.

Existing raw-bracket sites should likely keep working, although Mueller would not build a new site that way. Mixed forms may force Google to choose a version.

Can Google Images and Discover render JPEG XL?

No. Mueller said on Oct. 15 that Google Images and Discover previews did not support JPEG XL.

His post followed wide rollout of JPEG XL support in Chrome 155 on Oct. 6 and Firefox 158 on Oct. 13.

As of Oct. 22, Google’s image SEO documentation listed BMP, GIF, JPEG, PNG, WebP, SVG and AVIF as supported formats for an img element’s src attribute. JPEG XL was absent.

Mueller said the img element inside a picture element would work for any search engine as a fallback.

Image-led sites should put JPEG, WebP or AVIF in that src attribute, even if they list JPEG XL as a picture source.

Discover’s guide points to og:image and schema.org markup for preview images. Those references should also use a listed format.

How will AI agents reach publisher sites?

In an interview with Alex Heath on the Sources podcast, Google CEO Sundar Pichai said agents will reach websites through human-like browsing and protocols built for agents. He expects both paths to develop.

Pichai cited momentum behind agent and payment protocols, including the Universal Commerce Protocol that Google backs. He called protocols a far more scalable approach but expects them to take longer to develop.

He also expects agents to provide a unified experience across Search and Gemini. Google’s hosted agents browse as Google-Agent, one of the user-triggered fetchers that Google says generally ignore robots.txt rules.

Blocking a conventional crawler does not necessarily govern a user-triggered agent fetch. A standard content-usage preference remains unapproved.

Agent traffic could arrive through two channels with different controls, so publishers need separate evidence of crawler access, model-use preferences and user-triggered fetches in server logs.

What should publishers monitor next?

Each October development concerns a signal and its reach. A temporary 503 tells Google an outage is brief. Robots directives restrict compliant crawling, not later use. A discovery signal does not grant access.

Google’s policy statements and Mueller’s observations describe current behavior; Cloudflare’s signal grants no access, and Pichai’s protocol comments remain forecasts for now.

Server logs show which crawlers request a site. Each platform’s documentation identifies the relevant crawler token and control.

Related reading includes Google’s video series on unusual technical SEO questions, an Ask an SEO discussion of robots.txt versus server-level AI crawler blocking, and its URL parameter guide.

Publishers should monitor server logs, robots.txt directives, Google’s image-format list and Cloudflare pilot disclosures as agent protocols develop and the IETF usage proposal advances.