Stradiji Audit Finds Most Websites Block LLM Crawlers
Stradiji's AI Visibility Study and Audit, reported by WBOC TV, finds most audited websites block LLM crawlers, raising questions about publisher visibility inside AI-generated answers.
- Stradiji's AI Visibility Study and Audit found most audited websites block LLM crawlers
- WBOC TV surfaced the study this week
- Blocking is typically done via robots.txt disallow rules against AI user agents such as GPTBot, ClaudeBot, and PerplexityBot
- Google-Extended is the user agent that controls AI training access separately from Google Search indexing
- Cloudflare now blocks AI crawlers by default for sites that opt into its service
Stradiji's AI Visibility Study and Audit, surfaced this week through WBOC TV, concludes that the majority of websites the firm scanned are configured to block the crawlers used by major large language models.
The finding places site operators squarely inside a fast-emerging policy question in search and AI: whether to allow the user agents that feed systems such as ChatGPT, Claude, Perplexity, and Google's generative AI products to retrieve page content.
What did Stradiji measure?
Stradiji's audit, framed as an "AI Visibility Study," evaluates whether websites permit retrieval by the user agents associated with commercial LLM products. A site that serves a robots.txt disallowing these crawlers effectively opts out of being cited inside AI-generated answers, even when those answers could otherwise have surfaced the page.
The headline conclusion - that most sites "close the door" - captures a measurable skew toward exclusion. Stradiji has not, in the materials referenced by WBOC, published a per-crawler breakdown; the public framing is aggregate. SEO teams reading the finding should treat "most" as directional until the firm releases methodology, sample size, and the crawler list used in the scan.
What does blocking an LLM crawler actually do?
A robots.txt directive is advisory, not enforced. Compliant operators honor it; others ignore it. For sites that want to opt out, the practical lever is one or more lines in /robots.txt, for example:
User-agent: GPTBotpaired withDisallow: /User-agent: ClaudeBotpaired withDisallow: /User-agent: anthropic-aipaired withDisallow: /User-agent: PerplexityBotpaired withDisallow: /User-agent: Google-Extendedpaired withDisallow: /
Google-Extended is the toggle governing whether a site's content can train Google's generative AI products while still being indexed for Search. Disallowing it does not affect Google Search ranking. That nuance is often missed in coverage that conflates AI-training opt-out with general SEO exclusion.
Which verticals are most likely to opt out?
News and publishing brands have led the opt-out wave, driven by licensing disputes and unresolved questions about compensation for content that feeds model training. E-commerce sites have moved more selectively, partly because product-feed visibility inside shopping-oriented AI surfaces is becoming a measurable revenue channel. B2B SaaS and affiliate-heavy niches tend to fall in between, depending on whether operators view AI citations as net traffic gain or loss.
What to monitor next?
Watch for Stradiji to publish its crawler list and sample size, which will determine whether the "most" finding holds across vertical, traffic tier, and region. Also worth tracking: any move by Cloudflare, which now blocks AI crawlers by default for sites that opt into its service, to publish aggregate opt-in rates. If that number climbs past 30 percent of top-ranked domains, the AI citation pool would narrow meaningfully, and publishers that keep their doors open would pick up a compounding presence inside LLM answers.
via Google News: generative engine optimization (Source)