Forty-two percent of mid-market domains block AI crawlers, according to a CNV Creative analysis cited by The Des Moines Register. The figure offers one of the first quantified measurements of AI-blocking adoption outside enterprise newsrooms.

CNV Creative, a digital agency, surfaced the number through a robots.txt scan of mid-market publisher and brand sites. The Des Moines Register carried the report without direct quotation from CNV's analysts and did not include sample-size detail in its headline summary.

What is CNV Creative measuring?

The 42% captures mid-market domains that disallow at least one AI crawler user agent in robots.txt files. Mid-market typically describes sites positioned between small personal blogs and the largest enterprise publishers, often serving niche verticals with editorial teams of five to fifty staff members.

CNV did not publish the user agents it tracked in the headline summary. AI crawlers commonly include the following user agents:

• GPT bot
• Claude crawler
• Common Crawl bot
• Vendor fetchers issued by Google, Meta, Microsoft, and Apple

Knowing which bots CNV counted would sharpen the 42% benchmark considerably.

Why does this number matter for publishers?

Publisher resistance to AI training scrapers has hardened since 2023. Several major newsroom groups signed paid content deals with AI vendors during 2023 and 2024, locking in licensing revenue and excluding their content from training datasets. Mid-market sites lack that negotiation leverage, which leaves robots.txt blocking as the cheapest available counter-measure against AI training use.

The technical path is simple. Each AI vendor publishes a distinct user agent string. Publishers add disallow rules to robots.txt for those specific bots. Google Search and Google AI crawlers run as separate processes with separate user agents, so blocking AI bots does not impact Google Search indexing, crawling, or rankings.

How should SEO teams read this?

The 42% reflects publisher intent rather than a Google algorithm change. Sites that block AI crawlers keep their Google Search rankings fully intact. The CNV data shows where mid-market publishers have drawn lines on AI training data, nothing more.

SEO teams should audit their own robots.txt files for current AI bot rules. Unintended AI bot blocks rarely hurt SEO directly today, but they can cut off future AI Overview traffic if Google chooses to surface a site's content through generative search interfaces. The choice to block should follow a deliberate policy, not a default copied from a tutorial.

What remains unknown in the CNV data?

The headline summary did not include sample size, geographic split, vertical breakdown, or the full crawler list CNV scanned. The Des Moines Register's coverage did not add methodology detail. SERP Journal has requested the full report from CNV Creative.

Until that release lands, the 42% figure stands as a directional benchmark rather than a comprehensive audit of the mid-market segment. Publishers comparing their own robots.txt configurations to the CNV figure should treat it as a high-level indicator only.

What should publishers monitor next?

Watch for the full CNV Creative report, which should clarify sample size, crawler list, and methodology. Track Google's AI Overviews expansion for shifts in publisher incentive, since AI referral traffic could eventually outweigh training-data concerns. Monitor AI vendors for new licensing programs that target mid-market publishers specifically.

The 42% figure marks a starting measurement, not an endpoint. As AI licensing matures and robots.txt enforcement tightens across the open web, the number will move in one direction or another.