AI Agent Traffic Surges 1,700% as Google Tightens SERP Scraping Rules
Cloudflare reports AI agent traffic surged 1,700% year-over-year, with non-human requests now exceeding half of all internet traffic. Google is tightening SERP scraping defenses, and rank trackers say they are getting caught in the crossfire.
- Cloudflare reports AI agent traffic grew 1,700% year-over-year on its network
- Cloudflare now handles an average of 115 million HTTP requests per second, up from 63 million at end of 2024, with peaks above 150 million
- For the first time, more than half of all internet traffic on Cloudflare's network is non-human
- Ryan Jones, developer of SERPrecon, said rank tracking is becoming unsustainable due to aggressive AI scraping of Google
- Sundar Pichai told analysts Google reduced the cost per AI Mode response to its lowest level since launch
AI agent traffic across Cloudflare's network grew 1,700% over the past year, pushing HTTP requests from an average of 63 million per second at the end of 2024 to roughly 115 million today, with peaks above 150 million. For the first time, more than half of all internet traffic is no longer human.
That surge is now reshaping how Google handles automated access to its search results — and rank trackers say they are getting caught in the crossfire.
Why is Google tightening anti-scraping defenses?
Ryan Jones, developer of the SERPrecon rank tracking service, posted on X that the situation has reached a breaking point. "We're getting to the point where nobody will have rank tracking anymore because of the crazy amount of AI scraping going on," Jones wrote.
When one user jokingly suggested routing scrapes through AI, Jones replied: "That's why the rank trackers are failing. Google is blocking all the AI trackers AND AI itself... and the rank trackers are getting caught up."
The volume explains the pressure. Cloudflare, which fronts a meaningful share of web traffic, published the figures this week. The 1,700% year-over-year rise in AI agent requests is not pure bot traffic: a portion is hybrid visits in which agents browse on behalf of humans, a category that still produces value for publishers.
How expensive is AI Mode to run?
Cost is the second pressure point. Traditional Search returns pre-indexed pages; AI Mode runs LLM inference across multiple sub-queries and reasoning steps before assembling a synthesized answer. Each query consumes more compute than a classic blue-link result.
Sundar Pichai addressed the economics on Google's Q2 earnings call, pointing to two new releases: "Yesterday we announced new models — Gemini 3.6 Flash and Gemini 3.5 Flash-Lite — which are cost effective and highly efficient. We're seeing tons of demand for our workhorse Gemini Flash series, because it hits the sweet spot of performance and cost."
Pichai added that engineering and hardware work had reduced the cost per AI Mode response to its lowest level since launch, "even as we've brought more advanced AI capabilities." That framing — aggressive cost compression as usage scales — sets the context for why Google treats every external scraper as load it cannot afford.
Could SERP data feed distillation attacks?
A less-discussed motive has emerged in Google's recent research papers. The publications, which cover systems such as the SAFE AI spam detector and the Scalable Cluster Termination System (S-CTS), describe adversaries that mutate content in response to Google's defenses in order to recover rankings.
The papers do not name distillation, the technique of training a smaller model to mimic a larger one by harvesting its outputs. But the behaviors described match a pattern in which attackers systematically probe Google's SERPs to rebuild a model that reproduces Google's ranking logic. Rank tracker data would be a ready-made input for that pipeline.
Google has framed its anti-scraping posture publicly as a war on rank trackers. The underlying fight may be over whether competitors can keep reconstructing Google's search algorithm one query at a time.
What should SEOs monitor next?
Three signals are worth watching:
- Cloudflare's request-per-second and human-share ratios, which Google can cite to justify further throttling.
- Cost-per-query disclosures in upcoming Google earnings calls, particularly any move beyond Gemini Flash efficiency into other model tiers.
- Whether Google's published research on adversarial content begins to reference distillation-style attacks directly, which would confirm the defensive theory.
Jones's prediction — that traditional rank tracking may become unsustainable — is one operational outcome. The structural question is whether Google's tightening targets SEO tooling, the AI scrapers themselves, or the reverse-engineering pipeline that turns SERP data into competing models.
via blog.cloudflare.com (Original)
More from Tom Whitfield
Show full bio
News editor covering marketplaces and e-commerce at SERP Journal.
65 articles