Google explains why CDN migrations spike Googlebot traffic at your origin
Google's November 11 post explains why CDN launches spike Googlebot requests at the origin: cold edge caches, faster hostload signals, and the danger of blanket bot rules.
- Google's November 11, 2026 post explains crawler traffic spikes right after CDN migrations
- Cold CDN caches turn first requests for every URL into origin-bound cache misses
- Fast edge responses raise Google's crawl capacity limit (hostload), accelerating crawling
- Blanket WAF/CAPTCHA rules without verified-bot exemptions can push pages out of Search
- Google recommends 429 or 503 responses for at most 1–2 days as a temporary crawl brake
Switching your site to a Content Delivery Network can cause crawler traffic to your origin server (the backend source of truth holding your original content) to spike almost immediately — the opposite of what the migration was supposed to fix. That is the central warning in a November 11 post from Google's Search team, published alongside Episode 5 of its video series "The Non-Canonical Cabinet," which walks site owners through the CDN transition and its traps.
The mechanics, as Google explains them, come down to how crawling infrastructure calculates a site's crawl capacity limit, also known as hostload.
- Slower, unstable responses decrease crawl rate: Google's crawlers continuously monitor host health and slow down when response times rise, to avoid overwhelming the server.
- Fast, stable responses increase it: consistent speed reads as healthy infrastructure, and — provided there is crawl demand — the crawl capacity limit gradually rises.
The moment DNS points to a fast CDN edge network, connection and response times drop dramatically. To Google's systems, it looks as though the server received a massive hardware upgrade, so crawlers ramp up their request rate to match.
Why cold caches make it worse
A higher crawl rate would be harmless if the CDN absorbed every request at the edge. The catch: a brand-new CDN starts with an empty, or cold, cache. The first request for any URL at an edge location is a cache miss — the edge server must fetch the response from the origin, store a copy, and only then serve subsequent requests as cache hits directly from the edge.
Human visitors cluster around a small set of popular pages — the homepage, top articles, active promotions — which warm up quickly. Crawlers do the opposite: they systematically explore the breadth of a site's URL inventory. Combine the two behaviors and every uncached URL a crawler touches triggers a miss that passes straight through to the origin. Google notes that crawlers do not ramp up immediately and many CDNs include built-in connection pacing, but until edge caches warm up, the origin is still doing the heavy lifting.
When blanket bot protection goes too far
What does this change for site owners running firewall rules? The post warns against a predictable failure mode: an admin sees spiking request graphs during cache warm-up, logs into the CDN dashboard, and turns on blanket bot protection. As the video's host Cherry puts it: "BOTS SHALL NOT PASS."
The problem, Google writes, is in the name. Without exemptions, blanket rules treat all automated traffic the same — including crawlers the site depends on. When a CDN or Web Application Firewall (WAF) intercepts requests with a CAPTCHA-style challenge, Google's crawlers fail the test. The post's aside: "Googlebot has many talents; spotting traffic lights and chickens is not one of them. In fact even Gary cannot solve them sometimes."
Left enabled without verified-crawler exemptions, those rules leave crawlers stuck on error pages. From Google's perspective the site appears deliberately restricted or broken; crawlers back off, and pages, shopping listings — and, in the post's phrasing, chicken pictures — begin to drop out of Search. That is a confirmed search-engine statement, not tool speculation.
How to migrate without scorching the origin
Google's recommended playbook, in its own words:
- Allow verified bots when blocking. Most major CDNs and WAFs offer a toggle or managed rule to let known crawlers bypass challenges while filtering unwanted traffic; site owners can verify Google's crawlers against Google's published IP ranges. Better still, target specific unwanted bots via robots.txt or rules rather than blocking all automated clients.
- Pre-warm caches before opening the floodgates. Before routing all traffic through the new CDN, programmatically fetch the highest-traffic pages, upcoming peak-traffic pages such as seasonal launches, and the most frequently crawled URLs at a controlled pace so edges are ready.
- Use 429 or 503 as a temporary brake. If crawl volume still strains the origin during warm-up, configure rate-limiting to return 429 (Too Many Requests) or 503 (Service Unavailable) for a short window — a few hours, or 1–2 days at most — per Google's documented guide to urgently reducing crawl rate.
The recap distills it to three points: faster responses mean faster crawling; caches only shield you on the second visit; and verified crawlers must be exempted from challenge pages, with temporary 429/503 responses as the pressure valve.
The post also points to the full five-episode first season of The Non-Canonical Cabinet, covering broken URLs, robots.txt, 404s, and redirects, and invites viewers to submit edge cases for a second season, budget pending. Site owners planning a CDN move in the coming months should monitor origin response times and log crawler behavior through the warm-up window — the period Google identifies as the migration's real bottleneck.
via youtu.be (Original)
More from Tom Whitfield
Show full bio
News editor covering regulation, policy and their impact on Search rankings and algorithm updates, generative engine optimization at SERP Journal.
173 articles