Cloudflare to Block Google AI Crawlers for Free Users as Bot Traffic Tops 50%

3 articles · Updated · The Verge · Sep 26

September 15 is the target date for Cloudflare to make Google AI training crawlers blocked by default for free customers, while letting site owners opt out.
More than half of internet traffic is now bot-driven, Matthew Prince said, arguing the ad-funded web cannot support AI agents that consume bandwidth and content without clicking ads or sending traffic back.
Cloudflare says it wants websites to choose whether to block bots, allow them, or charge for access, and is building micropayment-style tools with partners including Stripe and Coinbase.
Google has signaled more transparency about what its crawler is doing, Prince said, but Cloudflare warned it could block Google more broadly if AI and search crawling are not clearly separated.
Cloudflare, which says it sits in front of more than 20% of the web, is pushing the move as part of a broader shift toward paid AI access and away from the web’s traditional advertising model.