Cloudflare, the network infrastructure company, is set to implement a default block on mixed AI-powered web crawlers beginning September 15. This policy targets crawlers utilizing bots for a variety of functions, including searching and training models.
The company emphasizes the need for distinct solutions to differentiate between search training and agent-based crawling. This decision stems from concerns voiced by publishers and web owners, who want to maintain their visibility in search and AI services without offering their content at no cost.
“The internet risks transitioning to a ‘zero-click’ ecosystem, where chatbots gather original content for instant responses, completely bypassing the original sources,” Cloudflare noted. They highlighted the shift in focus from SEO to AI Optimization (AEO) and now to Generative AI Optimization (GEO).
This change is also anticipated to alleviate stress on web infrastructure, as over half of the traffic generated by AI crawlers consists of unnecessary re-scanning of pages. The new policy will apply to both new and existing Cloudflare clients, including those on the free plan, although users will have the option to disable the blocking feature manually.
