Starting September 15, network infrastructure provider Cloudflare will begin blocking all mixed AI-based web crawlers by default on its pages.
Cloudflare’s new Attribution Business Insights dashboard helps website owners understand crawler behavior, appetite, and potential value. https://t.co/1F9HO3Fiv7
— Cloudflare (@Cloudflare) July 1, 2026
This refers to crawlers that use the same type of bots for searching, training models, and AI agents. Cloudflare insists on separating solutions for search training and agent-based crawling.
Company representatives explained that publishers and website owners want to maintain visibility in search and AI services but are unwilling to provide content for free.
“The internet risks transitioning to a ‘zero-click’ ecosystem, where chatbots gather original content for instant responses, completely bypassing the original sources. We have already seen a significant shift from an SEO-focused space to an AEO world, and now to a GEO one,” the statement said.
This measure is also expected to reduce the load on web page infrastructure. According to Cloudflare, over 50% of traffic from AI crawlers comes from re-scanning pages, which rarely provides any benefit.
The change will affect new customers of the provider, existing users' sites, and all clients on the free plan. Hosting owners will have the option to manually disable the blocking feature.
At the same time, Cloudflare is expanding its commercial scheme for publishers. The new Pay Per Use system entails payment not only for access to material but also when the content is valuable for AI services.
In the initial phase, the company is collaborating with Ceramic.ai and You.com. Publishers receive payments if their materials appear in the search results of chatbots.
It is worth noting that on November 18, 2025, Cloudflare experienced significant technical issues that disrupted the operation of numerous websites and applications, including Web3 services.