The internet is entering a new phase as AI technology changes how people find information online, with rapid advances in AI tools reshaping search and content discovery. For years, websites allowed search engines to crawl their pages in return for referral traffic. But AI search is changing that. Many AI tools now answer questions directly, so users often never visit the original website. Many publishers say that balance no longer works in their favor.
Cloudflare wants to change that with its new Cloudflare AI policy. The update gives website owners more control over how AI companies access and use their content. It is available to all Cloudflare customers, including users on the free plan.
Instead of asking publishers to either allow or block every AI bot, Cloudflare now offers more detailed controls. Publishers have been asking for clearer rules on how AI uses their content. This update is the company’s response to those concerns.
Cloudflare has grouped AI traffic into three categories:
- Search crawlers help websites appear in search results.
- Agent crawlers complete tasks for users through AI assistants.
- Training crawlers collect website content to improve AI models and automation tools.
Here’s a simple example. A news website wants search engines to index its stories because search helps readers find its content. More visitors also mean more advertising revenue. At the same time, the publisher may not want those same stories used to train AI models without permission. The new Cloudflare AI policy lets website owners control those permissions separately.
Previously, many website owners had only two choices. They could allow AI bots or block them completely. The updated AI crawler policy offers a more flexible approach. Website owners can now decide whether a crawler can index content, complete AI-powered tasks, or collect information for AI training.
Cloudflare also wants AI companies to use separate crawlers for different jobs. Instead of combining Search, Agent, and Training into one bot, the company says each activity should have its own crawler. That makes it easier for website owners to understand why a bot is visiting their website. It also strengthens Cloudflare’s blocking of AI crawlers that don’t match a site’s preferred settings.
The new default settings will take effect on September 15, 2026. New websites joining Cloudflare will automatically block training and agent crawlers on ad-supported pages. Search crawlers will remain allowed by default. Existing customers can keep these settings or change them at any time.
Cloudflare has also launched BotBase for Enterprise customers. The dashboard lists verified bots and explains what they do. It also shows how they use website content. Publishers can choose whether bots can temporarily access content, create searchable references with attribution, or reproduce larger portions of it. These tools make AI traffic easier to track and manage. This update is about more than blocking bots. It gives website owners more choice as AI becomes a bigger part of the web.
The policy could affect several groups:
- Publishers can protect original reporting while still appearing in search results.
- E-commerce businesses can allow AI shopping assistants but block large-scale product scraping.
- Bloggers and creators gain more control over how their work is reused.
- Educational websites can stay visible in search while protecting learning materials.
- AI companies may need to separate search, agent, and training crawlers.
- Website owners can set different rules for different types of AI traffic.
The update also comes as publishers and AI companies continue debating content licensing, attribution, and fair payment for original work. Because Cloudflare supports millions of websites, its decision could influence how other web infrastructure providers handle AI traffic.
Whether the rest of the industry follows remains to be seen. However, the Cloudflare AI Policy signals a clear direction. AI companies may need to be more transparent about how they collect and use online content, a trend that reflects broader industry insights into the future of AI governance and digital publishing.
Author’s Note:
Cloudflare’s latest policy signals a broader shift in how AI and the open web will coexist. As AI becomes a bigger gateway to online information, giving publishers greater control over their content could soon become the new industry standard.
Leave a comment