Cloudflare is rolling out a new policy that blocks mixed-use web crawlers from accessing publisher content unless publishers explicitly opt in. The move targets AI companies that combine traditional search indexing with model training and agent operations in a single crawl.
How the New Policy Works
Cloudflare will block crawlers that combine search, AI agents, and model training from advertising-supported pages by default. Publishers who want their content available to these crawlers can opt in, but the default is now exclusion.
The company argues that AI companies have been getting a free ride on publisher content for years, using it to train models that generate direct competition to the original sources. By blocking these crawlers by default, Cloudflare forces a negotiation where publishers have leverage.
Beyond Blocking: Charging for AI Value
The blocking policy is just the first step. Cloudflare plans to expand its publisher monetization tools beyond access control to actual content licensing. The goal is to create a system where AI companies pay publishers not just for crawling access, but when AI systems generate value from published content.
This could mean per-query fees, licensing arrangements, or revenue-sharing models between AI companies and the publishers whose content powers their outputs. Cloudflare is positioning itself as the infrastructure layer where these transactions happen.
Separating Search From AI Training
A key part of the policy is pushing AI providers to separate their search crawlers from their AI training and agent crawlers. Currently, many AI companies run a single bot that serves multiple purposes. Cloudflare wants distinct crawlers with distinct access rules, giving publishers granular control over how their content is used.
Search crawlers that index content for traditional web search would remain separate from training crawlers that hoover up entire sites for model training. This separation makes it possible for publishers to allow search indexing while blocking training, or vice versa.
What This Means for Publishers
For publishers, the policy shift is significant. Major news outlets, blogs, and content creators have watched their work get consumed by AI models without compensation. Cloudflare’s move gives them a technical mechanism to enforce boundaries.
Publishers using Cloudflare’s free tier will see AI crawlers blocked by default. Those on paid plans get access to more granular controls, including per-bot rules and analytics showing which AI companies are crawling their content and how frequently.
Industry Reaction
The response from AI companies has been mixed. Some have signaled willingness to negotiate licensing deals, recognizing that unrestricted access to publisher content may not survive the year. Others argue that crawling public web content should remain free, pointing to the precedent set by search engine indexing.
Cloudflare hosts a significant portion of the internet’s traffic, giving its policy real teeth. Sites behind Cloudflare’s network represent billions of pageviews, and AI companies that can’t crawl that content will face gaps in their training data and agent capabilities.
Frequently Asked Questions
What does Cloudflare’s new AI crawler policy do?
Cloudflare blocks AI crawlers that combine search indexing, AI agents, and model training from accessing publisher content by default. Publishers must explicitly opt in to allow these crawlers, and Cloudflare plans to add monetization tools so publishers can charge AI companies for content access.
Does this affect Google Search crawling?
No. Cloudflare’s policy specifically targets mixed-use crawlers that combine search with AI training and agent operations. Traditional search engine crawlers from Google, Bing, and other search engines are not affected by the default blocking.
Can AI companies still crawl websites behind Cloudflare?
Only if the publisher has opted in to allow AI crawlers. Cloudflare also encourages AI companies to separate their search crawlers from training crawlers, so publishers can grant search access while blocking training access, or the reverse.
How will publishers get paid for AI content use?
Cloudflare is building monetization tools beyond simple access blocking. These will include licensing arrangements, per-query fees, and revenue-sharing models where AI companies compensate publishers when their content generates value in AI outputs. The exact pricing models are still being developed.
Does this apply to all Cloudflare users?
The default blocking applies to all sites behind Cloudflare’s network. Free-tier users get the default blocking, while paid plan subscribers get access to more granular controls for managing individual AI crawler access and analytics.
