Cloudflare to block AI agent and training crawlers on ad pages

Cloudflare announced on July 1 that, beginning Sept. 15, it will block AI agent and training crawlers by default on pages that display advertising. The company activated new bot-classification controls for all customers on July 1 and set new defaults that apply to newly onboarded domains, new sites created by existing customers and all free-tier accounts. Operators can change the defaults in security settings before Sept. 15.

Cloudflare divides automated crawlers into three categories. Search crawlers index pages so users can be referred to them later. Agent crawlers fetch pages in real time to answer a user’s query. Training crawlers collect material to incorporate into a model’s weights. On ad-supported pages, Agent and Training traffic will be refused by default while Search traffic remains allowed.

The company enforces the policy at the network edge, not with robots.txt, so blocked requests receive server-level refusals. The classification that separates Search, Agent and Training is based on observed behavior at Cloudflare’s edge rather than only on a declared user-agent string.

Cloudflare CEO Matthew Prince described the policy as intended to “encourage mixed-use crawlers to separate search from agent use and training.” The company also recommends that operators and crawler developers negotiate access where needed rather than relying on changing identifiers.

The policy affects both publishers and system builders. Google’s crawler combines search indexing and training behaviors in a single user agent; as a result, a site that blocks Training by default may also block Google’s crawler and affect search indexing. Agent-based tools that routinely fetch ad-supported pages for research, monitoring or customer support may be denied access on those pages under the new defaults.

Cloudflare said more than half of AI crawler traffic is used to re-fetch pages that have not changed. Some platforms and publishers are testing pay-per-crawl or pay-per-use arrangements to manage access and reduce repeated fetches, while other operators will keep pages open to search crawlers only.

Site operators should verify account tiers and review security settings before the Sept. 15 deadline. Developers of agentic systems should identify which Cloudflare-protected domains their bots contact and begin negotiations for access where necessary. Without changes or agreements, bots encountering the default settings may receive 403 refusals and lose access to ad-supported pages.