Cloudflare replaces its blanket AI bot block with granular controls for search, training, and agent crawlers
Cloudflare just handed every site owner a remote control for AI crawlers. Search bots in, training bots out—by default.
Why it matters
Cloudflare's granular AI bot controls shift power back to publishers and creators, forcing AI companies to negotiate access rather than scrape by default. This marks a strategic pivot in the infrastructure layer that could reshape data licensing economics across the AI supply chain.
The key facts
4 to knowGranular controls for Search, Training, and Agent bots now available to all Cloudflare customers
Training and Agent bots blocked by default on ad-supported pages starting September 15, 2026
Replaces previous blanket block-all approach with category-based management
Affects data ingestion pipelines for LLM training and agent deployment workflows
Go to the source
The Decoderthe-decoder.com
Publisher excerpt: Cloudflare is giving all customers granular AI bot controls. Site owners can now manage Search, Training, and Agent bots separately instead of blocking them all at once. Starting September 15, 2026, Training and Agent bots will be blocked by default on ad-supported pages.