The DropJuly 6, 2026via The Decoder
Cloudflare replaces its blanket AI bot block with granular controls for search, training, and agent crawlers
Why it matters
Cloudflare's granular AI bot controls shift power back to publishers and creators, forcing AI companies to negotiate access rather than scrape by default. This marks a strategic pivot in the infrastructure layer that could reshape data licensing economics across the AI supply chain.
Key signals
- Granular controls for Search, Training, and Agent bots now available to all Cloudflare customers
- Training and Agent bots blocked by default on ad-supported pages starting September 15, 2026
- Replaces previous blanket block-all approach with category-based management
- Affects data ingestion pipelines for LLM training and agent deployment workflows
The hook
Cloudflare just handed every site owner a remote control for AI crawlers. Search bots in, training bots out—by default.
Cloudflare is giving all customers granular AI bot controls. Site owners can now manage Search, Training, and Agent bots separately instead of blocking them all at once. Starting September 15, 2026, Training and Agent bots will be blocked by default on ad-supported pages.