Cloudflare gives website owners new AI traffic controls
Cloudflare launches finer AI traffic controls for all customers, letting site owners distinguish between search bots, AI agents, and training crawlers with granular per-page rules.
BY FOUNDERBUILT AI NEWS
Cloudflare announced new AI traffic management options for all customers as part of its second Content Independence Day. Site owners can now distinguish between Search bots, AI Agent bots, and Training crawlers, replacing the previous one-size-fits-all block approach with granular controls.
The update lets founders protect ad-monetized pages from AI scrapers while still allowing search engine crawlers. This is a major shift from the blanket AI bot blocking that many sites adopted in 2024-2025, giving publishers more flexibility to control how AI companies access their content on a per-page basis.
Cloudflare's AI traffic audit insights page will show which AI services are crawling your site, how often, and what content they access. For founders running content businesses, this provides the data needed to decide which AI crawlers to allow, rate limit, or block entirely without harming SEO performance.