Cloudflare’s AI Bot Rules Hit Mixed‑Use Crawlers, Gray Scrapers Remain
On Sept 15 Cloudflare will block AI‑training and agentic use for mixed‑use bots while still letting them index for search, forcing crawlers like Google to renegotiate access. Yet publishers keep losing the bulk of their articles to a massive gray‑scraping economy that operates outside any gatekeeper controls.