AI Bot Paywalls Force Sites to Choose Which Crawlers Can Cite Them
Cloudflare and AWS now use HTTP 402 to charge AI crawlers per request, letting publishers set fees and decide which bots get access. The move reframes bot crawling from a free traffic source to a consent‑based trade, forcing sites to pick which AI agents can still cite their content.