AI paywalls force site choices, Googlebot ignores robots.txt

Cloudflare and AWS now use HTTP 402 to charge AI crawlers per request, letting publishers set fees and decide which bots get access. The move reframes bot crawling from a free traffic source to a consent‑based trade, forcing sites to pick which AI agents can still cite their content.

Alphabet’s Q2 2026 release shows consolidated revenue at $119.8 billion, a 24% year‑over‑year rise, with Google Search & Other up 17% to $63.27 billion. The filing details exact segment numbers, yet Google’s statement on web‑traffic clicks stays unverified, leaving advertisers without a clear measure of ad effectiveness.

AgencyAnalytics' AI Tracker lets agencies quantify brand visibility, position, sentiment, and source citations in AI‑generated answers. By treating these four metrics as a single chart, firms can finally answer the top 2026 client request, whether their brand shows up in ChatGPT and what impact it has.

The new Ahrefs tool creates a custom llms.txt file in seconds, using data from 137,000 domains. It helps sites tell AI crawlers and GEO services how to index content, potentially improving search visibility without any cost.

John Mueller revealed that a mis‑configured robots.txt, specifically a user‑agent: * block overridden by a more specific Googlebot section, caused Google to index spammy search‑box URLs on a Shopify store. The oversight shows that relying on generic directives can let unwanted pages slip through, hurting SEO and waste crawl budget.

Ahrefs’ analysis shows AI search is moving from vague hype to hard metrics: searches for AI tracking tools jumped 184%, Google’s new AI performance report still omits clicks, and brands now chase AI citations, but a 1,900% citation spike delivered almost no conversions, prompting a fresh focus on citation ROI.

Buffer’s analysis of 1.8 million YouTube uploads shows long‑form videos get the strongest engagement when posted Sunday at 10 a.m., while Shorts peak on Friday evenings (4 p.m., 6 p.m., 7 p.m.). Timing lifts early signals, helping the algorithm surface content to a wider audience.

Google’s AI & Economy ATLAS v1.0 analyzes 15 million de‑identified Gemini app, AI Mode, and API interactions. It reveals that roughly 86 % of AI usage occurs in everyday tasks, shopping, planning, and government queries, while workplace adoption touches 88 % of occupations but remains shallow and collaborative.