Googlebot ignores generic robots.txt rules when a specific Googlebot block exists, letting spam index
John Mueller revealed that a mis‑configured robots.txt—specifically a user‑agent: * block overridden by a more specific Googlebot section—caused Google to index spammy search‑box URLs on a Shopify store. The oversight shows that relying on generic directives can let unwanted pages slip through, hurting SEO and waste crawl budget.