ChatGPT’s user‑triggered fetcher ignores robots.txt, OpenAI admits
OpenAI’s own docs say the ChatGPT‑User fetcher runs when a user asks a question, and that robots.txt directives may not apply. New data shows this bot reaches disallowed pages far more than any other AI crawler, meaning a “Disallow” line offers little protection. Site owners should instead manage visibility via OAI‑SearchBot controls.