A shared robots.txt signal could simplify AI training opt-outs
Website operators will soon be able to publish one shared robots.txt signal for participating AI providers, rather than maintaining separate crawler rules for each provider. OpenAI says it plans to implement the voluntary signal in the coming months, while its existing crawler controls remain in force. The move is based on ongoing IETF work and gives publishers a simpler way to state their training preferences across services.