On October 30, 2026 OpenAI said it will honor a common, voluntary signal placed in robots.txt files that indicates a website operator’s preference about using site content for foundational generative AI training. The company stated this shared marker is intended to let publishers communicate the same choice to multiple AI providers at once.
Why the change matters
OpenAI noted that, until now, website operators often had to identify individual crawlers and maintain separate rules for each provider. The shared signal aims to reduce that administrative burden by enabling a single robots.txt entry to be recognized across participating providers, according to the announcement.
Scope and next steps
The company said it plans to implement support for the signal “in the coming months” and will publish a follow-up update with implementation details. OpenAI emphasized the new signal is voluntary and will not replace the existing crawler controls it already honors; those controls will continue to apply and be respected.
OpenAI described the effort as drawing on work by the Internet Engineering Task Force’s (IETF) AI Preferences working group and indicated the move reflects continued participation in IETF standards discussions. The company added it will adapt its practices as those international conversations and the broader landscape evolve, citing considerations of appropriate controls and fairness.
The announcement also included an encouragement for other AI providers to honor the same signal. OpenAI committed to sharing implementation details when available and to maintaining support for current robots.txt controls during the transition to the shared approach.
Concrete timeline details beyond the “coming months” were not provided in the announcement; OpenAI said it will provide another update with implementation specifics when they are ready.
Original source: OpenAI News