One robots.txt line is meant to cover AI training at several companies. The clocks differ.
A draft by Kevin Kelley of Anthropic defines a policy token, __usage_ai-training. OpenAI says it will honor a shared signal in the coming months. Google aims at 2027. Microsoft says it will start on 1 February 2027. The Straits Times says Anthropic has already started. The draft says the token is a training preference. A block on crawling is a separate group in the same file.
Feature4 minEasy
James Okonkwo4 min read
- OpenAI
- Microsoft
- Anthropic
- Publishers

Photo: OpenAISource page
OpenAI, Google, Microsoft and Anthropic said on Friday that they will honor a single robots.txt signal for whether a site’s pages may be used to train generative AI models. The text that defines the signal is an individual Internet-Draft by Kevin Kelley of Anthropic, dated 30 October and set to expire on 3 May 2027. The datatracker page says the draft is not endorsed by the IETF and has no formal standing in the standards process. Emma Roth’s account for The Verge, published at 8:26 p.m. UTC on Friday, calls Kelley a technical lead for standards at Anthropic. The Straits Times wrote the commitments up on Saturday morning.
What a publisher actually writes
The draft adds a policy token, __usage_ai-training, used as a User-Agent line in the robots.txt file a site already publishes. Its example is a group that disallows the whole site and allows /press-kit/. That is a preference against AI training for every path except those under /press-kit/, where the preference runs the other way. The token has to match exactly, aside from case. A group written for User-Agent: * does not apply to it. If the file has no __usage_ai-training group, the draft says no preference has been expressed. Disallow on a path is a preference against the use. Allow on a path is a preference in favor of it. The draft’s definition of AI training, quoted from the IETF AI Preferences working group’s vocabulary, is using an asset to modify the learned parameters of a generative AI model. Training of models used only for non-generative tasks is outside that category, even if the model could generate.
The draft says the token is not a directive for crawling. Rules for a crawler’s own product token still mean what they meant before. A company may choose to skip a fetch if the only reason it would have fetched the page was a purpose the site has opted out of. A publisher that wants to limit crawling uses the ordinary crawler groups in the same file. Private agreements may override the preference. Microsoft’s blog, by Krishna Madhavan, a principal product manager for Bing, uses the same token and says both training and grounding default to Allow when a site publishes nothing. Its whole-site opt-out is a __usage_ai-training group with Disallow: /.
Four companies, four dates
OpenAI’s post, dated 30 October, says it will honor a common robots.txt signal for foundational generative AI training, aims to implement it in the coming months, and will post implementation details when it does. The signal does not replace the crawler controls OpenAI already documents. Google’s post, by John Mueller and also dated Friday, says the company aims to support the control in 2027. If a site opts out with the policy token and does not explicitly set Google-Extended, Google plans to treat that as a disallow of Google-Extended, the training control it has offered since 2023. Company-specific rules keep working beside the shared one. Google says sites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini apps.
Microsoft’s dates are the most specific. It will start honoring the new preferences as it crawls on 1 February 2027. Sites that use the older NoArchive or NoCache tags, and want those limits to survive, are told to publish the new preferences before 1 March 2027. After that date those older tags will no longer be read as a preference for training or grounding. A temporary grounding switch in Bing Webmaster Tools opens on 15 December 2026, only for sites using the old tags as of that day, and lasts until a recrawl or 30 September 2027, whichever comes first. There is no temporary switch for training. Microsoft also splits off a second, page-level preference, AI grounding, which is not in robots.txt. DisallowExtended limits grounding to Microsoft-operated products such as AI Mode, AI Overview, Copilot and Microsoft 365, and leaves out grounding Microsoft sells to others, including Web IQ and a grounding API in Foundry. Madhavan writes that choosing Disallow or DisallowExtended for grounding will significantly reduce a site’s visibility in AI-generated experiences, and that Bing search eligibility is unchanged. Training and grounding are independent: a site can disallow training and still allow grounding.
Anthropic’s own timing is in the secondary accounts. The Straits Times says Anthropic said it has already started rolling the signal out. The Verge says Anthropic has already adopted it, that Google and Microsoft plan to support it next year, and that OpenAI’s phrase is “in the coming months.” The Straits Times, citing Microsoft’s blog, says the setting applies to future training, does not by itself stop other crawling, and that scraping stays allowed until a publisher changes the setting. It also says Microsoft and Google have each warned that opting out of training can reduce the chance a publisher’s work shows up in AI apps.
We believe creating a common signal that conveys such preferences to multiple providers at once is an important step. And we look forward to working with other AI companies on the standards and their implementation across the industry.
— Andy Stone, Meta, as carried by The Verge and The Straits Times
The Straits Times says Stone wrote that on X. The story carries an AFP credit. The Verge says the comment supports the setting and does not say Meta will honor it. The same Straits Times story notes the New York Times’ December 2023 lawsuit against OpenAI and Microsoft over the use of its articles, and the companies’ fair-use defense, alongside Cloudflare’s default block on AI bots for new customers since July 2025 and the Really Simple Licensing standard for charging crawlers. Those are the surroundings. The control in the draft is the token. Anthropic, on the newspaper’s account, has already started to roll it out. OpenAI, Google and Microsoft have named later dates.


