Google, Microsoft, OpenAI and Anthropic back one AI training opt-out

Google, Microsoft, OpenAI and Anthropic will honour one shared AI training opt-out in robots.txt. It lets a website keep its content out of generative AI training with a single rule. Today, a site that wants to say no has to write separate rules for each company’s crawler.

The shared signal comes from an IETF draft by Kevin Kelley of Anthropic, published on Friday. The same day, Google , OpenAI and Microsoft announced they would support it. Kelley announced the draft on the IETF’s AI preferences mailing list .

It works through robots.txt, the file sites already use to tell crawlers what they may fetch. A site adds a group for a new “policy token”, __usage_ai-training . It then lists the paths it does not want used for training. The draft’s example blocks a whole site except its press kit.

“Publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work,” Krishna Madhavan, principal product manager for Microsoft Bing, wrote.

Anthropic has already started work on it, Kelley wrote. He said it would complement Anthropic’s existing crawler controls rather than replace them.

“It’ll take us some time to add support across our services and work through the inevitable complications and edge cases that I’m sure we’ll discover,” Kelley wrote.

OpenAI said it aims to implement the signal “in the coming months”. Its existing crawler controls, such as GPTBot, will still apply, it said.

Google said it aims to support the signal in 2027, and its crawler documentation says early 2027. Google-Extended is Google’s own AI training control. If a site opts out with the token and does not mention Google-Extended, Google will treat it as a block on Google-Extended. Sites that allow Google-Extended can appear in Gemini app answers and get clicks, Google Search Advocate John Mueller wrote.

Microsoft will honour the new preferences from 1 February 2027. They will replace the NoArchive and NoCache tags it has used since 2023. Microsoft is also adding a separate page-level setting for grounding, when an AI answer draws on a live web page. One option allows Microsoft’s own products, such as Copilot. It blocks the grounding services Microsoft sells to other companies.

The draft says the token states a preference about how sites want their content used. It is not an order about crawling. A site with no such group has expressed no preference, and existing rules for all crawlers do not count. Private agreements between a site and an AI company can override it.

The draft is an individual submission. The IETF’s datatracker says the IETF has not endorsed it and it has no formal standing in the standards process.

Microsoft said the new preferences “do not establish licensing or payment arrangements”. Blocking AI grounding “will significantly reduce your content’s visibility in AI-generated experiences”, it added. It does not change a page’s place in Bing search results.

Meta communications head Andy Stone voiced support for a common signal on X, The Verge reported . He did not say whether Meta would honour it.

Google said the proposal is consistent with the EU’s Code of Practice for general-purpose AI under the AI Act. The code recognises robots.txt as a main way for publishers to state machine-readable preferences.

Among the people the draft thanks for shaping the discussion are staff from the Financial Times, the BBC, Bertelsmann and the European Publishers Council.

Publishers have also pushed for payment. In July, Cloudflare told AI crawlers to pay or get blocked . Its chief executive has warned that bots could reach 1,000 times human web traffic . In Australia, the ABC and SBS want AI companies to pay for news .