🔍 Read the full analysis: A Guide To What The Shared AI Opt-Out Includes And Excludes on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
OpenAI, Google, Microsoft and Anthropic have announced support for a shared robots.txt signal that lets website operators request that their content not be used to train generative AI models. The signal is voluntary, does not block crawling or AI answers based on live web content, and will be adopted on different timelines; Meta expressed support without committing to honor it.
OpenAI, Google, Microsoft and Anthropic said on October 30 they will support a shared robots.txt signal that lets website publishers state that their content should not be used to train generative AI models. The change could replace separate training opt-out instructions for participating companies, but it is a voluntary request, not a technical block, and does not cover every use of web content in AI products.
The proposed rule uses the user-agent label __usage_ai-training alongside Disallow: / in a site’s robots.txt file. It is set out in an individual Internet-Draft titled “Machine-Readable Training Preference Signal,” written by Anthropic standards lead Kevin Kelley and published on October 30. The draft is connected to the IETF’s AI Preferences working group, but it is not an adopted internet standard.
The companies have described different schedules. Anthropic says it has started implementation, while cautioning that support across its services will take time. OpenAI aims to implement the signal “in the coming months.” Microsoft says it will honor it beginning February 1, 2027, as it crawls websites. Google aims to support it in 2027; its crawler documentation specifies early 2027.
Microsoft is also introducing a separate page-level preference for AI Grounding, or use of web content to inform answers in Bing, Copilot and Microsoft 365. Microsoft says both preferences start on February 1, 2027. Its existing NoArchive and NoCache page tags will stop counting as preferences for training or grounding after March 1, 2027. The companies say their current crawler-specific controls remain relevant during the changeover.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
For publishers, one common instruction could reduce the work of expressing a training preference to several AI companies. Today, site operators may need separate rules for crawlers such as GPTBot, Google-Extended and ClaudeBot, as well as other company-specific controls. The shared rule offers a common format, but its practical reach depends on each company adopting and following it.
The distinction between a request and enforcement matters. A robots.txt rule communicates a site operator’s preference; it does not prevent a crawler from accessing a page. Nor does it remove material already incorporated into a trained model. Publishers should not treat the signal as a guarantee that content will be excluded, nor as a mechanism for securing compensation. Microsoft says its choices do not guarantee inclusion, traffic or payment.
The scope also matters for readers and businesses that depend on web visibility. The shared rule concerns foundational model training, not every way an AI service may use a page. Microsoft’s separate grounding control illustrates that training and use of current web material in AI answers are distinct choices. Search eligibility is separate as well: Microsoft says its preferences do not change eligibility for Bing search results, and Google says Google-Extended does not affect Google Search.
Products Worth Considering
Apple Watch SE 3 GPS 44mm Smartwatch
Samsung 27-Inch Odyssey QD-OLED G8 4K 240Hz Gaming Monitor
Samsung Galaxy Tab S10 FE 128GB Wi-Fi Android Tablet, Blue
As an affiliate, we earn on qualifying purchases.
How the Opt-Out Is Structured
The proposal is an individual IETF Internet-Draft, not a finalized standard. At an AI Preferences working-group meeting on October 21, one of the group’s chairs said, “We haven’t declared consensus on anything yet.” The draft’s publication and company announcements therefore mark a move toward a shared convention, not completion of an IETF standards process.
Google says that if a site uses the shared training signal but has no separate rule for Google-Extended, it plans to interpret the shared signal as if the publisher had disallowed that crawler. OpenAI says its existing crawler controls will continue to apply. Anthropic’s crawler help page, as of October 31, still described its existing per-crawler controls. Publishers may need to keep those company-specific rules in place until the relevant services confirm support is active.
Meta spokesperson Andy Stone said the company supports publishers’ ability to express preferences and called a common signal an important step. That statement did not commit Meta to honoring the rule. The source report also said it had found no statement on adoption from Apple, Amazon, xAI or Perplexity. The proposal’s acknowledgments list people from technology companies, publishers and advocacy groups, but participation in drafting or acknowledgment is not the same as a commitment to implement the signal.
“has already started implementing support for this draft”
— Anthropic, in its announcement of the draft
Adoption and Scope Still Vary
Several practical questions remain open. The draft does not have formal standing in the IETF standards process, and the working group has not declared consensus. Company announcements also differ in certainty and timing: Anthropic reports implementation has started, OpenAI gives a broad target, and Google and Microsoft point to 2027 dates. The material does not establish when every service within each company will apply the signal.
It is also unclear whether Meta or other major AI services will adopt the rule. Support for publishers’ ability to express preferences is not the same as a commitment to honor this particular signal. The proposal itself does not establish a way to verify compliance or a consequence for ignoring a site’s request. It also does not settle how companies will treat content used for live answers, search features or other purposes beyond training.
Finally, the signal cannot undo use of content in models that have already been trained. The source material describes no process for removing previously used material, and makes clear that a robots.txt instruction does not stop crawling. Publishers should distinguish the requested training opt-out from any separate controls they use for search visibility, AI answer generation or account-level settings for their own AI chats.
Publishers Face a Staged Changeover
Website operators considering the shared rule can add the proposed __usage_ai-training group and Disallow: / to robots.txt, while checking each company’s instructions and keeping existing crawler-specific rules until adoption is confirmed. The signal is for a website’s content; it is not a setting for opting out of training on a person’s conversations with an AI assistant.
Microsoft’s announced dates call for its new training and grounding preferences to take effect as it crawls from February 1, 2027, with NoArchive and NoCache ceasing to count for those preferences after March 1. Google has specified early 2027, while OpenAI has said “in the coming months.” Those milestones should be treated as company plans, not evidence that every service has already switched.
The next developments to watch are whether the IETF working group reaches consensus, whether the draft changes, and which additional companies make explicit implementation commitments. Until then, the shared rule is a developing convention: a clearer way to state a preference, but not a guarantee of exclusion from AI systems.
Key Questions
What does the shared AI opt-out ask companies to do?
It asks participating companies not to use a website’s content to train generative AI models. The proposed robots.txt group uses User-Agent: __usage_ai-training and Disallow: /. It expresses a preference rather than technically blocking access.
Does the signal stop AI companies from crawling a website?
No. The source material describes the rule as a training preference, not a crawl blocker. It also does not remove content from models already trained or guarantee that a company will comply.
Does the opt-out cover AI-generated answers?
Not by itself. The shared signal is described as covering model training, not answers generated using live web content. Microsoft announced a separate AI Grounding preference for content used in answers across Bing, Copilot and Microsoft 365.
When will the companies honor the signal?
Anthropic says implementation has started; OpenAI aims to implement it in the coming months. Google aims for early 2027, while Microsoft says its preferences take effect from February 1, 2027. The companies’ schedules differ, and full service-by-service timing is not specified.
Has Meta committed to honoring the shared rule?
No commitment is stated in the source material. Meta spokesperson Andy Stone expressed support for publishers’ ability to set preferences and for a common signal, but did not say whether Meta would honor it.
Source: ThorstenMeyerAI.com
