OpenAI, Google, Microsoft and Anthropic back a shared robots.txt token for AI training opt-outs

OpenAI, Google, Microsoft and Anthropic have committed to honor a single robots.txt signal that lets a website state, in one place, that its content should not be used to train AI models. The mechanism is a new policy token, __usage_ai-training, described in an Internet-Draft titled "Machine-Readable Training Preference Signal" that was published on October 30, 2026 and authored by Kevin Kelley of Anthropic. Instead of listing GPTBot, ClaudeBot, Google-Extended and every other training crawler by name, a site operator writes one ordinary robots.txt group addressed to the token, and every participating provider is expected to read that group as the site's training preference.

The four companies have not moved on the same schedule. Anthropic has already started implementing support, according to the companies' statements on the day of the announcement. Google's own crawler documentation now says it intends to support the shared signal in early 2027, and Google said it will treat an opt-out expressed through the token as if the site had disallowed Google-Extended, unless the site addresses Google-Extended directly. OpenAI said it aims to implement the signal in the coming months and to publish implementation details afterwards. Microsoft said it plans to support the signal in 2027 and to add a separate setting that distinguishes training from grounding, the retrieval step that pulls live web content into chat answers.

For publishers, developers and anyone who maintains a robots.txt file, the practical consequence is a simpler opt-out once the providers' implementations are live, layered on top of the controls that already exist rather than replacing them. The signal is a voluntary preference, not an access-control mechanism, and the draft itself has no formal standing in the IETF standards process. Both points shape what the token can and cannot do.

··········

HOW THE __USAGE_AI-TRAINING TOKEN WORKS.

The draft deliberately avoids inventing new robots.txt syntax. It adds no new directive names and no new grammar, and it leaves the path language, the longest-match precedence and the group structure of robots.txt, as defined in RFC 9309, unchanged. What it introduces is a policy token placed in the User-Agent line of a normal group. The token follows the RFC 9309 product-token syntax, which is why it is written with letters, underscores and a hyphen, and the leading double underscore makes it unlikely to collide with the name of any real crawler.

The example given in the draft expresses a preference against AI training for an entire site, with a single exception for a press-kit directory:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Under the draft's interpretation rules, a disallowed path signals a preference against the use, and an allowed path signals a preference in favor of it. In the example above, the site therefore objects to AI training on everything except /press-kit/, where it actively states that training is acceptable. Consumers of the signal look up a path against the token using the same path-matching and precedence rules that crawlers already apply under RFC 9309, so the most specific matching rule wins in the usual way.

A group can name several policy tokens on consecutive User-Agent lines, and policy tokens can be mixed with ordinary crawler product tokens in the same group. That flexibility matters for sites that already maintain long robots.txt files: an operator can attach the new token to an existing group of training crawlers rather than restructuring the file.

··········

THE MATCHING RULES THAT DECIDE WHAT COUNTS AS AN OPT-OUT.

Several rules in the draft are narrower than many site operators might assume, and they determine whether an existing robots.txt file already expresses a preference. The most important is that wildcard groups do not count. A group that begins with User-Agent: * is not interpreted as matching the policy token, so a site that blocks all crawlers through a wildcard has not, under this mechanism, expressed a training preference. If no group in the file names __usage_ai-training, the draft says that no preference has been expressed at all.

Token matching must be exact, ignoring case. The draft notes that some deployed robots.txt parsers match product tokens more loosely, but it instructs consumers not to rely on that behavior. Proprietary policy tokens that may appear in a file are not read as preferences under this document, which keeps the shared signal separate from any vendor-specific experiments.

The draft also separates usage preferences from traffic management. Rules written for a crawler's own product token keep their existing RFC 9309 meaning, and the policy token is explicitly not a directive for crawling behavior. In practice, a site can tell a crawler it may fetch pages, for example to keep appearing in a search or answer engine, while using the policy token to say that the fetched content should not be used for training.

........

Rule in the draft	What it means for a robots.txt file
Wildcard groups excluded	User-Agent: * does not express a training preference
No token present	No preference is expressed at all
Exact, case-insensitive match	Misspelled or loosely matched tokens should not be relied on
Disallow / Allow	Disallow signals a preference against training; Allow signals one in favor
Separate from crawl rules	Crawler product-token rules keep their RFC 9309 meaning
Precedence	More specific expressions should win over general ones
Private agreements	Licensing deals between a site and a provider may override the preference

........

··········

WHAT EACH COMPANY HAS COMMITTED TO AND WHEN.

The four companies made their commitments on the same day, but the details differ, and only some of them are documented on official pages so far. Google's position is the most precisely written down. Its crawler documentation states that Google intends to support the shared machine-readable training preference signal, with "user-agent: __usage_ai-training", in early 2027. Google said company-specific controls will continue to function alongside the joint signal, and that an opt-out expressed through the shared token without an explicit Google-Extended rule will be interpreted as if the publisher had disallowed Google-Extended. An explicit Google-Extended rule therefore remains the stronger, more specific instruction for Google.

Anthropic is both the origin of the draft and the first to act on it: the company has said it has already begun implementing support. Anthropic's existing crawler documentation still describes the per-bot controls, under which disallowing ClaudeBot signals that a site's future materials should be excluded from training datasets, while Claude-User and Claude-SearchBot serve user-directed retrieval and search.

OpenAI said it aims to implement the signal in the coming months, described the existing robots.txt controls as continuing to be respected, and said it will publish implementation details once support is in place. Microsoft said it will support the signal in 2027 and plans more granular settings that let publishers separately control the use of their content for AI training and for grounding model responses, while remaining in Bing search results.

........

Company	Status of support	Stated timing	Existing control that keeps working
Anthropic	Implementation under way	Already started	ClaudeBot
Google	Planned	Early 2027	Google-Extended
OpenAI	Planned	Coming months	GPTBot
Microsoft	Planned, with a separate grounding setting	2027	Existing Bing controls

........

None of the four has published a date by which every one of its training pipelines will read the token, and the companies have not said whether the preference will be applied to content collected before their implementations go live. Those are the two questions that will decide how much the signal changes in practice.

··········

WHAT CHANGES FOR PUBLISHERS AND SITE OPERATORS.

Until now, a site that wanted to keep its content out of AI training had to track each provider's training crawler or control token and write a separate rule for each one. OpenAI's documentation, for example, treats GPTBot as the training crawler and OAI-SearchBot as the crawler that surfaces sites in ChatGPT search, and states that each setting is independent of the others. Google uses Google-Extended as a control token without a separate user agent, and its documentation says Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. Anthropic separates ClaudeBot from Claude-User and Claude-SearchBot. The shared token does not remove any of this, but it gives operators one line that all four companies have said they will read.

A site that wants to opt out of training everywhere the token is honored can add a dedicated group for __usage_ai-training with Disallow: /. Operators who already block training crawlers by name do not need to remove those rules, since every company has said its existing controls continue to work, and the provider-specific rules will remain the only effective mechanism until each implementation is live. Keeping both is the conservative choice for the transition period, which for Google and Microsoft runs into 2027.

The Google-specific behavior deserves attention from publishers that care about citations in Gemini. Because Google will read a shared opt-out as a Google-Extended block, and because Google-Extended also governs grounding in Gemini Apps and Grounding with Google Search on Vertex AI, a site that uses the shared token without a Google-Extended rule could also step out of that grounding use. A publisher that wants to refuse training generally but stay eligible for grounding in Gemini would need to address Google-Extended explicitly, accepting that Google-Extended covers both uses. Microsoft's planned split between training and grounding points in the same direction, although Microsoft has not yet published how that setting will be written.

Operators should also check how their file treats wildcards. A blanket User-Agent: * block does not express a training preference under the draft, so a site that relies only on such a block will need a group that names the token explicitly to be covered by the new mechanism.

··········

THE LIMITS OF A VOLUNTARY SIGNAL.

The token is a statement of preference, and its effect depends entirely on whether the companies that read it choose to respect it. Robots.txt has always worked as a coordination mechanism between site operators and well-behaved crawlers rather than a security control, and nothing in the draft changes that. Scrapers that ignore robots.txt today can ignore the policy token in the same way, so operators who need hard enforcement still have to rely on access controls at the server, network or contractual level.

The draft's standing is also modest. It is an individual Internet-Draft, published on October 30, 2026 and due to expire on May 3, 2027 unless it is revised, and the IETF notes that such a document is not endorsed by the IETF and has no formal standing in its standards process. It sits alongside the wider work of the IETF's AI Preferences working group, which is developing a vocabulary for content-usage preferences, but the commitments made so far come from the four companies rather than from a finished standard.

The draft also allows private agreements between a site and a provider to override the expressed preference. Publishers with licensing deals, which are increasingly common between news organizations and AI companies, will have to read those contracts alongside their robots.txt files, a theme that also surfaced in the UK regulator's case examined in our coverage of the Meta CMA investigation into publisher content terms.

··········

TECHNICAL SYNTHESIS: ONE TOKEN, FOUR TIMELINES.

What is established is the mechanism and the intent. The __usage_ai-training token is defined in a dated Internet-Draft authored at Anthropic, it reuses the existing robots.txt grammar without modification, and it comes with specific rules: wildcard groups do not count, the absence of the token means no preference, and the signal is separate from crawl permissions. Google has written its early-2027 plan and its Google-Extended interpretation into its official crawler documentation, and Anthropic, OpenAI and Microsoft have each stated that they will honor the signal.

What remains uncertain is the timing and the scope. Only Anthropic has said that implementation has started; OpenAI has given no date beyond the coming months, and Google and Microsoft point to 2027. None of the companies has said whether the preference will apply to data already collected, Microsoft has not published the syntax of its grounding control, and the draft itself has yet to move beyond an individual submission. For now, site operators gain a clear way to state a training preference once, with the practical effect arriving provider by provider over the coming year.

··········

FOLLOW US FOR MORE.

·····

DATA STUDIOS

·····

datastudios.org