OpenAI, Google, Microsoft and Anthropic will honor one robots.txt line against AI training

What was announced on October 30

Four AI labs said on October 30, 2026 that they will read one robots.txt group as a website’s choice about AI training. Until now, opting out meant a separate rule for each lab’s crawler: GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google, and NoArchive or NoCache tags for Microsoft. The new group speaks to all of them at once:

User-Agent: __usage_ai-training
Disallow: /

__usage_ai-training isn’t a crawler. It’s what the IETF draft behind it calls a “policy token,” written where a crawler’s name usually goes. The draft, published the same day by Kevin Kelley, a technical lead for standards at Anthropic, “introduces no new directive names and no new grammar,” so Allow and Disallow work by path exactly as they do today. Its own example keeps a press kit open while opting the rest of the site out:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

“Training” here means “using an asset to modify the learned parameters of a generative AI model.” Training models used only for non-generative tasks isn’t included. The draft is an individual submission, “not endorsed by the IETF,” and it expires on May 3, 2027 unless it’s renewed or adopted. The word for training comes from the AI Preferences working group’s vocabulary draft, which is still being worked out.

When each company starts

None of them reads it everywhere yet:

Company	When it honors the token	What it said
Anthropic	Started, no finish date	“Anthropic has already started implementing support for this draft. It’ll take us some time to add support across our services.” (IETF list)
OpenAI	No date	“We aim to implement the signal in the coming months, and will share another update with details about our implementation when we do so.” (OpenAI)
Microsoft	February 1, 2027	“The new preferences will be effective on February 1, 2027, as Microsoft crawls.” (Bing)
Google	Some time in 2027	“We aim to support this new control along with other AI providers in 2027.” (Google)

Meta didn’t commit. Its communications head, Andy Stone, called a common signal “an important step,” The Verge reported, without saying whether Meta will honor it.

What to add to your robots.txt now

1. Add the shared group. It costs nothing, and Anthropic is already starting to read it. Give it its own group: the draft says wildcard groups, User-Agent: *, “MUST NOT be interpreted as matching the policy token.”
2. Keep your per-crawler rules. OpenAI says its existing crawler controls “will continue to apply and be respected,” and Google says “Existing company-specific controls will continue to function alongside this new joint signal.” Until each company switches the token on, User-agent: GPTBot, User-agent: ClaudeBot and User-agent: Google-Extended with Disallow: / are what it reads.
3. Know how a specific rule beats the shared one. Google says that when a site opts out with the token “and Google-Extended is not explicitly specified,” it will treat Google-Extended as disallowed. If you name Google-Extended, that rule decides for Google. The draft says “specific expressions SHOULD take precedence over general ones,” so allowing one lab and opting out of the rest stays possible.
4. If you use NoArchive or NoCache for Bing, plan the move. Microsoft says the new preferences replace them, and to “complete the steps above by March 1, 2027; after that date, NoArchive and NoCache will no longer be read as a preference for training or grounding.”

What it doesn’t do

• It doesn’t block crawling. The draft says the token “is not a directive for crawling behavior,” and Bing says these settings “govern use of content, not crawl access.” Crawlers can still fetch your pages; you’re stating what they may be used for.
• It doesn’t take you out of search or AI answers. Bing says the preferences “do not change eligibility for traditional Bing search results.” For Bing, Copilot and Microsoft 365 answers, Microsoft is adding a separate page-level grounding preference, <meta name="Bingbot" content="ai-grounding:disallow"> or the matching X-Robots-Tag header, and warns that it “will significantly reduce your content’s visibility in AI-generated experiences.” OpenAI’s OAI-SearchBot, which decides whether you appear in ChatGPT search, has its own rule.
• It doesn’t bind anyone. OpenAI calls it a “voluntary effort,” and a crawler that ignores robots.txt will ignore this too.
• It doesn’t decide whether an AI agent can open your site. OpenAI says of ChatGPT-User, which visits pages when a person asks, that “robots.txt rules may not apply.” Separately, Search Engine Journal reported on October 29 that a site-safety check OpenAI hasn’t documented by name can stop ChatGPT and Codex from opening a site at all, with errors such as “Browser use is not permitted on [domain],” and that there’s no public way to appeal.

What hasn’t been said

• Whether it reaches back. No company has said whether opting out affects content it collected before it started reading the token, or models already trained.
• OpenAI’s and Anthropic’s dates. OpenAI has no date, and Anthropic hasn’t said when its support will cover all its services or which crawlers read it so far. Anthropic’s crawler help article still documents only ClaudeBot, Claude-User and Claude-SearchBot.
• Whether the others add a grounding setting. Microsoft’s is its own; Google, OpenAI and Anthropic haven’t announced one.
• Whether the IETF adopts it. For now it’s one person’s draft, and the working group’s discussions continue.

We’ll update this memo as each company switches the token on.