Google, OpenAI, Microsoft and Anthropic agreed on one AI training opt-out. Should a CPG brand use it?

On Friday, October 30, the people who run the biggest AI models agreed on something small and useful. Google, Microsoft and OpenAI each said they will read one shared line in robots.txt that tells them whether a website's pages may be used to train generative AI models. The same day, an Anthropic engineer submitted the IETF draft that defines the line, and PPC Land reports he told the working group Anthropic has started building support.

The short version for a packaged goods brand: this was built mostly for publishers who want to keep their writing out of model training. A brand that wants assistants to describe its products correctly has the opposite goal. Most CPG sites should not add the new line, and the more urgent job is making sure nobody adds it for you by accident.

What was announced

The mechanism is an ordinary robots.txt group. Instead of naming a crawler on the User-Agent line, the site names a "policy token", __usage_ai-training, and then uses Disallow and Allow exactly as it would for a bot. The draft's own example disallows the whole site and allows one folder, /press-kit/, which "expresses a preference against AI training for all paths except those under /press-kit/."

Three details in the draft matter more than the headline.

First, it is about use, not access. The draft says plainly that "this policy token is not a directive for crawling behavior." A crawler that reads your pages for search keeps doing so. If you want a bot to stop fetching pages, you still need a group for that bot's own name.

Second, the wildcard does not count. The draft says groups for User-Agent: * "MUST NOT be interpreted as matching the policy token." If your file never names the token, you have expressed no preference at all, which in practice means the companies' defaults apply.

Third, the companies keep their own controls. Google says Google-Extended keeps working, OpenAI says its existing robots.txt controls "will continue to apply and be respected," and Microsoft is building a page-level version on top.

Table: Who committed to the shared AI training token, and when it takes effect

Company	What it said on Oct 30, 2026	When
Microsoft	Will honor the shared token, plus a page-level training setting and a separate grounding setting	From February 1, 2027, as it crawls
Google	Will treat a shared-token opt-out as a Google-Extended disallow when Google-Extended is not named	2027; its crawler docs say early 2027
OpenAI	Will honor the common signal; existing crawler controls stay	"In the coming months"
Anthropic	Draft author says support is being implemented	No date given

Sources: Google Search Central Blog, Microsoft Bing Blog and OpenAI, October 30, 2026; Anthropic's statement as reported by PPC Land, October 31, 2026.

The Google detail most brands will miss

Google's post contains one sentence that changes the decision for a brand. When a site disallows the shared token and does not mention Google-Extended, "Google plans to interpret that as if the publisher had disallowed Google-Extended." A few lines later: "Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps."

Grounding is the step where an assistant fetches live web pages while it writes an answer and links to them. PPC Land read Google's crawler documentation alongside the post and concluded that, at Google, the one line refuses both training and Gemini grounding, while at Microsoft it refuses only training. That is PPC Land's reading rather than Google's words, but Google's own sentence about allowing Google-Extended points the same way.

So a brand that adds the shared opt-out to keep its copy out of training may also, once Google implements it, take its product pages out of the pool Gemini Apps cites with links. For a company whose growth plan includes being recommended by Gemini, that is a bad trade. Google's opt-out does not touch AI Overviews or AI Mode in Search, and it does not affect Google Search rankings. It is the Gemini app that is at stake.

At Microsoft, the one line refuses training. At Google, unless you also allow Google-Extended, it is read as refusing Gemini grounding too.

Microsoft's grounding switch, and its warning

Microsoft went further and split the two jobs. AI Training is the shared robots.txt preference, with an optional page-level override. AI Grounding is a separate page-level preference with three values: Allow (the default), DisallowExtended, which limits grounding to Microsoft's own experiences such as Copilot and excludes grounding services it sells to third parties, and Disallow.

Microsoft is unusually direct about the cost. If you choose DisallowExtended or Disallow for grounding, "you will see reduced brand visibility in AI experiences," and a caution in the same post adds that these settings "will significantly reduce your content's visibility in AI-generated experiences." It also says that neither choice "guarantees inclusion, traffic, or payment."

There is a transition for sites that used Bing's older NoArchive or NoCache tags as AI controls. Temporary grounding preferences open in Bing Webmaster Tools on December 15, 2026; new preferences take effect from February 1, 2027; after March 1, 2027 the old tags are no longer read for training or grounding; and temporary preferences expire on September 30, 2027.

Table: Microsoft's AI preference calendar

Date	What happens
Dec 15, 2026	Temporary grounding preferences open in Bing Webmaster Tools, only for sites using NoArchive or NoCache
Feb 1, 2027	Microsoft starts honoring AI Training and AI Grounding preferences as it crawls
Mar 1, 2027	NoArchive and NoCache stop being read as training or grounding preferences
Sep 30, 2027	Any remaining temporary grounding preference expires

Source: Microsoft Bing Blog, October 30, 2026.

Why we think most CPG brands should leave training on

A publisher's product is its writing, so keeping it out of training protects the thing it sells. A brand's product is on a shelf. The website exists so that people, and now assistants, can find out what the product is, what is in it, who it suits and where to buy it. Those facts are meant to be repeated, and the more accurately a model has absorbed them, the less a shopper depends on a stale reseller listing or a forum guess.

We made the case for working on training data in our training-data playbook: assistants often decide which brands to name before they search. A training opt-out moves a brand further from that list, not closer.

There are reasonable exceptions. A brand with a paid recipe library, a licensed content archive, or a research report it sells has something worth protecting. The draft's design handles that well: disallow the shared token for that folder only and leave product, ingredient and FAQ pages alone. Path-level rules are the point of the format.

Your site is only part of the picture

One limit is easy to forget. Your robots.txt only speaks for your own domain. The product pages that assistants quote most often for packaged goods frequently sit on retailer sites, review sites and recipe blogs, and each of those sites sets its own preferences. A brand cannot opt its Amazon or Walmart listing in or out of training, and it would not want to try. What a brand controls is whether its own site, usually the most accurate source for ingredients, sizes and usage, is available to the models and to the assistants that ground their answers in it.

That is another reason to treat the new token as a narrow tool. If your own pages drop out while retailer copies stay in, the assistant still describes your product. It just does so from a page you did not write, which may be out of date after a reformulation or a pack change.

What to do this month

Nothing changes until 2027, so this is housekeeping rather than a fire drill. We would do four things.

1. Open your live robots.txt and search for __usage_ai-training, Google-Extended, GPTBot and ClaudeBot. Write down what each group says and who added it. Our Common Crawl check of brand sites found blocks that nobody on the marketing team knew about.
2. Ask whoever runs your CDN, bot-management tool and ecommerce platform whether any setting will write the new token for you. PPC Land notes that Cloudflare's training opt-out already writes vendor tokens such as Google-Extended into customers' files. If a vendor adds the shared token, find out whether it will also add an explicit Allow for Google-Extended.
3. If you do want a training opt-out for some content, scope it to paths. Put the protected material in its own folder and disallow only that folder, so product pages stay available.
4. If your site ever used NoArchive or NoCache on product pages, list those pages now. Microsoft's transition only applies to sites using those tags, and you will want to decide before December 15 whether you meant them as AI controls at all.

How to check the result

Once the companies implement the signal, the evidence will show up in answers rather than in robots.txt. Ask Gemini, Copilot and ChatGPT the questions shoppers ask about your category, check which pages they cite, and repeat the same questions after any robots.txt change. Bing Webmaster Tools already reports how often your site is shown as a source in AI-generated answers, which makes Microsoft the easiest place to watch. XENA Intelligence asks assistants such as ChatGPT, Perplexity and Gemini this kind of shopper question and scores products on how they show up, with the answers kept product by product in SearchPanel; a spreadsheet and a calendar reminder will do the same job for a small catalog.

The draft itself expires on May 3, 2027 unless it is revised, and the IETF datatracker notes that it has no formal standing yet. Expect details to change. The principle for a CPG brand is unlikely to: let assistants learn and cite your product facts, and protect only the content you actually sell.