Technical ·
Should you add the new shared AI training opt-out to your robots.txt?
Under an hour ChatGPTGoogle
Short answer
For most brand sites, no. If you do want a training opt-out, scope it to the folders you actually need to protect and add an explicit allow for Google-Extended, or Google will also read it as refusing Gemini grounding.
On October 30 Google, Microsoft and OpenAI said they will honor one robots.txt group, User-Agent: __usage_ai-training, as a preference against AI training, with Microsoft starting February 1, 2027 and Google in early 2027. It was built for publishers, whose writing is the product. A brand is in a different position: its ingredients, sizes and instructions are meant to be repeated accurately. There is also a catch at Google. If the token says Disallow and the file does not name Google-Extended, Google plans to treat that as a Google-Extended disallow, and Google says sites that allow Google-Extended are the ones eligible for grounding with clicks in Gemini Apps. The more likely risk is not that your team adds the line, but that a CDN or security setting adds it for you.
Do this
- Open your live robots.txt and search for __usage_ai-training, Google-Extended, GPTBot and ClaudeBot. Note what each group says and who added it.
- Ask whoever runs your CDN, bot management and ecommerce platform whether any setting writes AI training rules into the file for you.
- If you need a training opt-out for paid recipes or licensed content, disallow only that folder under the new token and leave product and FAQ pages open.
- If you keep any opt-out, add a separate User-Agent: Google-Extended group with Allow: / for the pages you want Gemini to cite.
- Put a reminder in February 2027, when Microsoft starts reading the new preferences, to check that the file still says what you meant.
Effort is our estimate for a small brand team. Assistant tags come from the assistants this tip names. See also AI shopping assistants compared.