Skip to content

robots.txt now has a single AI training switch

Blog cover with the official CommandLyne logo: the headline One robots.txt line now controls AI training in white type beside a policy file card with a blue toggle switch fanning out to four provider nodes on a dark navy grid.
CommandLyne TeamAI Product & Engineering

Nov 2, 20265 min read

Until this week, telling AI companies not to train on your content meant maintaining a list. One rule for Google, another for OpenAI, another for Anthropic, and a fresh entry every time a provider appeared.

On October 30, 2026, Google published its support for a shared AI training control (opens in a new tab) in robots.txt. Anthropic, OpenAI and Microsoft committed to the same signal. One directive now carries the preference across every participating provider.

How the signal works

The mechanism comes from a proposal in the AIPREF working group at the IETF, written by Anthropic's technical lead for standards, Kevin Kelley. Instead of naming crawlers one at a time, a publisher sets a single policy token:

User-Agent: __usage_ai-training
Disallow: /

Allow: /press-kit/

The Disallow line opts out of AI training across participating providers. The Allow line opens specific paths back up, so a press kit can stay available for training while the rest of the site stays out.

What changes for a publisher

BeforeNow
Providers to manageOne rule per AI developerOne shared policy token
A new provider appearsAdd another ruleCovered by the same signal
Partial opt-inSeparate per-crawler rulesOne Allow path inside the token
Search indexingUnaffectedUnaffected, since the token targets training

Two details matter if you run a site.

Search is untouched. Google reads the opt-out as a directive aimed at Google-Extended, the training control it has offered since 2023, rather than at the crawler that indexes pages for Search. Opting out of training keeps your pages in search results.

It carries regulatory weight. Google notes the proposal is consistent with the EU Code of Practice for General Purpose AI model providers under the EU AI Act, which recognises robots.txt as a primary method for publishers to express machine-readable preferences. The signal therefore has a basis that reaches past the providers who have committed to it so far.

Grounding is a separate decision, and it is a lever. Google's post is explicit: sites that specifically allow Google-Extended remain eligible to appear in responses and receive clicks through grounding in Gemini Apps. Training and grounding can now be set independently, which gives you a way to stay visible in AI answers while keeping your archive out of training runs.

The same question, one layer down

A robots.txt token answers a public question: may an outside crawler read this? Inside a company, the identical question shows up privately, many times a day. Which agents may read the customer database? Which can see salary band data? Who authorised the last one?

A file of preferences works for a website because the audience is a crawler that reads and obeys. Inside your own systems the readers are agents acting with your credentials, and the controls that hold are the ones enforced at the point of access.

That enforcement layer is what CommandLyne (opens in a new tab) provides around enterprise AI work:

  • Scoped tool access, so agents reach only the specific tools each task needs
  • Permission-aware memory, keeping personal memory private to each user while team memory is shared only across authorised teams
  • Full audit trail, recording every prompt, data query and tool execution, with visibility into what context was retrieved for any answer
  • Human sign-off, enforcing approval before sensitive actions execute
  • Customer-managed encryption, using your own Google Cloud keys through KMS and CMEK

Deployment follows the same principle. CommandLyne runs on a Codimite-managed platform, as a dedicated single-tenant environment, or self-hosted inside your own Google Cloud infrastructure, where your IAM, networking, billing and keys stay under your control.

robots.txt tokenGoverned enterprise access
Who it addressesOutside crawlersYour own agents, inside your systems
How it is expressedA preference in a fileAn enforced scope at the point of access
Who honours itProviders who have committedThe platform, on every call
What it coversTraining on published pagesEvery prompt, query and tool execution
What you get backA clear public signalAn audit trail and an approval record

What to do this week

If you run a public site, the practical work is small and the return is clear. Add the policy token to robots.txt, decide deliberately which paths to open with Allow, and treat the grounding decision as separate from the training decision, because it now is.

Then apply the same reasoning inside your organisation. Public content has one signal. Internal data has an audience of agents, and the controls that matter there are the ones enforced on every request rather than declared once.


Try CommandLyne to put enforced, auditable access around every AI action in your organisation, or book a demo and we will map it to your stack.

Try CommandLyne (opens in a new tab) | Book a demo (opens in a new tab)

See CommandLyne in action

Book a demo to see how CommandLyne brings your agents, models and workflows together under one governed workspace.