Until this week, telling AI companies not to train on your content meant maintaining a list. One rule for Google, another for OpenAI, another for Anthropic, and a fresh entry every time a provider appeared.
On October 30, 2026, Google published its support for a shared AI training control (opens in a new tab) in robots.txt. Anthropic, OpenAI and Microsoft committed to the same signal. One directive now carries the preference across every participating provider.
How the signal works
The mechanism comes from a proposal in the AIPREF working group at the IETF, written by Anthropic's technical lead for standards, Kevin Kelley. Instead of naming crawlers one at a time, a publisher sets a single policy token:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
The Disallow line opts out of AI training across participating providers. The Allow line opens specific paths back up, so a press kit can stay available for training while the rest of the site stays out.
What changes for a publisher
| Before | Now | |
|---|---|---|
| Providers to manage | One rule per AI developer | One shared policy token |
| A new provider appears | Add another rule | Covered by the same signal |
| Partial opt-in | Separate per-crawler rules | One Allow path inside the token |
| Search indexing | Unaffected | Unaffected, since the token targets training |
Two details matter if you run a site.
Search is untouched. Google reads the opt-out as a directive aimed at Google-Extended, the training control it has offered since 2023, rather than at the crawler that indexes pages for Search. Opting out of training keeps your pages in search results.
It carries regulatory weight. Google notes the proposal is consistent with the EU Code of Practice for General Purpose AI model providers under the EU AI Act, which recognises robots.txt as a primary method for publishers to express machine-readable preferences. The signal therefore has a basis that reaches past the providers who have committed to it so far.
Grounding is a separate decision, and it is a lever. Google's post is explicit: sites that specifically allow Google-Extended remain eligible to appear in responses and receive clicks through grounding in Gemini Apps. Training and grounding can now be set independently, which gives you a way to stay visible in AI answers while keeping your archive out of training runs.
The same question, one layer down
A robots.txt token answers a public question: may an outside crawler read this? Inside a company, the identical question shows up privately, many times a day. Which agents may read the customer database? Which can see salary band data? Who authorised the last one?
A file of preferences works for a website because the audience is a crawler that reads and obeys. Inside your own systems the readers are agents acting with your credentials, and the controls that hold are the ones enforced at the point of access.
That enforcement layer is what CommandLyne (opens in a new tab) provides around enterprise AI work:
- Scoped tool access, so agents reach only the specific tools each task needs
- Permission-aware memory, keeping personal memory private to each user while team memory is shared only across authorised teams
- Full audit trail, recording every prompt, data query and tool execution, with visibility into what context was retrieved for any answer
- Human sign-off, enforcing approval before sensitive actions execute
- Customer-managed encryption, using your own Google Cloud keys through KMS and CMEK
Deployment follows the same principle. CommandLyne runs on a Codimite-managed platform, as a dedicated single-tenant environment, or self-hosted inside your own Google Cloud infrastructure, where your IAM, networking, billing and keys stay under your control.
| robots.txt token | Governed enterprise access | |
|---|---|---|
| Who it addresses | Outside crawlers | Your own agents, inside your systems |
| How it is expressed | A preference in a file | An enforced scope at the point of access |
| Who honours it | Providers who have committed | The platform, on every call |
| What it covers | Training on published pages | Every prompt, query and tool execution |
| What you get back | A clear public signal | An audit trail and an approval record |
What to do this week
If you run a public site, the practical work is small and the return is clear. Add the policy token to robots.txt, decide deliberately which paths to open with Allow, and treat the grounding decision as separate from the training decision, because it now is.
Then apply the same reasoning inside your organisation. Public content has one signal. Internal data has an audience of agents, and the controls that matter there are the ones enforced on every request rather than declared once.
Try CommandLyne to put enforced, auditable access around every AI action in your organisation, or book a demo and we will map it to your stack.
Try CommandLyne (opens in a new tab) | Book a demo (opens in a new tab)

