On 30 October 2026, The Verge reported that OpenAI, Google, Microsoft and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI. Search Engine Roundtable reported it first. The idea is a shared “policy token” signal: a publisher sets its AI training preferences once, and they apply across multiple AI companies instead of being set for each one individually.
What people are excited about, and what actually changed
Four of the biggest names in AI agreeing on one signal is a real change for anyone who has been keeping a list of bot names in a robots.txt file and hoping each company honours its own entry. One setting, applied everywhere, is less work and less guessing. For a news publisher whose words are the product, I understand the appeal.
So the first question I ask of any launch applies here. Is there a need for it, and for whom? For publishers, yes. For a shop, a clinic or a service business, the need is much less obvious, and the risk of switching it on without thinking is higher than the risk of leaving it alone.
What changed is that saying no got cheaper. What did not change is how AI engines decide which business to name when someone asks a question. That part still works the way it did last week, and the opt-out switch has nothing to do with it.
The switch is a decision about how well the AI knows you
My view on AI search has not moved. The AI doesn’t pick the biggest brand. It picks the one it understands most clearly and can quote most precisely. Ranking is dead. A page-one position on Google no longer does the job on its own, because answers now come from a broader set of engines and clicks have collapsed. On stridec.com, non-branded positions one to three got a 0.08% click-through rate between June and August 2026. Being named in the answer is the outcome that pays.
Read the new switch against that. Opting out is asking the machines to know less about you. The feed summary talks about training preferences. Whether an engine reads your page to answer a question tonight is a different matter, and before anyone touches a setting, they should know which of the two they are turning off. A business that wants customers from AI answers wants to be read, understood and quoted. Everything else follows from that stance.
What this looks like for a real business
Take a clinic. A patient asks ChatGPT or Google which clinic nearby does a particular treatment, how long it takes and roughly what it costs. The engine names the clinic whose pages say those things plainly. If the clinic’s agency ticked the opt-out box because it sounded like the safe option, the clinic becomes a page the engines are told not to learn from. Nothing has been protected. A clinic’s treatment list is not something a competitor steals by reading it. The thing of value is the appointment, and the appointment now depends on being named.
The same applies to a shop. Expect a split: some platforms will stay open and indexable, others will restrict access to keep their grip on shoppers. The giants can afford to wait that out or shape how it turns out. A smaller business can’t. For a smaller business the shift is already decided: get data-centric, and make sure that wherever your information shows up, it is precise, detailed and an accurate reading of what you are.
The file is a chore. Automate it and move on
This signal joins llms.txt and the rest of the “AI-ready” checklist. My rule for all of them is the same. Do it, don’t worship it. AI search engines are smarter than to put much weight on a simple text file. But if something can be automated at no cost and no downside, let the machine handle it. On my own site the content engine writes the llms.txt entry for every post it publishes, and I never think about it. I use Claude for almost everything, and file hygiene is exactly the kind of task it should own.
The mistake is treating the file as the strategy. When AeroChat got into Google’s AI Overview within three weeks of launch, alongside Tidio and Intercom, it was because we wrote down exactly what AeroChat is, who it’s for and how it’s different, with no marketing language, and built comparison pages around the questions that actually produced an AI answer. No file did that. Your own pages get you cited fast. Other people’s pages keep you there.
What to do
- Decide the stance first. If you sell anything and want AI answers to send you customers, stay open to crawling and to being read for answers. Only a business whose words are the product has a reason to think harder.
- Check what is already set. Look at your robots.txt and whatever your agency or hosting provider has added before you adopt the new signal. People tick these boxes on your behalf more often than you’d think.
- Hand the file work to a script or a tool. Robots, llms.txt and this token should take none of your week.
- Spend the saved time on the pages. State plainly what you are, who you serve, what you charge and how you differ, with specific facts. Engines quote numbers and drop adjectives.
- Test it and date it. Ask the engines the questions your buyers ask and see whether you are named. Answers change week to week, so note the day you checked.
Source: The Verge: The biggest names in AI are backing a new way to opt out of scraping