On 13 November 2026, Search Engine Journal published a piece by Chris Green, Technical Director at Torque Partnership, on AI bot management. His argument is that SEOs should move from block lists to deliberate bot policies. He splits bots into traditional search crawlers, AI training crawlers, retrieval bots, user-initiated fetches and autonomous agents. He says blocking a training crawler and blocking a retrieval crawler have different consequences, and that lumping them together as “AI bots” is a mistake.
The claim: treat all AI bots the same and block them
Many site owners have done this, or been told to. The reasoning is easy to follow. AI companies took content to train models, so shut the door on all of them.
Green’s piece has no statistics or measured data, so I’ll say that up front. It is an argument about how to think, and I think the argument holds up.
What actually changed
Some of those bots collect content to train a model. Others fetch a page at the moment someone asks a question, so the answer can include what your site says. Agents go further and browse like a person, doing things like comparing products for a buyer.
Green warns that blocking an agent doing a product comparison could remove a site from the answer entirely, not just lower its ranking. That is the part a block list misses. If a bot is the one reading your facts on behalf of a buyer, blocking it means the buyer’s answer gets written without you.
What didn’t change is that robots.txt is voluntary. Green points out that real enforcement happens in your CDN, your web application firewall and your bot mitigation service. Those can override your robots file “without the SEO team even knowing.” So you can write a sensible policy in one place and have another layer undo it.
My take: being named in the answer is the outcome
I’ve said before that ranking on page one no longer decides the result. Being named in the answer does. The AI doesn’t pick the biggest brand. It picks the one it understands most clearly and can quote most precisely. That only works if the bots that fetch answers can read your pages.
Take a clinic with a good page for each treatment, with prices, opening hours and who the doctors are. Those are the precise facts an engine can quote. Now suppose someone adds a block list from a blog post, and the firewall starts turning away the bots that fetch pages for answers. The clinic has done the hard work of being clear, and the engines can’t see it. Nobody on the team notices, because rankings look fine and no alert fires.
So I’d split the decision in two:
- Retrieval and agent bots: let them read. These are the ones that put your facts into an answer or a comparison, and you want them accurate.
- Training crawlers: decide separately. This is a real business choice with fair arguments on both sides. It depends on what your content is worth to you and what you are comfortable with. Don’t let it ride along with the first decision.
I also wouldn’t expect the robots file to carry the strategy. It states your intent. It doesn’t enforce anything on its own, as Green’s point about CDNs and firewalls shows. The same goes for the other tidy files and “AI-ready” checklists. If something is cheap, harmless and can be automated, do it and move on. It is still not the strategy. The strategy is having facts that are precise and different enough from your competitors’ that an engine can name you with confidence.
This gets more important as more of the buying conversation becomes agents talking to agents. Big platforms may decide to block AI agents to protect their hold on shoppers. Amazon blocking AI shopping agents is the example I keep in mind. They can afford to wait and see. A small business can’t. For a smaller operator, wherever your data shows up needs to be accurate and detailed, and that starts with letting the right readers in.
Where this shows up in my own work
The method I use at Stridec starts from the same idea. Write down exactly what a business is, who it is for and how it differs, in plain facts, then make sure the engines can read it. Numbers survive in AI answers. Adjectives don’t.
What a business should do
- List every layer that can block a bot: your robots file, your CDN, your firewall and any bot protection plugin. Check what each one actually does, not what you think it does.
- Write down which bots you want reading your facts. Retrieval bots and agents are the default yes for any business that wants to be recommended.
- Make a separate decision on training crawlers, and record why.
- Put one person in charge of the policy, with SEO, engineering and security in the same conversation, as Green suggests.
- Search for your own business in the AI engines, and note whether you are named. Do it again after any change to your firewall or bot rules, and date what you see, because answers change week to week.
Source: Search Engine Journal: AI Bot Management: Why SEOs Need More Than Block Lists