What happened
On 2 November 2026, Slobodan Manic published a test on Search Engine Journal. He gave three Claude models, from the smallest to the largest, the same 20 products from a Shopify catalogue and asked them to shop. Across 1,980 picks the models often disagreed. On coffee beans, two of the three picked the dark roast with the most reviews, and the largest model picked a medium roast 48 times out of 50. On water bottles, two models leaned towards one bottle and the third picked a different one every single time.
Order mattered too. Moving a product from first to last on the list cut how often it was picked, for all three models. And one bottle showed 447,221 reviews in the catalogue feed when its own product page showed 6,537. The models repeated the big number as a reason to choose it.
His conclusion: Google rankings are a black box a store can still watch. The AI shopping decision layer is a black box with nothing to monitor.
The claim worth testing
The worry is fair and the test is good work. The part I would push back on is “nothing to monitor”. Look at where the three models agreed and where they split.
They split when the listings gave them nothing to decide on. Several bags of coffee, several bottles, all described in roughly the same way. With no attribute that answered the question, each model fell back on its own habit. One trusted the review count. One went for what it called the “safe” choice, 222 times out of 300. One reacted to whatever sat first in the list.
They agreed when a specific fact entered the picture. When the sunscreen request added a budget and a skin type, all three models dropped the top-ranked product at once. The disagreement was never random. It appeared exactly where the catalogue had no detail precise enough to win.
What the spread tells a store
This is my position on AI search in general, and it applies here. The AI does not pick the biggest brand. It picks the one it understands most clearly and can quote most precisely. A listing that says “premium, smooth, all-day comfort” gives a model nothing to quote. A listing that says 750ml, fits a car cup holder, keeps water cold for 24 hours, dishwasher safe, gives it four facts to match against a question.
Take a Shopify store that sells water bottles. If three models pick three different products from your category, none of the listings has an attribute that answers the shopper’s question. That is a list of work. Find the attributes shoppers actually ask about, put them on the listing as specific facts, and make sure the numbers are true. The review-count mismatch in the test shows what happens otherwise. The model quoted a wrong number as a reason to buy. It will quote your right numbers the same way, if they are there.
You can monitor it. Ask the engines yourself.
There is no ranking report for this layer, and I don’t expect one. But a store can do what Manic did. Ask the question a shopper would ask, in several models, write down what each one picked and why, and date the record. Do it again next week. AI answers are regenerated on every request and change week to week, so an undated result is worth nothing. A dated set of results across engines is a trend you can act on.
I learned this on my own product. In August 2026 AeroChat ranked third on Google for “best Shopify chatbot” but wasn’t named in the AI Overview. At the time I put it down to the AI preferring a neutral article over a seller’s own page. A month later I went through the full list of sources behind that answer. One article named AeroChat first, and none of the others mentioned us at all, so the AI didn’t either. Ranking and being named are separate. Finding out which one is broken meant asking the engine and reading its sources, which any store can do.
What to do
- Pull your products the way an agent sees them. Manic checked the catalogue through Shop MCP. Confirm that review counts, prices and attributes match your product pages.
- Rewrite listings in the words shoppers use when they ask, with specific attributes: size, capacity, price, who it is for, what it is not for. Engines quote specific facts and drop marketing language.
- Run your top five shopper questions through at least three models. Record the pick and the stated reason. Date it.
- Repeat monthly, or weekly in a busy season. Where the models split, your listings are not specific enough. Where they agree on a competitor, read why and close that gap.
Source: Search Engine Journal: 3 AI Models Shopped Shopify’s Catalog – Each Picked A Different Winner