In 2025 Shopify added a question to its performance and peer reviews asking how each person used AI. It followed CEO Tobi Lütke’s April memo that made “reflexive AI usage” a baseline expectation, and a rule that teams explain why AI couldn’t do a job before asking for more headcount. On the Sources podcast published on 28 October 2026, Lütke said the answers to that question “didn’t matter”, and he isn’t sure it is still in the review at all.

He says the question was a signal, and that is all it was

Lütke’s own account is worth reading closely. The peer question asked “how reflexively a colleague shows up with an AI prototype”. He admitted that “sometimes you play a little bit of tricks there”. What the question did, in his words, was “give a company a reputation to live up to”. Today he says he doesn’t have to enforce these things, because the behaviour happens on its own.

So test it the way I test any claim. Was there a real need? Yes. A large company wanted its people to pick up a new tool quickly, and a line in the review form is a cheap way to say “this matters here”. What changed? People started showing up with AI in hand. What didn’t change? The question could never tell Shopify whether the work got better. Lütke’s September comment about “slop grenades”, unread AI output passed from one colleague to another, shows the gap. Adoption can go to 100% while quality goes nowhere, and a usage rating would never see it.

Counting who touched the tool measures the toolset

My order for anything AI is mindset, skillset, toolset. A peer rating on AI use sits at the bottom of that list. It tells you who opened the tool. It cannot tell you whether that person thought about the problem first, whether they checked what came out, or whether what they handed over was any good. The tool was never the skill.

I have said this on stage and I will keep saying it. The technology was the easy part. Changing how people think and work was the hard part, even for a tech team. Shopify’s question pulled on the easy part. The memo set the expectation, which is useful. The rating then tried to measure the expectation instead of the result.

What this looks like in a smaller business

Take a marketing team of three in a retail shop. The owner reads the Shopify story and tells everyone to use AI. A month later every one of them does. Blog drafts, product descriptions, ad copy, all AI-assisted. Grade usage and everyone gets full marks. The owner has learned nothing about the business.

The questions that would have told the owner something are different. Did the product pages bring in more enquiries? Did the drafts need less rework than before? Which pieces got the shop named in an AI answer? Every piece of content now has two readers, the person who reads it and the machine that decides whether to quote it, and the only honest grade comes from results with both.

Or take a clinic where the front desk uses AI to draft replies to patients. Usage is at 100% in a week. What matters is whether reply times fell, whether fewer bookings went missing, and how many drafts the manager had to step in and fix. That last one is the judgement question, and no usage rating asks it.

What I measure in my own work

I use Claude for almost everything, and nobody grades me on how often I open it. In my content work Claude did the building, the counting and the checking, and I made the decisions. The measure was never the number of prompts. The measure was whether the brands got named in AI answers, dated, so the claim can be checked. Within three weeks of launch, Google’s AI Overview was citing AeroChat alongside Tidio and Intercom. That is an outcome, and it would have looked the same whether it took ten prompts or a thousand.

The second thing I watch is where the output had to be corrected. Brief, produce, check, then fix the brief. If the same correction keeps coming up in the same place, that points at the skillset, and no change of tool will fix it. If the correction stops coming up, the person has learned something, which is what a review was supposed to find out in the first place.

What a business should do

Set the expectation once, the way Lütke’s memo did, and say it plainly. Then take the usage question out of the review. In its place:

  • Pick two or three outcomes per role that the work exists to produce. Reply time, rework rate, enquiries, bookings caught, being named in AI answers. Pick the ones you can count.
  • Add one judgement measure. How often did this person’s AI-assisted output need correcting, and what kind of correction was it?
  • When the same correction keeps appearing, fix the brief or train the person. Leave the tool alone.
  • Review the outcomes every month, with dates, and compare like for like.

Grade the work. The tool will take care of itself.

Source: Search Engine Journal: Shopify CEO: Peer Ratings On AI Use ‘Didn’t Matter’