Three Claude Models, Same 20 Products, 1,980 Picks: No Agreement on What to Buy

A test of 1,980 product picks found that three Claude models, given identical Shopify catalogs and identical instructions, never once agreed on the top product to buy across six searches. The experiment, run by Slobodan Manic and originally published on No Hacks, probed the layer beneath agentic commerce: the language model that chooses one product off a list when a shopper's agent does the buying.

The setup relied on Shop MCP, which Shopify announced on Oct. 13, 2026. In Shopify's words, it "lets your agent or app search Shopify Catalog, build carts, and complete Shop Pay checkout through a single MCP connection." It covers over a billion products and requires no API key. Any company building a shopping agent can search the catalog and get back a product list.

But a list is not a purchase. Something must pick one product, and when an agent runs on a language model, the model picks. The agent company's instructions sit on top, and they're private. So the test isolated the layer underneath.

How the test worked

The tester searched Shopify's catalog once per query, as an anonymous U.S. buyer, and saved the top 20 results. Eight skincare searches were fetched on Oct. 15. Six more — backpack, coffee beans, running shoes, water bottle, wireless earbuds, yoga mat — came on Oct. 24. Two searches ("scented candle," "dog food") were dropped because a handful of stores filled all 20 slots.

Each model saw the title, store, price, star rating, review count, a one-line summary and three features, numbered 1 to 20. The instruction told each model: "You are a shopping agent... Choose the one product you would buy for them. Reply with JSON only."

The models were Claude Haiku 4.5, Sonnet 5 and Opus 5, run through Claude Code with default settings, no tools, no web access, no memory between calls. Three sets of runs totaled 1,980 picks:

• Six product searches, search words only: 900 picks (50 per model per search).
• One sunscreen search asked two ways — with and without a budget and skin type: 600 picks.
• Eight skincare searches shown in catalog order and reversed: 480 picks.

What did the models pick?

On the six product searches, the three models' most-picked product matched in none. Haiku and Sonnet agreed on three (backpack, coffee beans, water bottle), Sonnet and Opus on two (wireless earbuds, yoga mat), Haiku and Opus on one (running shoes).

Each model was consistent with itself, and inconsistent with the others. Sonnet's top pick averaged 94% of its picks across the six searches, Opus's 89%, Haiku's 67%. On any one model, one product gets nearly every pick. Change the model and a different product does. On three of six searches the different product came from a different store: on water bottle, Haiku and Sonnet picked nobl's bottle while Opus always picked Owala's; on yoga mat, Haiku picked POPFLEX and the others Liforme; on running shoes, two models chose a Brooks shoe from Fit2Run while Sonnet chose an adidas shoe from Commonwealth Running Co.

The catalog's ranking barely mattered. Haiku picked the catalog's first-ranked product 8% of the time, Sonnet never, Opus 20%.

Takeaway: which store gets the order depends partly on which model the shopper's agent runs on.

Reviews — even unverifiable ones — drove choices

On backpack, coffee, water bottle and yoga mat, Haiku and Sonnet favored the most-reviewed product, and said so. On coffee, Haiku explained: "EASE Dark Roast has the highest review count (24,398 reviews) with a 4.9/5 rating."

The water bottle result exposed a data problem. The catalog listed nobl's bottle at 5.0 stars from 447,221 reviews. nobl's own product page showed 5.0 from 6,537 on Oct. 24 — roughly 68 times fewer. The catalog's record for the Blush color said 4.2 from 1,674. The 447,221 figure appears nowhere on nobl's pages, yet Haiku cited it in all 45 of its picks and Sonnet in 33 of 39. Whatever the catalog says is what the model reasons from, and the model cannot check it.

Opus played it safe, out loud

Opus 5 called its choice "safe" or "safest" in 222 of 300 picks. Haiku did that once; Sonnet 11 times. "With no stated preferences, a medium roast is the safest crowd-pleaser," Opus said on coffee. It picked the everyday backpack over the roll-top and the medium roast over the dark — never the most-reviewed product. On sunscreen, 17 of its 100 reasons ruled out a "tint, powder, or stick" format the shopper never asked about.

List position changed picks, invisibly

Reversing the list so the catalog's top product sat last changed outcomes on 2 of 8 skincare searches. Across all eight, out of 80 per version, Haiku's picks of the top product fell from 28 to 11, Sonnet's from 15 to 11, Opus's from 24 to 10. On mineral sunscreen, all three models nearly stopped picking it. In 480 reasons, not one mentioned list position — a shopper reading the agent's explanation would never know it played a part.

Requests with constraints moved every model

Adding "under $25, for oily skin" to "mineral sunscreen SPF 50" dropped the $48 top-ranked sunscreen to zero picks from all three models, in 298 of 300 picks landing under $25. Sonnet picked ATTITUDE's $22.95 sunscreen (4.7 stars, 292 reviews) all 100 times. Opus split between ATTITUDE (67) and Sun Bum's $17.49 lotion (33) — the lowest-rated of the six under $25, but the only listing saying "without clogging pores." Opus never picked the best-rated option, Badger's 4.9-star sunscreen.

The models also repeat store claims verbatim. "Clinically proven" appeared in 63 reasons, and all 22 "11x faster" retinal claims came straight from listings. A claim in a listing is now a claim an agent makes on the store's behalf.

What stores can actually do

The tester's advice for Shopify merchants: check what the catalog shows agents about your products; write listings in the words customers use for their needs; keep claims true, since models repeat them unchecked; and test your category by running your top search against competitors across a few models.

The decision layer between shopper and store is a black box — unlike Google rankings, which stores can at least monitor. With agents, every shopper gets a private version of the decision inside an app the store never sees. The test covers only three Anthropic models, one bare instruction, fixed lists from two days, and nothing actually bought. Watch whether other vendors' models show the same divergent defaults, and whether agent builders expose any window into the choice layer.