New Research: AI Brand Recommendations Are Wildly Inconsistent

The Study

In late 2025, Rand Fishkin and SparkToro conducted one of the most rigorous studies yet on AI recommendation consistency. The setup: 600 volunteers submitted 2,961 prompts across three major AI platforms, covering everything from chef's knives to cancer hospitals to SaaS providers.

The question was straightforward: when you ask AI the same thing multiple times, do you get the same answer?

The short answer: no.

Response lists varied in three dimensions: which brands appeared, what order they appeared in, and how many were listed (anywhere from 2-3 to 10+). Run the same prompt 100 times and you'd get a different answer nearly every time.

Fishkin's conclusion was blunt:

"Any tool that gives you a 'ranking position in AI' is full of baloney."

That's a provocative claim. But the data backs it up — and the implications go deeper than just debunking a metric.

The Pattern Within the Chaos

Here's where it gets interesting. While individual responses were wildly inconsistent, aggregate patterns were remarkably stable.

Consider the headphone study. 142 different people each wrote their own prompt about buying headphones for family travel — all worded differently, averaging just 0.081 semantic similarity between prompts. Despite this extreme variation in how people asked, the brand visibility percentages converged:

• Bose: 55–77% visibility across platforms
• Sony: 55–77% visibility
• Sennheiser: 55–77% visibility
• Apple: 55–77% visibility

The AI platforms consistently drew from the same consideration set. The same brands kept showing up — just not in the same order or combination on any given run.

You're Either In the Set or You're Out

This is the key insight from the research: AI recommendations aren't a ranking ladder. They're a consideration set with a sharp boundary.

You're either a brand the AI consistently includes — showing up in 50%, 70%, 90% of responses — or you're one that rarely appears. The middle ground is thin.

And the gap is widening. Separate research from Search Engine Land found that in the digital marketing space, the top 10 experts increased their share of AI citability from 30.9% to 59.5% in just two months — a 92% increase in concentration. The Herfindahl-Hirschman Index (a standard measure of market concentration) rose 293% in the same period.

The brands that are "in" are getting more dominant. The brands that are "out" are falling further behind.

What Gets You Into the Consideration Set

If ranking position is meaningless but presence is everything, the natural question is: what determines whether you're in?

A follow-up analysis by Search Engine Land identified what they call the "corroboration threshold": approximately 2-3 independent, high-authority sources confirming the same claim about a brand triggers consistent AI inclusion.

They also described a three-graph model for how AI retrieves information:

1. Entity graph — Knowledge graphs with verified connections (Wikipedia, Wikidata, Google Knowledge Panel)
2. Document graph — Indexed URLs scored for relevance and authority
3. Concept graph — The LLM's own parametric knowledge from training data

Brands present across all three achieve significantly higher consistency than those in only one or two. It's the combination of structured entity data, crawlable web presence, and established authority that pushes you over the threshold.

What This Means for Your Strategy

The SparkToro research doesn't say AI visibility doesn't matter — it says it matters differently than we assumed. Position is noise. Presence is signal.

And presence is driven by things that are largely within your control: structured data that helps AI understand who you are, citation signals that establish credibility, content that's extractable and clear, and crawler access that lets AI systems actually reach your pages.

The Bottom Line

This research is a wake-up call, but not a discouraging one. The inconsistency in AI recommendations isn't random — it's telling us that superficial tracking metrics don't work. What does work is building the kind of technical foundation and genuine authority that gets you into the consideration set in the first place.

The brands showing up in 70%+ of AI responses didn't get there by accident. They got there because their content is structured, their authority is corroborated, and their pages are machine-readable.

That's the game now. Not chasing a rank that doesn't exist — but understanding the foundations that determine whether AI considers you at all.

---

Sources: SparkToro's AI Consistency Research and Search Engine Land's follow-up analysis.