When does AI search the web? When buyers say your name.

At a glance

Two ways to ask the same question

A buyer can ask an AI assistant about software in two ways: by naming the product, or by describing the need. To the buyer they feel like the same question. We wanted to know whether the assistant treats them the same way: does it look things up on the web, or does it answer from what it learned in training?

The difference matters because the two sources can disagree. The web shows what a product does today. The assistant's memory reflects what was written about a product before the model was trained, which can be months or years old, and which covers famous products far more than small ones.

Naming a product makes the assistant check the web

When the question named a product, the assistant searched the web before answering 78% of the time. When the same question only described the need, it searched 7% of the time.

The gap is 70 percentage points, and the 95% interval runs from 58 percentage points to 82 percentage points, so this is not a close call. In everyday terms: a buyer who names a product usually gets an answer checked against the web. A buyer who only describes the need usually gets an answer from memory.

The less famous the product, the more it checks

The assistant did not treat every name the same way. For well-known products it searched about half the time, 53%. It seemed confident it already knew them: asked by name about Trello, Asana, Wrike or Microsoft Project, it never searched in any of its tries. For lesser-known products it searched almost every time, 99%.

Naming a lesser-known product raised the search rate 48 percentage points more than naming a famous one, with a 95% interval from 23% to 71%. That is sensible behavior: when the assistant is unsure, it checks. The consequence is easy to miss. For a famous product, the answer often comes from memory even when the buyer names it.

When only the need is described, memory decides who is named

Questions that only describe the need are where many buying journeys start, because early on a buyer does not know the names yet. The assistant answered more than nine in ten of them from memory.

Every question that described the need was built from one real product whose own description states the capability asked about. So for each answer given from memory we checked one thing: did the assistant bring up the product the question came from?

It brought up the product in 64% of answers when the product was well known, and in 31% when it was lesser known. Other products it named may fit just as well. The point is who gets remembered, and the pattern was close to all or nothing: the assistant either brought a product up almost every time or never did.

Something we did not expect

A few questions that only described the need did make the assistant search, and most of them asked about capabilities that sound new: AI agents that sort customer feedback, test cases written automatically, creating tasks from WhatsApp. The assistant seems to check the web when a topic feels recent, whether or not a product is named. We did not set out to test this, so we treat it as a lead for the next round rather than a finding.

What it means for software vendors

When buyers already know a vendor's name, the assistant looks the product up, and what it finds shapes the answer. Clear pages that say plainly what a product does are what it reads. When buyers name a product, the AI does the research. When they do not, it does the remembering.

When buyers name a product, the AI does the research. When they do not, it does the remembering.

When buyers do not know a vendor's name yet, the assistant answers from memory, and a page the vendor publishes today does not change that answer directly. What counts is how often, and in how many places, the product has been written about, because that is what the model learned from. For a lesser-known product this second problem is the harder one, and it is where the buyer's search begins.

What it means for buyers

A buyer who has a product in mind and asks about it by name is much more likely to get an answer checked against the web. The list that comes back from a question that only describes the need is a starting point drawn from memory, and it leans toward the names the assistant already knows.

Methodology