When AI finds the current fact, it uses it.

At a glance

The question every vendor asks

Software changes faster than AI models are trained. Prices move, plans are renamed, features launch. So a vendor who fixes a fact on their own site wants to know: when a buyer asks an AI assistant about it, will the assistant use what the site says today, or repeat what it remembers from before?

Our companion study, When does AI search the web?, found that naming a product makes the assistant search most of the time. This study asks the next question: once it has searched, whose version wins?

Whose version wins

Every one of the 39 questions was a real disagreement: the assistant's memory gave one answer, consistently, and the vendor's own page gave another. With web search available, the assistant searched on every question.

In 97% of answers the search surfaced the current fact. In every one of those answers, 100%, the assistant went with the web. Not once did it keep its memory's version, and not once did it split the difference.

Asked without search, the assistant gave the old, larger storage figure every time. The vendor's page gives the smaller one, and with search, every answer did too. In 6% of answers, like this one, the old value survived only as history. All of those were about well-known products, where they appeared in 15% of answers.

Harder questions, same answer

Short questions that ask for one number make the fact easy to find, so we asked the same things the way buyers do: open-ended, with context, and with no request for a one-line answer. The assistant still went with the web in 97% of answers, blending the two in the rest.

We also put the disputed facts inside comparisons, where the fact decides which product wins. The assistant followed the web in 89% of answers and blended in 11%, and it picked the right product every time.

When the fact changed recently, the old version usually wins

The last bar in that chart is the one that matters most. We compared the pages we captured from vendors over the summer with their live pages today, and found 10 facts that had changed: prices, plan limits and what the cheapest plan includes. These are the facts a vendor has just fixed.

Only 36% of answers gave the current fact. 32% gave the value the assistant remembered, 28% gave an older value it found on the web, and 4% blended the two. In total, 64% of answers were out of date or mixed.

The reason was the search results, not stubbornness. 60% of the answers cited at least one source that still carried the old value: comparison articles, review sites and, in one case, the vendor's own help documentation.

When the vendor's live pricing page was among the sources, 75% of answers were current. When it was not, none were. We noticed this split while reading the answers rather than planning it in advance, so we treat it as a strong lead rather than a tested result. It matches the pilot: the one fact the search never surfaced there was a pricing change only weeks old, and every answer repeated the old plan.

Where memory still tips the answer

A single fact has one right answer. A choice between two products is different, so we tested that too. We gave the assistant a buyer's requirement and two search results that described two real products equally well, one famous and one lesser known, and asked it to recommend one. Then we swapped the names and ran it again, with every version shown in both orders.

The well-known name won 71% of the time. Being listed first helped any product, but the famous name still won 55% of the time from second place. In 8% of answers the assistant justified its pick with a reputation the search results never mentioned, calling one product the long-standing industry standard when both descriptions said the same thing.

So memory does not overrule a clear, current fact that the search has found. It fills the gaps when the web is out of date, and it tips close calls toward the names it already knows.

What it means for software vendors

The web wins when it is current. When a buyer names your product and the assistant finds your own page, what that page says today is what the buyer hears, even when the assistant remembers something different.

After a change, that is the hard part. For a while, most of the web still describes your old plan: reviews, comparison articles, blog posts and sometimes your own help pages. Unless the assistant reaches your live page, it repeats the old version. So publish the change where an assistant will find it, state it plainly on your pricing page, update your own documentation, and get the old pages about you corrected.

And when a buyer's choice between you and a famous competitor is close, the famous name starts with an edge. Your evidence has to be clearly better, not just as good.

What it means for buyers

An answer checked against the web is only as current as the pages it found. For anything that changes, such as prices, plan limits and which plan includes a feature, confirm the answer on the vendor's own pricing page before you rely on it. And when an assistant prefers one product over another, ask it to point to the evidence.

Limitations

These are pilots on one assistant, through its developer API rather than the consumer chat app. The recently changed facts are few, 10 in all, so their ranges are wide, and the split by whether the vendor's page was cited was found after the fact. The name comparison used search results we wrote, not live search.

No answer followed memory alone when the current fact was found, but pilots of this size cannot rule out that it sometimes would. The full study will test more changed facts and other leading assistants.

Method

We wrote our hypotheses down before each run. The pilot predicted that a meaningful share of answers would follow memory, which was not supported. The extension predicted that open-ended and comparison questions would let memory in more often, which was not supported, and that recently changed facts would go wrong more often, which was.

We screened 140 specific questions about 27 products in the Juukbox project management catalog. The truth for each was the vendor's own current page, fetched and quoted. We asked each question without web search to record the assistant's memory, and kept the 39 where memory consistently contradicted the vendor's page, after checking each by hand. Each went to gemini-3.8-flash with the Google Search tool available and never forced: 5 times as a short question and 3 times as an open-ended one. 9 of the disagreements were also asked inside comparisons.

For recently changed facts we compared our summer captures of vendor pages with the live pages, kept 10 changes confirmed on the live page, and asked each 5 times. For the name swap we wrote two equally strong descriptions for each of 20 pairs of products and showed every version with the names swapped and in both orders, 240 answers in all. We saved every source the assistant cited and labelled every answer by hand. Intervals are 95% bootstrap intervals over questions or pairs.