When AI searches, the web beats its memory.
We found 39 facts about software products where an AI assistant's memory disagreed with the vendor's own page, then let it search. Whenever the search found the current fact, the assistant used it: 100% of the time. Memory only won when the web itself was out of date.
At a glance
Key findings
- When the search found the current fact, the assistant used it every time. 100% of those answers followed the web, 0% followed memory and 0% blended the two.
- Memory still leaves a trace. In 6% of those answers the assistant gave the current fact and mentioned the old one as history, all of them about well-known products.
- Memory wins when the web is stale. The one fact the search never surfaced was a pricing change only weeks old: the pages it found still described the old plan, and so did every answer.
The question every vendor asks
Software changes faster than AI models are trained. Prices move, plans are renamed, features launch. So a vendor who fixes a fact on their own site wants to know: when a buyer asks an AI assistant about it, will the assistant use what the site says today, or repeat what it remembers from before?
Our companion study, When does AI search the web?, found that naming a product makes the assistant search most of the time. This study asks the next question: once it has searched, whose version wins?
When it finds the current fact, it uses it
Every one of the 39 questions was a real disagreement: the assistant's memory gave one answer, consistently, and the vendor's own page gave another. With web search available, the assistant searched on every question.
In 97% of answers the search surfaced the current fact. In every one of those answers, 100%, the assistant went with the web. Not once did it keep its memory's version, and not once did it split the difference.
What the assistant did when search found the current fact
Share of answers where the search surfaced a fact that contradicted the assistant's memory.
How much storage (in MB) does ClickUp's Free Forever plan include?
Searched the web before answering
ClickUp's Free Forever plan includes 60 MB of workspace storage (formerly 100 MB), which is primarily consumed by file attachments, doc images, and Clip recordings.
Asked without search, the assistant gave the old, larger storage figure every time. The vendor's page gives the smaller one, and with search, every answer did too.
Memory leaves a trace
The ClickUp answer above shows the one way memory did show through. In 6% of answers the assistant gave the current fact and mentioned the old value as history: formerly this, now that. It never let the old value decide the answer.
Every one of those traces was about a well-known product, where they appeared in 15% of answers. For products in the middle and lesser known, there were none. The assistant has simply read more about famous products, including their past.
Memory wins when the web is out of date
One question broke the pattern, and it is the most useful result in the study. Basecamp's own pricing page now says there are no per-user fees. The assistant's memory said it charges per user. The search did not surface the new page. It found other sites still describing the old per-user plan, and every answer repeated it.
Does Basecamp charge per user?
Searched the web before answering
Basecamp offers both per-user and flat-rate pricing models. For smaller teams and startups, it provides a plan billed at $15 per user per month.
This is not memory overriding the web. The web the assistant found agreed with its memory, because those pages had not caught up with a change made only weeks earlier. The assistant followed the web, and the web was wrong.
What it means for software vendors
The web wins. When a buyer names your product and the assistant searches, what the web says about you today is what the buyer hears, even when the assistant remembers something different. That is good news: a correct fact on the web gets used.
It also means the risk has moved. The problem is less the assistant's memory than the old pages about you that are still out there: reviews, comparison articles and blog posts describing last year's plans and prices. After a change, updating your own site is the first step, not the last. Find the pages that still describe the old version and get them corrected.
What it means for buyers
An answer checked against the web is only as current as the pages it found. For anything that changes, such as prices, plan limits and which plan includes a feature, confirm the answer on the vendor's own pricing page before you rely on it.
Limitations
This is a pilot on one assistant, through its developer API rather than the consumer chat app. The questions were short and specific, named the product, and asked for a single answer, which makes searching likely and the fact easy to find. Longer, open questions may behave differently.
No answer followed memory when the web fact was found, but a pilot of this size cannot rule out that it sometimes would: the most we can say is that it happens rarely. The full study will look for facts vendors changed recently, where the web is most likely to be split, and test open questions too.
Method
We wrote our hypotheses down before running anything, including that a meaningful share of answers would follow memory. That hypothesis was not supported, and we report it as such.
We screened 140 specific questions about 27 products in the Juukbox project management catalog: features, prices, plan limits and packaging. The truth for each was the vendor's own current page, fetched and quoted. We asked each question without web search to record the assistant's memory, and kept the 39 where memory consistently contradicted the vendor's page. We checked every kept disagreement by hand and removed questions where the page itself was ambiguous.
Each kept question went to gemini-3.8-flash 5 times with the Google Search tool available and never forced, with a request to end on a one-line answer. We saved every source the assistant cited. An answer follows the web when it gives the vendor's current fact, follows memory when it gives the remembered one, and blends when it gives both as current or hedges between them. Every answer was also labelled by hand. Intervals are 95% bootstrap intervals over questions.
Start your own search.
Describe what you need and Juukbox will help you compare products, with the reasons behind every match.