AI search
Before matching a single product to a buyer, we tested whether AI answers can be trusted.
OpenDecision Research · · 6 min read
In our small sample, web search made no measurable difference to how often AI answers about software were right. Asked the identical question again, the AI sometimes changed its answer.
At a glance
Key findings
- Web search made no measurable difference in our sample. Checked against an expert, answers with web search agreed 84% of the time and answers from memory alone 83% to 85%. That gap is within the noise of 42 answers labeled by one person, so it shows no improvement here, not that search never helps. Memory was more than ten times cheaper per answer and, one question at a time, about twice as fast.
- The AI does not fully agree with itself. Asked the identical question again, it gave the same answer 93% of the time, so any accuracy figure for AI answers needs that ceiling beside it.
- Memory leans toward yes and almost never says it does not know. When memory disagreed with a web-checked answer it tended to credit a product with a feature, and it almost never answered that it did not know.
The question we had to answer first
Juukbox tells a buyer which products fit their requirements. That only works if the answer to a simple question is right: does this product do this thing? Before building on AI answers, we needed to know how often they are right, when looking things up on the web helps, and where the answers go wrong.
So we tested it, at a scale that cost real money: thousands of model calls and thousands of live web searches in the main comparison alone, across different ways of asking, several models and many prompt designs. We logged the cost of every run.
Web search made no measurable difference in our sample
We asked the same capability questions with web search switched on and switched off, and checked every answer against an expert's labels. Answers that searched the web agreed with the expert 84% of the time. Answers from memory alone agreed 83% to 85% of the time, depending on how the questions were grouped. Those figures sit within two points of each other on 42 answers labeled by one person, well inside the noise of a sample that small, so we read them as no measurable improvement from search in this sample, not as proof that search never makes answers more correct.
Memory was more than ten times cheaper per answer and, asked one question at a time, about twice as fast. This does not mean search is useless: it found things memory could not, as the next sections show. It means search is not a truth machine. It retrieves pages; it does not judge them better.
The AI does not fully agree with itself
We ran the identical set of questions twice, with web search, and compared the two runs. They gave the same answer 93% of the time. Across our tests that ceiling sat between 91% and 95%.
That matters for anyone who reports on AI answers. If the assistant changes its own answer on 7% of identical reruns, an accuracy claim without a repeatability figure beside it is not a measurement. We now report both.
Memory leans toward yes
When the memory-only answer disagreed with the web-checked one, it tended to disagree in one direction: it credited the product with the feature. And even when the prompt told it to say so, it almost never answered that it did not know.
For a buyer this is the real risk. An assistant answering from memory is more likely to tell you a product has what you need than to admit it is unsure.
What memory cannot see, and what nobody judges well
Search earned its keep on one kind of question: it found third-party add-ons that extend a product, which memory missed. Memory and search also disagreed most on features that move between plans and on certifications, the facts that change most often.
One kind of answer was hard for every method: "it does this, but only partly". With and without search, the assistant got 23% of those right. Finding the evidence is not the same as weighing it.
An order of fit needs enough requirements
Putting products in order of fit on a single requirement produced mostly ties: with one requirement, 67% of products tied for the top score, so the order was decided by the tie-break rather than by fit. In our tests ties broke alphabetically, so an order built on one or two criteria was closer to alphabetical than to a recommendation.
What we changed because of it
We do not rely on one method. We check the web where memory is weakest, we measure how often answers repeat before we trust them, and we treat "not known" as an answer rather than a gap to fill with a guess. And we built Juukbox to ask buyers for enough requirements that the order reflects fit.
These tests also raised the questions we are now studying in public, starting with when an AI assistant searches the web at all.
Limitations
The expert labels are thin: 42 answers across a few products, labeled by one person, all in project management software. Most tests used one model family through its developer API, which is not the consumer chat app. Costs are list prices computed from what each run used, not invoices.
Method
Each test asked whether a named product meets a named requirement, taken from real buyer requirements, and recorded a verdict. Arms compared the Google Search tool switched on against memory alone, one question per call against questions grouped together, and several models, prompt designs and reasoning settings. The main comparison ran on gemini-3.7-flash.
Answers were scored against 42 answers labeled by an expert, with partial credit for a near miss, and against an identical rerun of the same run. Every call, verdict and cost was saved, and every figure here can be regenerated from those records.