The most dangerous answer an AI search tool can give isn't "I don't know". It's a fluent description of a company that barely exists. We built a benchmark around exactly that case, and it re-ranked every tool we tested.
We built a Slack bot for an investment client that backs early-stage startups. The bot answers questions about companies in the portfolio, and when a query falls outside what it already knows, it falls back to live web search.
During testing, one search tool met an unfamiliar company and did not say it was unfamiliar. It generated a plausible-sounding description instead: a business category, a vague value proposition, all of it invented, none of it flagged. If you didn't already know the company had almost no public presence, the answer read as credible.
That moved our evaluation away from raw accuracy and toward a different question: which tools admit what they don't know, and which ones fill the gap with fiction?
Standard comparisons measure retrieval speed and accuracy on well-documented entities, which is where every tool performs about the same and almost nothing fails. The expensive failure lives elsewhere, with sparse-footprint entities: emerging startups, companies with thin web presence, vendors with no real digital identity. Performance diverges sharply there, and so do the consequences. A fluent but false description ranks worse than plain uncertainty, because it looks legitimate.
So we tested the gap rather than the documented case.
Four providers: Gemini, using native LLM web search, plus three dedicated search APIs in Perplexity, Exa and Parallel.ai. Each received identical structured prompts for 12 real companies.
One instruction applied uniformly across all of them: do not invent details, and if you cannot find enough information, say exactly UNKNOWN. That single line turns a retrieval benchmark into an honesty test.
Ten of the companies were ordinary and findable. Two were chosen to break things:
Ground truth was researched independently before any tool ran, so nothing was contaminated by what the tools returned. All 192 answers, 12 companies across 4 prompts across 4 providers, were then scored blind on whether the tool found anything, cited anything, and got it right without fabricating.
| Provider | Coverage | Right entity, no fabrication |
Cost per query |
Latency |
|---|---|---|---|---|
| Exa | 97.9% | 81.2% | $0.005 | 1.9s |
| Parallel.ai | 95.8% | 70.8% | $0.010 | 38.5s |
| Perplexity | 87.5% | 68.8% | $0.015 | 7.6s |
| Gemini | 85.4% | 52.1% | $0.037 | 7.3s |
The default option, native LLM search, was the least trustworthy at 52.1%, barely above a coin flip, and also the most expensive. Coverage was never the problem. Every tool returned answers. The problem was that most of those answers didn't survive scrutiny. The honesty gap, 52% to 81%, dwarfed the coverage gap of 85% to 98%.
The question "why does this company matter" underperformed for every single provider, landing between 42% and 67%, despite being the highest-value field for an investment team.
The two adversarial cases separated the field decisively.
The unfindable company. Exa, Perplexity and Parallel.ai all correctly answered UNKNOWN on the structured questions. Exa went further on the open-ended prompt, naming similar companies and confirming that none of them matched. Gemini confidently described an unrelated healthcare-AI firm, assembling a fabrication out of facts that were real but irrelevant.
The name collision. Eleven of twelve answers across all providers described the wrong business. Exa alone identified both candidates, distinguished between them, and surfaced the startup's actual domain.
The best performer won on honest uncertainty and citation-backed claims rather than confident assertion. It was also the cheapest, at one-seventh of Gemini's cost, and the fastest. Accuracy and economics pointed the same way, which does not always happen.
The test decided the build. The enrichment path runs on Exa's answer API with cost monitoring from day one, and it now backs both bulk brief completion and the Slack bot's live fallback. Perplexity stayed as a documented backup rather than a second production integration.
One caveat worth stating plainly: 12 companies and 192 scored answers give directional signal, not statistical power. That was the point. We were looking for behavioural differences at the boundary of what these tools know, and the difference was large enough that more samples would only have refined a conclusion we already had.
For any AI search implementation where a confident error damages a decision, support bots, research assistants, anything customer-facing or decision-facing, benchmark past the easy cases.
A week of testing against 12 companies changed which tool reached production. Before wiring search into anything customer-facing, that's an investment worth making.