Artificial Analysis has released the Search Index, a benchmark suite designed to evaluate web search APIs for autonomous AI agents across retrieval quality, query latency, and end-to-end task economics.
The initial evaluation tests seven dedicated search providers: Parallel, Exa, Firecrawl, You.com, Tavily, Keenable, and Brave.
Benchmark Setup and Evaluation Methodology
To isolate search API performance from model variance, the evaluation executes all tests with GPT-5.6 Luna inside Stirrup, an open-source benchmarking framework for tool-using agents. Each agent run allows up to 25 consecutive search and retrieval steps per task.
The Search Index evaluates providers across 1,700 structured tasks spanning three distinct test distributions:
- DeepSearchQA (900 tasks): Complex multi-query research questions requiring iterative information synthesis across disparate sources.
- BrowseComp Subset (200 tasks): Hard-to-find factual queries demanding multi-step site navigation and nested link traversal.
- AA-Omniscience (600 tasks): Deep-domain factual recall across six technical and scientific subject areas.
- Tool-Free Baseline: Direct model answering without search tools to measure net accuracy lift from web grounding.
Search Precision Lowers Total Inference Cost
A primary finding of the benchmark is that higher per-call search quality reduces total agent execution cost by decreasing downstream LLM token consumption. When search APIs return concise, highly relevant context, models require fewer reasoning steps, fewer clarifying searches, and fewer prompt tokens to reach an answer.
For instance, using Parallel Search Advanced reduced token consumption by more than 40% compared to the Basic version. Even though the advanced API charges a higher fee per query, the reduction in LLM inference tokens brought total cost per task down from $0.11 to $0.084.
Query Latency vs. End-to-End Task Duration
The benchmark highlights that raw API response times do not correlate directly with total task speed. In testing, faster endpoints with lower precision increased the overall number of reasoning loops:
- Parallel Turbo: 0.51 seconds per individual query, but a lower quality score of 67 forced additional search attempts, resulting in an end-to-end task duration equivalent to slower endpoints.
- Parallel Basic: 1.03 seconds per individual query, but a quality score of 73 yielded faster resolution in fewer tool cycles.
Artificial Analysis reported that Parallel and Firecrawl established the current efficiency frontier for combined cost and accuracy across multi-step agent workloads.
Sources
- New benchmark ranks search APIs for AI agents on quality, cost, and speed (The Decoder)
- Search API Benchmark Methodology (Artificial Analysis)
