DeepSearchQA Benchmark Methodology and Results

A benchmark that checks if an AI can find one correct answer tells a team almost nothing about whether it can finish a real research task. DeepSearchQA was built to close that gap, and its methodology shows why exhaustive, verifiable answer sets are still the hard problem in deep research agents.

The evaluation gap DeepSearchQA was built to close

Factuality benchmarks operate on a simple premise: ask a question with a single correct answer, check whether the model returns it, and move on. That design works well for questions like "What is the capital of France?" but it fails for the class of task real research agents actually face. An analyst does not ask for one fact. An analyst asks for every entity that meets a set of interdependent conditions, and the value of the answer lies in its completeness. A list that omits three qualifying companies is wrong in precisely the way a benchmark that accepts any single correct answer cannot detect.

The researchers behind DeepSearchQA give this gap a name. They call it the "Comprehensiveness Gap": the disparity between benchmarks that reward finding one correct answer and the real-world need for exhaustive, multi-faceted research. Think of a relocation task that filters cities by house price, then green space, then employment, then clean-air-zone policy. Each step depends on the last, and no single-answer benchmark can surface whether an agent handled all four filters correctly or only three. The benchmark that rewards the first correct city it finds will score a partial answer as a complete success.

The DeepSearchQA paper, published in January 2026 by Nikita Gupta, Riju Chatterjee, Lukas Haas, Connie Tao, Andrew Wang, Chang Liu, and colleagues at Google DeepMind, Google Search, Kaggle, and Google Research, frames this gap as the central motivation for the benchmark's design. The argument is straightforward: if evaluation cannot distinguish between an agent that found one right answer and an agent that found every right answer, then evaluation cannot tell builders which of their systems is actually useful. DeepSearchQA exists to make that distinction measurable.

What DeepSearchQA actually asks an agent to do

DeepSearchQA's 900 hand-crafted prompts across 17 fields are structured as causal chains. Each step depends on the successful completion of the previous one, so an error at step one propagates through the entire answer. The 17 fields include politics and government, finance and economics, science, health, history, geography, and media, among others. That breadth is deliberate: a benchmark spanning this many domains prevents a system from optimizing narrowly for one type of query.

Two answer types appear in the benchmark. With set-answer tasks, you have to return every qualifying entity and leave out every non-qualifying one. The set-answer tasks dominate: 580 of the 896 evaluated questions require exhaustive enumeration. This is the design choice that separates DeepSearchQA from nearly everything that came before it.

The task roles drawn from the paper make the dependency-chain structure concrete. The Epidemiologist task asks an agent to identify U.S. states that satisfy specified CDC death-rate conditions in 2019 and 2020. The Safety Auditor must identify U.S. states with NHTSA fatality rates below a threshold in two or more consecutive years between 2000 and 2005. The Video Gamer filters gaming hardware by sales threshold, then by CPU type, then by parent-company asset size drawn from official financial reports. The Demographer intersects Statistics Canada population-center counts, GDP growth, and Aboriginal identity population figures for Canadian provinces in 2016. In each case, the agent cannot answer the final question without first retrieving the intermediate data, and that data cannot be retrieved without first identifying the correct source. Ground truth for each prompt went through a three-phase verification protocol: three independent reviewers researched solutions without access to the curator's answer, cross-referenced their findings, and resolved conflicts, with ambiguous prompts removed. The anchoring reduces drift in the correct answer over time, though web pages can still change or disappear.

The three capabilities the benchmark is designed to stress-test

DeepSearchQA explicitly targets three capabilities that prior benchmarks left largely unmeasured. The first is systematic collation of fragmented information from disparate sources. The second is de-duplication and entity resolution to ensure precision. The third is reasoning about when to stop searching in an open-ended space. Each maps to a distinct failure mode. Stopping-criteria failure produces agents that either stop too early, confusing "I have not found it yet" with "it does not exist," or cast an overly wide net of low-confidence answers to artificially boost recall.

Stopping criteria is the hardest of the three. How does it know when it has found every qualifying state, every company with a P/E ratio under 20? The benchmark is designed so that guessing wrong on this question directly and visibly depresses the score. The paper's analysis identifies premature stopping and hedging as the two most common failure modes observed across evaluated architectures.

The causal-chain structure amplifies all three failure modes. A missed entity at step two cannot be recovered at step four. Collation errors compound rather than average out, so one early failure cascades through the entire answer set. That is not a flaw in the benchmark. It is the property of real research the benchmark exists to measure.

How DeepSearchQA scores performance: F1, Fully Correct, and what each metric reveals

DeepSearchQA uses outcome-based evaluation. Performance is judged solely on the completeness and correctness of the final answer set submitted by the agent, not on the search trajectory used to produce it. Nothing short of an exact match counts.

Semantic equivalence between a submitted answer and a ground-truth item is judged by Gemini 2.5 Flash in a zero-shot LLM-as-judge pipeline. Parcha AI's unofficial February 2026 evaluation, covering 896 of 900 questions using Claude Opus 4.6, identified nine questions where correct answers were penalized by the autorater precision scoring, and one ground truth error, through manual review of all non-perfect results. Automated scores need human auditing to be fully trusted.

The gap between F1 and Fully Correct is not cosmetic. The paper calls the roughly 15-point spread between Gemini Deep Research Agent's F1 and its Fully Correct rate the "Last Mile Problem." An agent can recover most correct items and still fail the exact-set test by including one extra low-confidence guess or missing one long-tail entity. That single error flips the Fully Correct score from pass to fail. The benchmark rewards precision in the strictest sense: an answer that is almost complete is still incorrect.

What the original evaluation revealed about the state of deep research agents

At launch, the benchmark exposed a wide capability gap between agentic systems with full search loops and reasoning-only models without them. Gemini Deep Research Agent achieved the leading Fully Correct rate of 66.09% (±3.16%) and an F1 of 81.90%. GPT-5 Pro High Reasoning reached 65.18% (±3.11%) FC and 78.98% F1. The paper reports the two rates as statistically comparable.

Reasoning models without agentic loops scored dramatically lower. Gemini 2.5 Flash achieved roughly half the F1 of the agentic leader, with a Fully Incorrect rate above 45%. A model can be strong at reasoning in general and still fail these tasks without the infrastructure to search, retrieve, and collate across many steps.

The failure examples from the paper's Table 5 show what those failures look like in practice. Gemini found qualifying countries but approximated quantities and ranked them incorrectly, which the paper labels a Quantitative Estimation Error. GPT-5 Pro stopped when it could not open an Excel file, a Tool Call Limitation. GPT-5 Pro also identified correct authors but did not apply the specified constraint, a Stopping Criterion Failure. These are not edge cases. They are the dominant patterns the benchmark was designed to surface, and they persist even in the highest-scoring systems at launch.

How researchers have used DeepSearchQA to drive architectural innovation

Because DeepSearchQA exposes distinct failure modes, research teams have used its scores to validate specific architectural interventions rather than general model improvements. The benchmark functions as a diagnostic instrument: it tells builders which component of their system is failing. JIT-Agent addresses scaffold dependency by generating task-adaptive harnesses for frozen backbone models. DeepSeek-V4-Flash with JIT-Agent scores 85.1, versus 76.2 for the vanilla scaffold, a gain of nearly 9 points from harness alone. GLM-5.2 with JIT-Agent reaches 93.9, versus 89.2 without it. SelfCompact addresses context-window pressure through compression. For GLM-4.7-Flash, MiniMax-M2.5, and MiMo-V2-Flash, compaction raises scores from 34.2 to 44.0, 52.0 to 61.3, and 46.5 to 56.8 respectively. Context retention over long causal chains is a significant bottleneck, and the benchmark makes that measurable.

LiveLedger addresses multi-constraint grounding. In an evaluation containing 43 DeepSearchQA questions within a 215-question multi-constraint set, it improves ReAct-120B accuracy from 39.1 to 50.7 and reduces the underverified answer rate from 76.3 to 49.8. Test-time compute scaling also moves scores sharply: allocating more compute via sampling at n=8 increases Fully Correct rate substantially compared with n=1, confirming that the benchmark responds to investment in inference compute, not just training. The JIT-Agent result is the sharpest evidence that scaffold quality and model capability are entangled in every leaderboard score, and that entanglement carries directly into the next question: what exactly is a leaderboard number measuring?

The harness problem: what leaderboard scores do and do not tell you

The most substantive challenge to interpreting DeepSearchQA leaderboard results is that a model's score depends heavily on the harness. The harness is the search infrastructure, tool orchestration, and scaffold surrounding the language model, not only the model itself. The same frozen backbone model changes by nearly 9 points simply by swapping a task-adaptive harness for a vanilla scaffold. A spread that large is enough to reorder models on the leaderboard, so direct comparison between self-reported numbers is unreliable.

Controlled-condition entries on leaderboards carry qualifiers such as "common browser harness and search backend across models" specifically to signal that the scaffold was held constant. Self-reported numbers from model launches typically use proprietary setups. If you read a leaderboard without attending to those qualifiers, you are comparing systems that differ in ways the scores cannot isolate.

The F1/Fully Correct trade-off is also unresolved at the architecture level. No current system has definitively solved the recall-precision balance. The benchmark makes the tension explicit, but it does not tell teams how to resolve it. Knowing that the harness matters is itself useful signal. It tells builders that search infrastructure quality is a first-class variable, not a constant to ignore when comparing agents.

Where the leaderboard stands and what the trajectory shows

The leaderboard has moved substantially since launch. Claude Opus 4.8 (Anthropic, closed) sits at 93.1% Fully Correct. Kimi K2.6 (Moonshot AI, open) follows at 92.5%. Apodex 1.1 (Apodex, closed) is at 92.4%. dots3-note Preview (Dots Studio, open) reaches 92.1%. Mercury 2 (Inception, closed) sits at 34.0%, a reminder that the spread across the field is still wide.

The distance from the original paper's leading score of 66.09% FC at launch to the current top of 96.0% is substantial. If you have a well-specified target, that trajectory shows how quickly research investment follows it. It also raises a question the benchmark's designers intended all along: how much of that improvement came from better models, and how much from better harnesses, better search infrastructure, and better stopping heuristics? The scores are now legible in exactly the way the methodology intended. They reward systems that can enumerate completely, resolve entities correctly, and know when to stop.

Why DeepSearchQA scores matter for teams building research agents on live web data

DeepSearchQA scores measure the output of an entire system: model, scaffold, and search infrastructure. That means a team's choice of search API is directly implicated in its benchmark position, not just its model selection. The commercial API spread of 28% to 82% FC on a held-constant question subset demonstrates that search infrastructure quality is not a commodity. What sits between the model and the web is a first-class variable, and DeepSearchQA is one of the few benchmarks that makes that visible.

Financial research is a primary use case explicitly featured in DeepSearchQA. Purpose-built financial research APIs are benchmarked on their ability to handle this class of query reliably. An infrastructure provider that cannot return clean, complete, source-attributed answer sets will fail the same Fully Correct test that a frontier model fails when its scaffold is inadequate.

For enterprise builders, the benchmark's emphasis on verifiable, cited answer sets, rather than plausible-sounding responses, aligns directly with compliance requirements. DeepSearchQA's methodology makes that demand explicit. The benchmark was built to measure exactly the property that determines whether a research agent can be trusted with consequential work: not whether it can find an answer, but whether it can find every answer, and prove it.