FinSearchComp Benchmark for Financial Search APIs

FinSearchComp tests whether financial AI agents can find and use the evidence an analyst needs, rather than reason over information already placed in front of them. Its central lesson for financial search APIs is that retrieval is the prerequisite for accurate answers, and its depth, freshness, and source quality determine how far an agent can go.

Why financial search benchmarks before FinSearchComp measured the wrong thing

When a benchmark supplies a 10-K excerpt, it measures reasoning over that excerpt separately from an agent’s ability to find it under time pressure. That distinction matters because finding, retrieving, and reconciling information is a substantial part of an analyst’s work. A correct calculation over supplied text cannot establish that a system can assemble reliable evidence in an open environment.

General-purpose browsing benchmarks address a different part of the problem. BrowseComp tests multi-step navigation toward obscure facts with short, verifiable answers, while deliberately avoiding long-form synthesis, ambiguity resolution, and integration of domain knowledge across sources. That makes it useful for testing persistence through navigation, but insufficient for judging financial decision support. Before FinSearchComp, no open financial dataset tested whether agents could search for data from start to finish in the open domain. To fill that gap, financial experts had to build tasks like those they do at work, with a way to score answers that could change over time.

What FinSearchComp actually measures through three analyst workflows

FinSearchComp, published as a conference paper at ICLR 2026, is the first fully open-source agent benchmark for realistic, open-domain financial search and reasoning. It has 635 questions on global and Greater China markets. Its three task families increase in difficulty by changing what an agent must retrieve and reconcile, rather than simply asking for more elaborate reasoning.

T1, Time-Sensitive Data Fetching, tests rapidly changing answers such as stock prices, exchange rates, and gold prices. The paper’s example, “IBM latest close price”, requires shallow but timely retrieval, with careful calendar alignment and ticker alias resolution. T2, Simple Historical Lookup, tests point-in-time facts such as earnings figures, employee counts, and balance sheet line items. Its example, “What was Apple's iPhone revenue in Q4 2018?”, requires the correct disclosure and reporting calendar, with attention to restatements, units, and currency. T3, Complex Historical Investigation, extends the task across multiple periods or sources, requiring provenance reconciliation, unit normalization, and synthesis across many data points.

Those distinctions also govern the evidence used to establish answers. T1 gets its answers from real-time financial APIs, while T2 gets them from company filings, regulator sites, and professional databases. T3 combines historical sources that require manual synthesis. A rubric-guided language model evaluator assigns binary scores using tolerance bands for minor rounding differences and data revision lags.

Why the benchmark is hard to game through the annotation process behind the 635 questions

The cost of constructing FinSearchComp is part of its credibility. The benchmark took the work of 70 professional financial experts, approximately 240 expert hours, and a blind arbitration process with several stages. Annotators and arbitrators held advanced finance degrees and came from institutions including Citadel, JPMorgan, and CITIC Securities. That investment matters because a financial question can appear precise while leaving its metric, period, or adjustment convention unresolved.

The process started with trusted primary sources and checks of one source against others. Explicit definitions distinguished contested measures such as split-adjusted prices, Static PE vs. PE TTM, while numerical intervals accommodated retroactively revised data points. After an expert created a question, one or two other experts solved it independently without seeing the answer. Disagrepancies went to senior arbitration, and questions without agreement were modified or rejected. These controls make the benchmark harder to shortcut because success depends on resolving the same evidentiary problems that experts had to resolve.

What the results expose about search as a prerequisite

When FinSearchComp tested 21 models and products, it found that models without search score zero on T1. Parametric memory cannot access real-time data, so this is a categorical limitation. For current prices and rates, no amount of recall supplies the missing observation. Search is therefore structural to the task.

Search augmentation produces substantial average gains across all three task families. The gains are most pronounced for T1, meaningful for T2, and smaller but still significant for T3. Without search, models get low scores above zero on the historical tasks, where facts learned during pre-training may be approximate or out of date. Web-enabled products also give far more accurate answers overall, on average, than API-only models. The pattern makes the way search is integrated a central variable in financial agent performance.

The smaller T3 gain reveals the limit of merely adding access to the web. Multi-hop investigation needs deep searches, checks that sources agree, and a way to tell primary evidence from secondary evidence. More reasoning alone does not resolve those requirements. FinSearchComp also identifies an “overthinking” penalty on T1, where advanced reasoning decreased performance by complicating simple retrieval and introducing errors. The useful capability is not maximum deliberation everywhere, but retrieval and reasoning matched to the task.

Where the frontier stands and what top model performance reveals about the remaining gap

FinSearchComp reports that Grok 4 (web) leads the Global subset, approaching but not reaching expert-level accuracy. GPT-5-Thinking (web) is the runner-up, trailing by a noticeable margin. Human expert performance on the Global subset is 75.0.

DouBao (web) leads on the Greater China subset, but all models are still far behind human experts. Human expert performance there is 88.3. Model origin and the geographic scope of integrated search tools correlate with performance on market-specific tasks. The “home field” pattern suggests that training corpora and search integration remain heavily regionalized, challenging any assumption that a single API will serve every market equally well.

Only a handful of models surpass a modest average accuracy threshold overall. Performance on T3 falls sharply for virtually all models, making complex historical investigation the clearest unresolved task. A strong position on another leaderboard therefore cannot stand in for evidence of financial search competence. The practical gap between agents and experts is partly a problem of how search infrastructure acquires and supports the reconciliation of evidence.

The API benchmark layer and what historical tiers reveal about search infrastructure

FinSearchComp’s historical tiers have been adopted for head-to-head evaluation of search API providers. T2 tests access to company filings, regulator sites, and professional databases, while T3 tests aggregation across historical sources. At the API layer, those tasks show which infrastructure connects to primary institutional data and which relies on general web crawling. The important distinction is the evidence an API can reach.

Direct integration with sources such as SEC EDGAR, FRED, and BLS gives retrieval systems access to primary data that query engineering alone cannot supply. Licensed institutional data, including S&P Global, is another way to get data beyond general web results. Agentic retrieval also lets a system plan and carry out research in several steps for each query. Together, source access and research orchestration explain what the historical tiers test in infrastructure terms.

Research-effort settings make the trade-off between depth, latency, and cost explicit. A deep tier costs more per call and has a moderate cap on latency for work such as earnings summaries and competitive benchmarking. An exhaustive tier costs the most per call and has a higher cap on latency for deep due diligence and full 10-K analysis. A reference implementation built on LangChain’s Deep Agents, with optional LangSmith tracing, demonstrates a macro research workflow with parallel execution and human-in-the-loop review at anomaly detection. Financial research involves a controlled sequence of retrieval and review.

Why shallow search fails on T3 and the retrieval depth problem agents must solve

FinSearchComp’s case analysis found two reasons why systems often do poorly on T3: they stop searching too soon or retrieve information that is out of date. An agent may stop at a news summary before reaching the primary filing. It may also treat cached or secondary material as if it were current primary evidence. In both cases, the returned result looks relevant enough to halt the investigation before its foundations are sound.

This “shallow search” pattern is a retrieval failure. A model can reason correctly over stale or wrongly scoped data and still produce the wrong answer. Financial infrastructure must therefore be tested to see whether the evidence it returns is primary, current, and correctly scoped, as well as relevant. The agentic layer must also know when an ambiguous or high-stakes claim needs verification. Allocating more effort to those claims directly addresses the failure mode, whereas treating every query identically leaves the hardest tasks under-investigated.

How compliance requirements change the calculus when selecting financial search infrastructure

A search API inside a financial agent is also a data processor receiving the agent’s queries. Those queries may contain customer PII, account identifiers, or confidential business context during normal operation. Under GDPR, that relationship requires a signed Data Processing Agreement, a defined retention policy, and a lawful basis for any transfer outside the EU. The infrastructure decision therefore concerns both the evidence coming back and the sensitive information going out.

The financial rules already in force also cover AI-generated outputs. FINRA Rule 3110, Reg S-P, SOX, and fiduciary duty rules make clear that compliance obligations apply even before legislation written specifically for AI. For enterprise financial deployments, the criteria include Zero Data Retention, GDPR with a signed DPA, data residency guarantees, and Bring Your Own Cloud. Zero Data Retention means queries and results are not stored after completion, while BYOC keeps queries within the enterprise VPC.

These requirements need to be understood as deployment conditions. SOC2 certification does not itself establish that zero data retention is enabled, and an available retention option is not the same as a default policy. Stronger compliance can also go hand in hand with accurate answers and fast service. Compliance and benchmark performance can coexist, but each must be evaluated on its own evidence.

What developers should evaluate when choosing financial search infrastructure for an agent

FinSearchComp gives developers a task-based checklist rather than a reason to select a provider from one headline score. For T1, the first question is whether the API connects to live financial data or relies on cached and crawled content that may be hours or days old. An API that cannot support the freshness requirement cannot support a live-market workflow. Calendar alignment and ticker resolution belong in that assessment because a fresh value can still answer the wrong query.

For T2, developers should examine native access to primary filing and regulatory sources, including SEC EDGAR, company disclosure databases, and government statistical agencies. Daloopa’s FinRetrieval research showed a huge gap in how accurately the same underlying model answered with access to a structured data API and with web search alone. For T3, the test is whether the infrastructure supports multi-step retrieval with configurable depth and verifies high-stakes claims before accepting them. Those questions separate source coverage from the ability to investigate what the sources mean.

Compliance is then a gate alongside the requirement for accuracy. Developers should evaluate ZDR, a signed DPA, SOC2, data residency, and BYOC against the intended deployment, rather than assume that a platform label settles each requirement. The final step is to benchmark candidate infrastructure directly against FinSearchComp tasks that reflect the agent’s own workflows. That shows how fresh and historically precise the answers are and how deeply the system searches for each task. Published leaderboards can guide the shortlist, but measurable performance on the work the agent must actually do is the stronger standard.