A financial research agent can return a polished report with plausible conclusions, working citations, and calculations that appear to reconcile. That establishes something about the output. It says much less about how reliably the system produced it.
Consider an agent asked to screen a US equity universe for companies with material exposure to data-center power demand and recommend two for further underwriting. The system has to interpret “material exposure,” establish the universe and period, and identify companies and sources. It then retrieves documents and data, extracts comparable evidence, calculates relevant quantities, resolves contradictions, decides when it has enough evidence, and synthesizes the result.
A failure anywhere in that sequence can change the recommendation. The agent can search the wrong period, miss a filing, or retrieve the right evidence but exclude it from working state. It can confuse reported and adjusted figures, calculate a ratio with inconsistent denominators, or stop before investigating a contradiction. Its conclusion may fail to reflect the evidence it collected.
The language model is one component. The useful unit for evaluation is the running research system: models, prompts, orchestration, tools, data sources, retrieval, working state, computation, runtime policies, and the mechanisms that turn all of these into analysis or decision support.
This system view changes the evaluation problem. The questions become whether each part of the research process works reliably, whether failures can be localized, and whether an observed failure points toward a controlled next experiment.
On September 2, I’ll discuss the adjacent problem in trading strategy research:
how to identify key design decisions, diagnose constraints exposed by the evidence,
and select experiments to improve strategy performance. Free registration here.
What a financial research agent actually does
Implementations differ. Some systems use one model repeatedly inside a tool loop. Others route work among specialized agents for research, financial analysis, valuation, review, and report generation. The distinction matters less for evaluation than the functions the system has to perform.
A useful functional decomposition has six stages.
The first is task interpretation and planning. The system translates a research request into an operational task: universe, period, subquestions, required evidence, analytical operations, and a stopping condition. A request such as “compare the economics of these companies” requires decisions about peer definitions, business segments, time periods, accounting measures, and which economic differences matter.
The second is information acquisition. The agent searches filings, earnings materials, transcripts, market data, company disclosures, economic series, industry sources, internal research, or other approved data. It also has to find the right document, parse it, identify the relevant passage or table, and respect the date when information became available.
The third is an evidence workspace. Useful research requires more than appending retrieved text to a context window. The system needs a representation of what it has established: extracted facts, source provenance, dates, unresolved questions, conflicting observations, intermediate calculations, and hypotheses that require additional evidence.
The fourth is computation and tool use. Financial research routinely requires deterministic operations that should be inspectable outside the language model: ratios, growth rates, valuation calculations, time-series transformations, reconciliations, screening operations, table extraction, and sometimes code execution. The inputs, transformation, and output should be preserved wherever practical.
The fifth is analysis and judgment. The system compares evidence, reconciles discrepancies, considers alternative explanations, applies the relevant financial concepts, and decides what the evidence supports. This stage may send the agent back for another filing, another calculation, or evidence on a competing hypothesis.
The sixth is verification and synthesis. Before returning the result, the system checks coverage of the original request, connects claims to sources, surfaces relevant uncertainty or disagreement, and produces the report, recommendation, estimate, or other decision support.
The resulting system is iterative. Analysis creates new research questions. Calculations expose missing inputs. Conflicting sources require another retrieval pass. The orchestrator selects the next operation and decides when to stop.
A financial research agent runs an iterative workflow over shared state and configuration. Instrumentation spans every stage so the team can evaluate components, trajectories, and repeated runs.
The figure is a functional model rather than a prescription for one implementation. A recent survey of deep research agents describes planning, multi-hop retrieval, iterative tool use, code execution, memory, and structured report generation across static and dynamic workflows and single-agent and multi-agent systems. The open Finance Agent v2 harness provides one concrete financial example, with web search, SEC EDGAR access, page parsing, stored-information retrieval, price history, and a calculator.
The architecture provides an evaluation map because each function introduces a different class of failure.
Evaluate the system where it can fail
A single aggregate score cannot distinguish these failures.
Task interpretation: Verify the universe, period, definitions, and requested output with coverage checks and explicit task decomposition. Otherwise, the agent may screen the wrong companies or answer only part of the request.
Retrieval: Measure required-source recall, retrieval success, and date validity. A missed filing or transcript leaves downstream components without required evidence.
Evidence selection and state: Inspect lineage, workspace contents, and provenance coverage. A relevant fact may be retrieved and then discarded or buried.
Calculation: Recompute results independently and test formulas, units, and tolerances. Correct revenue figures can still produce the wrong growth rate.
Analysis and judgment: Use counterfactual and sensitivity tests. A material risk disclosure should affect the risk assessment in a defensible direction.
Orchestration: Analyze trajectories, tool errors, and stopping behavior. Repeated searches can consume the budget while a missing calculation remains undone.
Synthesis: Grade required criteria and check citations. Correct research can still omit a requested company or material qualification.
Repeated execution: Run repeated trials and record failure frequency and decision variance. Similar tasks should succeed consistently.
Evaluator: Compare automated scores with independent finance-expert judgments and adjudicate consequential disagreements.
The metric depends on the task. A retrieval test can measure whether required sources or facts were recovered. A valuation task can reproduce calculations and impose numerical tolerances. A company screen can test coverage of the eligible universe and verify inclusion criteria. A forecast can be evaluated with proper scoring rules and calibration. A long report can be decomposed into required claims. A recommendation can be subjected to scenario or evidence-sensitivity tests.
These operations require different metrics. They share one engineering requirement: observability.
Instrument the research process before evaluating it
A research team cannot diagnose a system from the final answer alone.
For each run, preserve enough information to reconstruct what happened. Record the model and version, system and task prompts, tool definitions, query date, source access rules, searches, retrieved documents, extracted passages, API data, and calculations with their inputs. Also record errors, time and token budgets, tool-call limits, stopping reason, total cost, and final output.
This does not require exposing a model’s private internal reasoning. It requires recording the system’s observable actions and artifacts.
Suppose an agent’s report omits Intel’s decline in notebook volume while explaining its revenue slowdown. The system may never have found the relevant filing. It may have retrieved the filing but failed to identify the passage. It may have extracted the fact but failed to retain it in the evidence passed to analysis. The analysis may have had the evidence and given it no weight.
The final report looks the same in all four cases. The next experiment should be different.
AgentBoard provides useful methodological precedent. Its authors argue that final success rates reveal little about intermediate agent behavior, then add a progress-rate metric over intermediate states or annotated subgoals and trajectory analysis. Its environments cover embodied, game, web, and tool tasks rather than financial research. The engineering implication still applies: intermediate observations distinguish systems that receive the same final score after materially different runs.
Use report-level grading for what it can measure
FrontierFinance, released in August 2026 by seven Samaya AI researchers, addresses one part of the architecture: evaluating whether a long financial research report covers the facts, calculations, and qualifications required by the task.
The benchmark contains 220 open-ended finance research tasks spanning six parts of the investor workflow. The authors decompose them into 11,543 binary rubrics written and reviewed by finance professionals.
One released question asks what caused Intel’s revenue slowdown. The rubric separately checks reported revenue for 2023 and 2024, notebook and desktop volumes, demand explanations, pricing, and other evidence needed for the causal account. Must-have labels identify especially consequential criteria.
The public questions average 52.5 rubrics, with a median of 27 and a range from 3 to 475. Each report receives the fraction of its criteria that it satisfies, and the benchmark averages those scores by question. A screening problem containing several hundred checks therefore does not dominate a smaller company-research question because it has more rubric items.
This decomposition has support outside finance. CheckEval, an EMNLP 2025 study across 12 evaluator models, reports stronger correlation with human judgments, higher evaluator agreement, lower variance, and more traceable decisions from checklist-based evaluation.
DeepResearch Bench makes a related separation. Its RACE framework evaluates report quality against task-specific criteria and a reference report. Its FACT framework separately examines factual abundance and citation trustworthiness. The benchmark supplies methodological support for keeping report quality and information acquisition in separate scores. It does not provide evidence about financial-agent performance.
Criterion-level grading tells the team which required claims or calculations are missing. It does not identify why they are missing.
The model and the agent are different units of comparison
The system view also changes how leaderboard results should be interpreted.
FrontierFinance compares models using native web search, models inside the open Finance Agent v2 harness, and Samaya’s in-house system, which uses its own models, tools, retrieval engines, and financial data index.
The strongest reported native-search configuration satisfies 33.0% of all criteria. The leading Finance Agent v2 configuration reaches 49.2%. Samaya’s high-effort system reaches 56.0%.
Those results show that system configuration affects performance. They provide limited evidence about which component caused the difference.
Samaya’s 6.8-point advantage over the strongest Finance Agent v2 configuration combines changes in models, retrieval, tools, and data infrastructure. A team deciding whether to change its retriever, tool interface, model, context construction, or stopping policy needs controlled comparisons within its own system.
The configuration therefore belongs in the result. Record the model and version, prompts, tools, data providers, retrieval settings, inference parameters, orchestration rules, context limits, tool-call budgets, query date, evaluation date, and cost.
Finance adds an important temporal requirement. A system answering a historical investment question must exclude information that became available afterward. FrontierFinance gives every system the query date in its prompt, while Samaya also applies that date as a retrieval cutoff to its data index. These mechanisms provide different controls, and a model may have absorbed later information during training.
Diagnose the trajectory, not the tool-call count
Execution traces become useful once a report misses a criterion.
FrontierFinance provides a simple example. Claude Fable 5 leads the Finance Agent v2 group at 49.2% while averaging 16.6 tool calls per question. GPT 5.6 Sol averages 46.3 tool calls and scores 46.8%. More activity does not imply more complete research.
The benchmark also examines failed page retrievals. It classifies a URL as originating from parametric knowledge when its domain did not previously appear in the run’s search history. Those direct attempts have higher access-error rates than search-discovered URLs. Invalid addresses and pages that block crawlers consume the same finite execution budget as useful research.
A trajectory diagnostic should ask what the actions accomplished. Did searches add relevant evidence? Did the system return repeatedly to the same sources? Did tool failures trigger a sensible alternative? Was a calculation attempted once the required inputs were available? Did the agent identify unresolved questions? Did it stop because the requested evidence had been collected or because the run budget expired?
For the Intel example, suppose the trace shows that the filing and notebook-volume decline were both retrieved before the report was written. Another retrieval query is unlikely to fix the omission. The failure has moved downstream.
Retrieval and evidence use are separate tests
A source appearing in a report does not establish that its contents affected the analysis.
RAGChecker separates retriever properties such as claim recall and context precision from generator properties such as context utilization, faithfulness, and sensitivity to irrelevant information. Its experiments cover general retrieval-augmented generation, so the metrics require validation on financial research tasks.
Lost in the Middle shows that models’ use of relevant information can depend on where it appears in a long context. Access to the document does not imply uniform use of its contents. The experiments use multi-document question answering and key-value retrieval rather than investment research, but they identify a component behavior worth testing.
A new finance-specific preprint makes the distinction more consequential. Reading Is Not Using holds a focal risk disclosure constant while expanding unrelated filing context from 2,000 to 128,000 tokens. The authors report that direct retrieval of the disclosure remains strong while its influence on a downstream judgment approaches their experimental noise floor.
In their experiments, placing a targeted structured extraction immediately beside the decision restores more of that influence. Chunk-then-aggregate processing and instructions to reason further do not.
The paper is explicitly a very preliminary draft. Its primary experiments use constructed disclosures, its tests on actual filings are exploratory and capability-dependent, and a public replication package is still pending. The result is best treated as a hypothesis for evaluation.
First determine whether the agent can retrieve and identify the relevant evidence. Then change one material fact under controlled conditions and measure whether the downstream estimate, ranking, recommendation, or risk assessment moves in the expected direction.
A system can pass the first test and fail the second.
Test the same system more than once
Agent trajectories are stochastic. Search results vary, models sample different actions, sources become temporarily inaccessible, and an early choice can redirect the rest of a long run.
One successful execution provides weak evidence of reliability. Run important evaluation tasks repeatedly and measure the distribution of results. At minimum, record task success, criterion coverage, critical failures, cost, tool-call count, and whether the final decision changes.
τ-bench introduced pass^k to estimate the chance that an agent succeeds on all k repeated trials of the same task. Its retail and airline customer-service domains are far removed from investment research. The reliability principle applies directly: average single-run success can conceal a system that fails intermittently under the same underlying task.
Suppose an agent recommends Company A in seven runs and Company B in three despite having access to the same source set and evaluation date. The variation may reflect a close call supported by the evidence. It may instead come from one run missing a filing, another using a different denominator, and a third stopping before completing the peer comparison. The evaluation should reveal which case occurred.
This also changes how small benchmark differences should be interpreted. A one-run comparison between two configurations provides less information than repeated runs that estimate failure frequency and outcome variance on the same tasks.
Validate the evaluator too
Once long reports are decomposed into criteria, another system still has to determine whether each criterion was satisfied.
FrontierFinance uses majority vote across GPT 5.4, Gemini 3.1 Pro, and Claude Sonnet 4.6. The authors report that this three-model panel closely matched a nine-model committee in preliminary experiments. The paper does not report agreement with finance experts on the released tasks.
That leaves one component of the evaluation system unvalidated against the intended professional standard.
Model-based graders have documented failure modes. The MT-Bench and Chatbot Arena study, for example, found position and verbosity biases and weaknesses on some reasoning tasks. Its pairwise evaluation setup differs from FrontierFinance’s binary pointwise grading, so those findings do not establish that FrontierFinance has the same biases.
The appropriate test is local. Sample tasks across research types and difficulty levels. Have finance professionals grade the same criteria independently. Measure agreement, inspect disagreements that would change a deployment decision, and revise ambiguous rubrics or grading prompts before relying on the automated evaluator at scale.
The grader is another component of the system.
Define the evaluation contract before the run
These tests become easier to manage when the research team specifies them before comparing systems.
Task and decision: The research task, universe, output, and decision it informs.
Required evidence and behavior: Required claims, sources, calculations, tolerances, coverage, temporal rules, and sensitivity tests.
System configuration: Models, prompts, tools, data, retrieval, orchestration, budgets, dates, versions, and cost.
Component observations: Retrieval, evidence state, calculation, analysis, trajectory, synthesis, repeatability, and grader results.
Follow-up experiment: One component change, the controls held fixed, and the result that would change the decision.
Return to the data-center-power screen.
Before the run, define the equity universe and what qualifies as material exposure. Specify the evidence required to advance a company, the allowed source hierarchy, the relevant date, and calculations that must reconcile. Decide which omissions invalidate the result.
Then instrument the system. If a company is missing because its filing was never discovered, test retrieval. If the filing was found but the relevant segment disclosure was not extracted, test document processing. If the correct inputs were preserved but the exposure calculation was wrong, test computation. If the evidence was available and correctly calculated but the ranking barely responds when the material input changes, test analysis. If identical runs produce different selections because the agent stops at different points, test orchestration and repeatability.
Choose one intervention that addresses the observed failure and hold the rest of the configuration fixed.
A benchmark score describes how a configuration performs on a defined task set. An instrumented evaluation identifies what to change.
FrontierFinance makes one difficult part of that process unusually concrete: it turns long, open-ended financial reports into thousands of inspectable claim-level criteria. The broader evaluation begins with task interpretation and continues through evidence retrieval, computation, judgment, synthesis, repeated runs, and validation of the evaluator.
My free 30-minute Lightning Lesson on Wednesday, September 2, Managing Your Strategy Research Process, covers the adjacent problem in empirical ML and strategy research. We will identify the decision a result informs, diagnose the constraint exposed by the evidence, and select a controlled experiment that could change a proceed, revise, monitor, or stop decision.