Evaluating a financial research agent means deciding whether it can complete work you would otherwise assign to an analyst. That might be a calculation backed by filings, an assessment of a company’s funding risk, an editable valuation model, or code that reproduces a strategy paper. Each requires a different test.
We already know how to compare model predictions with observed outcomes. LLMs broadened the problem to answers that can be wrong in different ways. Agents broaden it again: they choose sources, call tools, revise their work, and deliver a result after many steps. Evaluation must now account for the environment in which those steps occur, the path chosen, and the evidence behind the result.
Financial agent benchmarks are beginning to address these problems. Some isolate retrieval or numerical reasoning. Others ask agents to conduct research, build spreadsheets, reproduce code, or interpret news. This article explains how those evaluations work, who builds them, and how to select them for your own system.
New on ML4T: find and inspect financial AI evaluations
Alongside this issue, we are launching a financial AI evaluation resource to help you choose tests for the system you are building. The searchable catalog covers 91 overlapping evaluation resources, including 66 classified as benchmarks, alongside suites, component datasets, and studies.
Open an individual evaluation page to see who built it, what the system is asked to do, what inputs and tools it receives, what it must produce, and how the result is graded. Where public examples are available, the page shows sourced tasks, marked as quotations or paraphrases, along with limitations and conditions for data and code access. You can inspect the work behind a benchmark score before deciding whether the test fits your own research process.
The accompanying background paper, Choosing Evaluations for Financial AI Systems, develops the framework in more detail, including evidence requirements, grader validation, repeated runs, and the cost of checking and repairing work.
For hands-on practice, my Agent Engineering workshop on November 21 covers building, evaluating, and deploying a multi-agent forecasting system. Subscribe to my Maven profile for updates on courses and workshops, incl. early-bird discounts through October 31.
From predictions to answers to completed work
In a conventional supervised-learning task, we specify an output and compare it with a target. A classifier predicts a label; a forecaster predicts a value or distribution. Accuracy, forecast error, calibration, and related metrics summarize particular aspects of performance. Choosing the target, validation period, and economic objective still requires judgment, but the scoring operation is often straightforward.
An LLM can produce a paragraph containing facts, calculations, qualifications, and a conclusion. Exact matching will reject many valid answers. A fluent response can also contain an unsupported claim that a similarity score overlooks. Evaluation therefore needs to separate requirements: whether facts are correct, evidence supports them, the calculation follows, and the conclusion answers the question. Reference answers, executable checks, expert review, and task-specific rubrics address different parts of that problem.
An agent adds decisions about how to do the work. It might search for filings, download a report, extract a table, run Python, discover an inconsistency, and search again. The model operates inside a harness: the software that supplies instructions and tools, maintains working state, and controls execution. Changing its search service, document parser, or time budget can change the result even when the model stays the same.
A benchmark for that system needs a task, an environment, and a definition of success. The environment determines which documents and tools are available, what actions are permitted, and when execution stops. The evaluator may inspect the final answer, the generated workbook or code, and the recorded tool calls and outputs. A frozen document collection supports repeatable comparisons; live web research tests a changing information environment.
We need both completion measures and ways to diagnose failures. A wrong answer could follow from failed retrieval, incorrect interpretation, a coding error, or an unsupported conclusion. Intermediate checks can distinguish those failures. They should enforce requirements while allowing valid alternative ways of completing the task. Requiring an agent to imitate one reference sequence would confuse agreement with that sequence and successful research.
What finance adds to the evaluation
Financial research combines several kinds of correctness. Sources must refer to the right entity and period. Accounting definitions and units must be compatible. Assumptions must be explicit, and conclusions must follow from the evidence. For historical decisions, the information must also have been available at the time.
Some requests have a bounded answer, such as an adjusted earnings figure. Others require judgment about liquidity, competitive position, or materiality. An editable model adds requirements about formulas and dependencies. A trading decision introduces realized outcomes, transaction costs, risk, and the possibility that an apparently good result came from favorable market conditions.
The relevant evaluation depends on which of these jobs the agent performs. A broad financial-knowledge score can help diagnose domain knowledge. It provides limited evidence that a configured agent can produce a research deliverable. Louis Bertucci and Murad Nuriyev’s Evaluating AI in Finance provides the broader taxonomy behind the catalog; the comparisons below focus on methods that change a practitioner’s evaluation choice.
How much of the catalog evaluates agents?
Of the 66 benchmark records, we classify 15 as evaluations of agents or configured research, search, and decision products. Another three mix protocols or leave the execution boundary unclear. The remaining 48 test bounded answers, fixed pipelines, or components such as retrieval. These tests can diagnose an agent’s capabilities. The protocols we inspected do not evaluate an adaptive workflow from request to completed work.
Classification of the inspected protocols in the 91-resource catalog, excluding suites, nested component-dataset entries, and studies. Counts describe what the published evaluations run, not every possible use of their datasets.
Names alone can mislead. FinAgentBench grades document and passage rankings without external tools in its published experiments. Conversely, Fin-RATE includes hosted web-search configurations alongside bounded QA and retrieval tests. We keep it in the mixed group with BizFinBench.v2 and CNFinBench, rather than treating every configuration as either a model test or a complete agent evaluation.
Who builds financial evaluations?
Of the 66 benchmark records, 49 list publication dates of 2024 or later, including 20 in 2026. Dates attached to suite adaptations do not establish when their underlying datasets first appeared. The newer research benchmarks also bring different builders into the field.
Most of the older resources come from academic language-processing research: sentiment labels such as Financial PhraseBank, financial opinion question answering such as FiQA, and numerical reasoning such as FinQA, which researchers at JPMorgan Chase built with university collaborators. Academic consortia then packaged many of these datasets into suites. FinBen, released in February 2024, gathers stock-movement, credit-scoring, fraud, and bankruptcy datasets under one evaluation. The catalog includes its component datasets as separate entries.
The newer benchmarks that grade completed research increasingly come from companies. Patronus AI built FinanceBench; Vals AI and Stanford researchers built the Finance Agent Benchmark; Kensho and S&P Global built BizBench and FrontierFinance; and Rogo, Samaya AI, Paper Instruments, and Sonto built the research and judgment benchmarks discussed below. These builders have access to realistic tasks and to experts who can grade them. Several, such as Rogo and Samaya AI, also sell AI systems for the work their benchmarks grade.
Who built a test doesn’t determine whether it is sound, but it should shape how you interpret the results. Check whether the builder’s own system appears in them, whether the full task set is public or only a sample, and whether the data sits behind an access agreement. Each evaluation page lists the authors, their affiliations (where the sources report them), and the data, code, and license terms.
What does a benchmark score establish?
The benchmark’s task and the evidence available to its grader determine what its score can establish. Running a multi-step agent does not require the evaluator to inspect every step. A response rubric can grade the delivered analysis while leaving its tool execution unexamined.
Each score answers a different question. Each row identifies a claim that the score alone cannot support; it does not imply that the benchmark is defective.
Component tests explain where research fails
FinanceBench supplies financial questions with reference answers, justifications, and evidence pages. Its public subset contains 150 questions. It is a useful starting point for testing answers grounded in company disclosures. Run a question with retrieved context, then with verified, sufficient evidence. Errors that remain in the second condition warrant investigating interpretation or calculation; a performance gap warrants inspecting retrieval and context selection.
FinDER and FinAgentBench examine financial evidence retrieval. Ranking metrics measure whether relevant passages appear near the top. They help diagnose search, but a well-ranked passage can still be used incorrectly in the answer.
FinQA evaluates numerical reasoning using executable programs and their results. That distinction exposes cases where the wrong derivation still produces the right number. TAT-QA combines tables and text, while ConvFinQA adds dependent calculations across conversational turns.
These resources isolate capabilities that an agent needs. They are especially useful during development, when a single end-to-end failure rate gives little guidance about what to improve. Passing them leaves the entire research workflow untested.
Research benchmarks differ in what the grader can see
BigFinanceBench checks the research derivation
BigFinanceBench contains 928 difficult financial-research tasks authored by 52 experts. Agents use a common harness with web search, EDGAR access, page fetching, and Python, limited to 50 steps. Judges inspect the answer and visible trajectory, including tool calls, outputs, and calculations. Weighted criteria grade the derivation; judges score final-answer correctness separately.
One task asks whether Dayforce’s management-adjusted EBIT is overstated when it treats capitalized software development as a real cost. Resolving it requires finding disclosures, understanding the adjustment, and calculating its effect. That is substantially richer than retrieving a reported number.
Choose this approach when you need to evaluate multi-source calculations and locate failures within the supporting work. The benchmark authors deliberately selected these questions to challenge existing agents, so the score does not estimate error frequency on ordinary analyst requests. The public release includes 50 tasks with references, rubrics, and traces; the full 928-task evaluation is larger.
DiligenceBench checks the completed analysis
DiligenceBench contains 150 equity-research tasks across 143 companies. Questions ask for judgments about cash-flow quality, funding, reserves, concentration, and other research concerns. Task-specific rubrics cover factual accuracy, analytical reasoning, and risk awareness, with negative weights for active errors.
The judge receives the task, the full response, and the criteria together. The benchmark permits different agent harnesses and supplies reference configurations with progressively more tooling, from web search to a writable Python workspace and finance-specific tools.
This design helps compare the research output of complete systems. It also makes attribution harder. A stronger result after adding tools and instructions reflects the changed configuration; it does not isolate a model improvement. Because grading centers on the response, retain the underlying evidence and calculations separately when you need to audit the research process.
Samaya’s FrontierFinance checks investor research against detailed rubrics
Samaya’s FrontierFinance reports 220 research questions across six investor workflows, with 11,543 binary grading criteria. Its evaluator uses a majority vote from three model judges. Judges receive the question, date, response, and rubric batches, not a separate source corpus or the full tool trajectory.
Choose this design when the deliverable is an investor-research response with many explicit requirements. Its grading code lets you inspect how those requirements translate into scores. Agreement among model judges differs from validation by financial experts. The dataset also requires an access agreement, so inspect availability before planning a local run.
These benchmarks examine related work at different resolutions. BigFinanceBench exposes derivation evidence, DiligenceBench emphasizes the completed analysis, and Samaya’s FrontierFinance applies detailed investor-research criteria. Their percentages have different meanings. Combining them into a single ranking would conceal those differences.
Workbooks and code need tests of the artifact
Kensho and S&P Global, with MIT collaborators, use the same name for a different FrontierFinance. Its 25 computer-use tasks require agents to build three-statement, discounted-cash-flow, leveraged-buyout, merger, and lender models. The deliverable is a workbook that someone can inspect and change.
Its rubrics include validity checks for periods, sources, and formula logic. The automated judge has SEC access and tools to inspect workbooks. Human graders graded a ten-task subset; even with detailed rubrics, the automated judge overestimated AI-generated work relative to human graders.
For spreadsheet agents, apply the same focus to the usable model. A correct displayed valuation can conceal hardcoded cells or broken dependencies. Check whether changing an assumption updates the intended outputs, whether statements reconcile, and whether another analyst can repair the workbook. Validate the grader too.
QuantCode-Eval targets another artifact: code reproducing quantitative strategies from 30 papers. It uses 572 executable property checkers to test requirements across data handling, modeling, signals, and portfolio construction. An attempt passes only when the task’s complete checker suite passes.
The evaluation separates successful attempts from tasks solved at least once in five attempts. The latter tells you whether retries found a correct implementation. It does not mean the agent is consistently correct or that you can select the good attempt without checking it. The release includes contracts and checkers; twenty original data panels require WRDS access. Reproduction correctness and future strategy profitability remain separate questions.
Judgment and investment outcomes require different references
Frontier Financial Judgment asks agents whether news is new, important, and positive or negative for valuation. It combines 82 analyst-designed synthetic events with real news and historical documents for four semiconductor and supply-chain companies. The reference is analyst judgment about the events.
Scoring each label separately differs from requiring all three labels to be correct for an event. The study’s background-escalation measure also lacks exhaustive expert negative labels, limiting its interpretation as a false-positive rate. For a monitoring agent, add adjudicated negative cases and the normal stream of duplicates and irrelevant articles. The costs of missed events and unnecessary escalation depend on that stream. Agreement with analyst labels does not measure realized price moves.
InvestorBench and StockBench instead evaluate agents making sequential investment decisions through simulated portfolio outcomes. These tests connect decisions across time and expose the agent to a market environment. Interpreting the result requires examining the universe, period, available information, execution assumptions, costs, and comparison portfolio. A favorable return in one setting provides different evidence than a well-researched answer or a reproducible strategy implementation.
What can you inspect and rerun?
Public tasks let you examine whether a benchmark resembles your work. A local rerun also needs grading code, source data, tools, and access rights. Public code and a permissive license in one repository don’t show you can reproduce the full evaluation.
Selected release inventories inspected through October 20, 2026. These are access and artifact checks, not independent replications. A public subset supports a different run from the full reported evaluation.
The catalog also includes FinGovBench, which adds a governance target. Given a declared financial workflow and a change, a method identifies affected components, chooses an action and revalidation scope, and decides whether release is permitted. Its public development and validation labels allow local scoring; test labels are withheld. The release remains a candidate pending independent expert adjudication. This tests governance decisions over supplied workflow records, rather than an agent independently conducting the underlying financial research.
Choose tests for the work you will assign
Start with the deliverable and the decision you need to make about the system.
For a filing-based question-answering assistant, combine FinanceBench-style grounded answers with retrieval diagnostics and FinQA-style calculation checks. Add complete requests using the sources, periods, and missing-information cases your analysts encounter.
For multi-source research and memos, inspect BigFinanceBench, DiligenceBench, and Samaya’s FrontierFinance. Choose whether your evaluation needs the derivation, the final analysis, or both. Adapt the rubrics to the actual research brief.
For spreadsheet construction or strategy reproduction, test the generated artifact. Use workbook behavior or executable properties alongside expert review of financial interpretation.
For news monitoring or investment decisions, use the relevant event or market environment. Include ordinary negative cases for monitoring and appropriate risk and execution assumptions for portfolio evaluation.
A brief filing example shows why these tests need to connect. Amazon’s June 2023 filing contains twelve-month operating cash flow of 61,841 and quarterly net income of 6,750. An agent can retrieve the document and divide correctly, yet report a misleading quarterly ratio of 9.16. Using the three-month cash flow of 16,476 gives 2.44. The retrieval and arithmetic pass the first answer; the reporting-interval check rejects it. This is a constructed candidate answer using real filing values.
For system selection, keep the tasks and grading fixed, record tools and budgets, and repeat runs. If you are comparing models, hold the harness fixed as well. If you compare complete systems, treat tooling differences as part of the comparison. Public subsets, licenses, data entitlements, and access gates determine what you can actually run.
Before relying on an automated grade, check its decisions against held-out expert judgments. Keep consequential requirements mandatory for acceptance; a high average rubric score can coexist with a wrong entity, period, or accounting treatment. Then evaluate representative requests, recording completion, escalation, errors in accepted work, and total effort including retries and human repair.
Use the catalog and individual evaluation pages to shortlist tests, inspect their tasks and grading, and check which materials you can access. Choose the evaluations whose work products and success conditions match the work you expect your agent to complete.
For hands-on practice, my Agent Engineering workshop on November 21 covers building, evaluating, and deploying a multi-agent forecasting system. You’ll modify agents and compare configurations on the same questions.