We already know how to compare model predictions with observed outcomes. LLMs broadened the problem to answers that can be wrong in different ways. Agents broaden it again: they choose sources, call tools, revise their work, and deliver a result after many steps. Evaluation must now account for the environment in which those steps occur, the path chosen, and the evidence behind the result.

Financial agent benchmarks are beginning to address these problems. Some isolate retrieval or numerical reasoning. Others ask agents to conduct research, build spreadsheets, reproduce code, or interpret news. This article explains how those evaluations work, who builds them, and how to select them for your own system.

New on ML4T: find and inspect financial AI evaluations

Alongside this issue, we are launching a financial AI evaluation resource to help you choose tests for the system you are building. The searchable catalog covers 91 overlapping evaluation resources, including 66 classified as benchmarks, alongside suites, component datasets, and studies.

Open an individual evaluation page to see who built it, what the system is asked to do, what inputs and tools it receives, what it must produce, and how the result is graded. Where public examples are available, the page shows sourced tasks, marked as quotations or paraphrases, along with limitations and conditions for data and code access. You can inspect the work behind a benchmark score before deciding whether the test fits your own research process.

The accompanying background paper, Choosing Evaluations for Financial AI Systems, develops the framework in more detail, including evidence requirements, grader validation, repeated runs, and the cost of checking and repairing work.

For hands-on practice, my Agent Engineering workshop on November 21 covers building, evaluating, and deploying a multi-agent forecasting system. Subscribe to my Maven profile for updates on courses and workshops, incl. early-bird discounts through October 31.

From predictions to answers to completed work

In a conventional supervised-learning task, we specify an output and compare it with a target. A classifier predicts a label; a forecaster predicts a value or distribution. Accuracy, forecast error, calibration, and related metrics summarize particular aspects of performance. Choosing the target, validation period, and economic objective still requires judgment, but the scoring operation is often straightforward.

An LLM can produce a paragraph containing facts, calculations, qualifications, and a conclusion. Exact matching will reject many valid answers. A fluent response can also contain an unsupported claim that a similarity score overlooks. Evaluation therefore needs to separate requirements: whether facts are correct, evidence supports them, the calculation follows, and the conclusion answers the question. Reference answers, executable checks, expert review, and task-specific rubrics address different parts of that problem.

An agent adds decisions about how to do the work. It might search for filings, download a report, extract a table, run Python, discover an inconsistency, and search again. The model operates inside a harness: the software that supplies instructions and tools, maintains working state, and controls execution. Changing its search service, document parser, or time budget can change the result even when the model stays the same.

A benchmark for that system needs a task, an environment, and a definition of success. The environment determines which documents and tools are available, what actions are permitted, and when execution stops. The evaluator may inspect the final answer, the generated workbook or code, and the recorded tool calls and outputs. A frozen document collection supports repeatable comparisons; live web research tests a changing information environment.

We need both completion measures and ways to diagnose failures. A wrong answer could follow from failed retrieval, incorrect interpretation, a coding error, or an unsupported conclusion. Intermediate checks can distinguish those failures. They should enforce requirements while allowing valid alternative ways of completing the task. Requiring an agent to imitate one reference sequence would confuse agreement with that sequence and successful research.

What finance adds to the evaluation

Financial research combines several kinds of correctness. Sources must refer to the right entity and period. Accounting definitions and units must be compatible. Assumptions must be explicit, and conclusions must follow from the evidence. For historical decisions, the information must also have been available at the time.

Some requests have a bounded answer, such as an adjusted earnings figure. Others require judgment about liquidity, competitive position, or materiality. An editable model adds requirements about formulas and dependencies. A trading decision introduces realized outcomes, transaction costs, risk, and the possibility that an apparently good result came from favorable market conditions.

The relevant evaluation depends on which of these jobs the agent performs. A broad financial-knowledge score can help diagnose domain knowledge. It provides limited evidence that a configured agent can produce a research deliverable. Louis Bertucci and Murad Nuriyev’s Evaluating AI in Finance provides the broader taxonomy behind the catalog; the comparisons below focus on methods that change a practitioner’s evaluation choice.

How much of the catalog evaluates agents?

Of the 66 benchmark records, we classify 15 as evaluations of agents or configured research, search, and decision products. Another three mix protocols or leave the execution boundary unclear. The remaining 48 test bounded answers, fixed pipelines, or components such as retrieval. These tests can diagnose an agent’s capabilities. The protocols we inspected do not evaluate an adaptive workflow from request to completed work.

Classification of the inspected protocols in the 91-resource catalog, excluding suites, nested component-dataset entries, and studies. Counts describe what the published evaluations run, not every possible use of their datasets.

Names alone can mislead. FinAgentBench grades document and passage rankings without external tools in its published experiments. Conversely, Fin-RATE includes hosted web-search configurations alongside bounded QA and retrieval tests. We keep it in the mixed group with BizFinBench.v2 and CNFinBench, rather than treating every configuration as either a model test or a complete agent evaluation.

Who builds financial evaluations?

Of the 66 benchmark records, 49 list publication dates of 2024 or later, including 20 in 2026. Dates attached to suite adaptations do not establish when their underlying datasets first appeared. The newer research benchmarks also bring different builders into the field.

Most of the older resources come from academic language-processing research: sentiment labels such as Financial PhraseBank, financial opinion question answering such as FiQA, and numerical reasoning such as FinQA, which researchers at JPMorgan Chase built with university collaborators. Academic consortia then packaged many of these datasets into suites. FinBen, released in February 2024, gathers stock-movement, credit-scoring, fraud, and bankruptcy datasets under one evaluation. The catalog includes its component datasets as separate entries.

The newer benchmarks that grade completed research increasingly come from companies. Patronus AI built FinanceBench; Vals AI and Stanford researchers built the Finance Agent Benchmark; Kensho and S&P Global built BizBench and FrontierFinance; and Rogo, Samaya AI, Paper Instruments, and Sonto built the research and judgment benchmarks discussed below. These builders have access to realistic tasks and to experts who can grade them. Several, such as Rogo and Samaya AI, also sell AI systems for the work their benchmarks grade.

Who built a test doesn’t determine whether it is sound, but it should shape how you interpret the results. Check whether the builder’s own system appears in them, whether the full task set is public or only a sample, and whether the data sits behind an access agreement. Each evaluation page lists the authors, their affiliations (where the sources report them), and the data, code, and license terms.

What does a benchmark score establish?

The benchmark’s task and the evidence available to its grader determine what its score can establish. Running a multi-step agent does not require the evaluator to inspect every step. A response rubric can grade the delivered analysis while leaving its tool execution unexamined.

Each score answers a different question. Each row identifies a claim that the score alone cannot support; it does not imply that the benchmark is defective.

Component tests explain where research fails

FinanceBench supplies financial questions with reference answers, justifications, and evidence pages. Its public subset contains 150 questions. It is a useful starting point for testing answers grounded in company disclosures. Run a question with retrieved context, then with verified, sufficient evidence. Errors that remain in the second condition warrant investigating interpretation or calculation; a performance gap warrants inspecting retrieval and context selection.

FinDER and FinAgentBench examine financial evidence retrieval. Ranking metrics measure whether relevant passages appear near the top. They help diagnose search, but a well-ranked passage can still be used incorrectly in the answer.

FinQA evaluates numerical reasoning using executable programs and their results. That distinction exposes cases where the wrong derivation still produces the right number. TAT-QA combines tables and text, while ConvFinQA adds dependent calculations across conversational turns.

These resources isolate capabilities that an agent needs. They are especially useful during development, when a single end-to-end failure rate gives little guidance about what to improve. Passing them leaves the entire research workflow untested.

Research benchmarks differ in what the grader can see

BigFinanceBench checks the research derivation

BigFinanceBench contains 928 difficult financial-research tasks authored by 52 experts. Agents use a common harness with web search, EDGAR access, page fetching, and Python, limited to 50 steps. Judges inspect the answer and visible trajectory, including tool calls, outputs, and calculations. Weighted criteria grade the derivation; judges score final-answer correctness separately.

One task asks whether Dayforce’s management-adjusted EBIT is overstated when it treats capitalized software development as a real cost. Resolving it requires finding disclosures, understanding the adjustment, and calculating its effect. That is substantially richer than retrieving a reported number.

Choose this approach when you need to evaluate multi-source calculations and locate failures within the supporting work. The benchmark authors deliberately selected these questions to challenge existing agents, so the score does not estimate error frequency on ordinary analyst requests. The public release includes 50 tasks with references, rubrics, and traces; the full 928-task evaluation is larger.

DiligenceBench checks the completed analysis

DiligenceBench contains 150 equity-research tasks across 143 companies. Questions ask for judgments about cash-flow quality, funding, reserves, concentration, and other research concerns. Task-specific rubrics cover factual accuracy, analytical reasoning, and risk awareness, with negative weights for active errors.

The judge receives the task, the full response, and the criteria together. The benchmark permits different agent harnesses and supplies reference configurations with progressively more tooling, from web search to a writable Python workspace and finance-specific tools.

This design helps compare the research output of complete systems. It also makes attribution harder. A stronger result after adding tools and instructions reflects the changed configuration; it does not isolate a model improvement. Because grading centers on the response, retain the underlying evidence and calculations separately when you need to audit the research process.

Samaya’s FrontierFinance checks investor research against detailed rubrics

Samaya’s FrontierFinance reports 220 research questions across six investor workflows, with 11,543 binary grading criteria. Its evaluator uses a majority vote from three model judges. Judges receive the question, date, response, and rubric batches, not a separate source corpus or the full tool trajectory.

Choose this design when the deliverable is an investor-research response with many explicit requirements. Its grading code lets you inspect how those requirements translate into scores. Agreement among model judges differs from validation by financial experts. The dataset also requires an access agreement, so inspect availability before planning a local run.

These benchmarks examine related work at different resolutions. BigFinanceBench exposes derivation evidence, DiligenceBench emphasizes the completed analysis, and Samaya’s FrontierFinance applies detailed investor-research criteria. Their percentages have different meanings. Combining them into a single ranking would conceal those differences.

Workbooks and code need tests of the artifact

Kensho and S&P Global, with MIT collaborators, use the same name for a different FrontierFinance. Its 25 computer-use tasks require agents to build three-statement, discounted-cash-flow, leveraged-buyout, merger, and lender models. The deliverable is a workbook that someone can inspect and change.

Its rubrics include validity checks for periods, sources, and formula logic. The automated judge has SEC access and tools to inspect workbooks. Human graders graded a ten-task subset; even with detailed rubrics, the automated judge overestimated AI-generated work relative to human graders.

For spreadsheet agents, apply the same focus to the usable model. A correct displayed valuation can conceal hardcoded cells or broken dependencies. Check whether changing an assumption updates the intended outputs, whether statements reconcile, and whether another analyst can repair the workbook. Validate the grader too.

QuantCode-Eval targets another artifact: code reproducing quantitative strategies from 30 papers. It uses 572 executable property checkers to test requirements across data handling, modeling, signals, and portfolio construction. An attempt passes only when the task’s complete checker suite passes.

The evaluation separates successful attempts from tasks solved at least once in five attempts. The latter tells you whether retries found a correct implementation. It does not mean the agent is consistently correct or that you can select the good attempt without checking it. The release includes contracts and checkers; twenty original data panels require WRDS access. Reproduction correctness and future strategy profitability remain separate questions.

Judgment and investment outcomes require different references

Frontier Financial Judgment asks agents whether news is new, important, and positive or negative for valuation. It combines 82 analyst-designed synthetic events with real news and historical documents for four semiconductor and supply-chain companies. The reference is analyst judgment about the events.

Scoring each label separately differs from requiring all three labels to be correct for an event. The study’s background-escalation measure also lacks exhaustive expert negative labels, limiting its interpretation as a false-positive rate. For a monitoring agent, add adjudicated negative cases and the normal stream of duplicates and irrelevant articles. The costs of missed events and unnecessary escalation depend on that stream. Agreement with analyst labels does not measure realized price moves.

InvestorBench and StockBench instead evaluate agents making sequential investment decisions through simulated portfolio outcomes. These tests connect decisions across time and expose the agent to a market environment. Interpreting the result requires examining the universe, period, available information, execution assumptions, costs, and comparison portfolio. A favorable return in one setting provides different evidence than a well-researched answer or a reproducible strategy implementation.

What can you inspect and rerun?

Public tasks let you examine whether a benchmark resembles your work. A local rerun also needs grading code, source data, tools, and access rights. Public code and a permissive license in one repository don’t show you can reproduce the full evaluation.

Selected release inventories inspected through October 20, 2026. These are access and artifact checks, not independent replications. A public subset supports a different run from the full reported evaluation.

The catalog also includes FinGovBench, which adds a governance target. Given a declared financial workflow and a change, a method identifies affected components, chooses an action and revalidation scope, and decides whether release is permitted. Its public development and validation labels allow local scoring; test labels are withheld. The release remains a candidate pending independent expert adjudication. This tests governance decisions over supplied workflow records, rather than an agent independently conducting the underlying financial research.

Choose tests for the work you will assign

Start with the deliverable and the decision you need to make about the system.

• For a filing-based question-answering assistant, combine FinanceBench-style grounded answers with retrieval diagnostics and FinQA-style calculation checks. Add complete requests using the sources, periods, and missing-information cases your analysts encounter.

• For multi-source research and memos, inspect BigFinanceBench, DiligenceBench, and Samaya’s FrontierFinance. Choose whether your evaluation needs the derivation, the final analysis, or both. Adapt the rubrics to the actual research brief.

• For spreadsheet construction or strategy reproduction, test the generated artifact. Use workbook behavior or executable properties alongside expert review of financial interpretation.

• For news monitoring or investment decisions, use the relevant event or market environment. Include ordinary negative cases for monitoring and appropriate risk and execution assumptions for portfolio evaluation.

A brief filing example shows why these tests need to connect. Amazon’s June 2023 filing contains twelve-month operating cash flow of 61,841 and quarterly net income of 6,750. An agent can retrieve the document and divide correctly, yet report a misleading quarterly ratio of 9.16. Using the three-month cash flow of 16,476 gives 2.44. The retrieval and arithmetic pass the first answer; the reporting-interval check rejects it. This is a constructed candidate answer using real filing values.

For system selection, keep the tasks and grading fixed, record tools and budgets, and repeat runs. If you are comparing models, hold the harness fixed as well. If you compare complete systems, treat tooling differences as part of the comparison. Public subsets, licenses, data entitlements, and access gates determine what you can actually run.

Before relying on an automated grade, check its decisions against held-out expert judgments. Keep consequential requirements mandatory for acceptance; a high average rubric score can coexist with a wrong entity, period, or accounting treatment. Then evaluate representative requests, recording completion, escalation, errors in accepted work, and total effort including retries and human repair.

Use the catalog and individual evaluation pages to shortlist tests, inspect their tasks and grading, and check which materials you can access. Choose the evaluations whose work products and success conditions match the work you expect your agent to complete.

For hands-on practice, my Agent Engineering workshop on November 21 covers building, evaluating, and deploying a multi-agent forecasting system. You’ll modify agents and compare configurations on the same questions.

The tests cover earnings releases, company event disclosures filed on Form 8-K, and responses to Securities and Exchange Commission (SEC) comment letters. For the event disclosures, Jev had to choose the filing category. For the letters, it had to predict whether SEC staff would issue further comments after the company’s response.

The classifiers learned from filings paired with the answers for each task. Those answers came from reported financial figures, the company’s chosen filing category, or subsequent staff correspondence. Jev received the text and questions without task-specific training examples.

Jev identified revenue changes with 98.5% accuracy and earnings-per-share changes with 99.0%. No trained classifier in these tests passed 73% or 64%, respectively. Trained classifiers were more accurate at assigning filing categories and predicting further SEC comments. Adding information about investment-company filings improved Jev’s SEC forecasts, but the trained text models still led.

Disclosure: TypeSafe provided API access for this unpaid evaluation and suggested the domain-information addition tested below.

Jev identified earnings changes more accurately than trained classifiers

For the earnings test, each model received a 2023 filing. It had to decide whether total revenue and diluted earnings per share under US generally accepted accounting principles (GAAP) rose from the same quarter a year earlier, and whether the company reported a net loss. The questions specified the requested figures and comparison period.

NVIDIA’s first-quarter fiscal 2024 release, filed May 24, 2023, illustrates the task. Its opening headline reports quarterly revenue up 19%. Revenue was $7.19 billion, compared with $8.29 billion a year earlier. The headline compares with the previous quarter. Year-on-year revenue fell 13%.

GAAP diluted EPS rose from $0.64 to $0.82, while adjusted EPS fell 20%. The release shows both growth and decline. Correct answers require the GAAP figures and the year-on-year comparison specified in the questions.

Jev answered revenue down, GAAP EPS up, and no net loss, assigning probabilities of 0.95, 0.98, and 0.97 to those answers. One fine-tuned text model, ModernBERT, got EPS right but predicted revenue growth and a net loss. SEC-BERT got EPS and net loss right but also predicted revenue growth.

The reference answers came from financial figures tagged in each company’s quarterly or annual report. Comparing the current and prior-year figures supplied both the training labels and the answers used for scoring. We trained and tested the classifiers on separate groups of companies.

Some releases omit the figures needed for the comparison. The headline results cover 802 revenue and 833 EPS comparisons for which both report numbers were found in the release text. This retrospective check uses reports that can arrive after the release and matches number strings without verifying their meaning. It cannot serve as an automatic eligibility test when new releases arrive.

On those selected releases, Jev scored 98.5% for revenue and 99.0% for EPS. The strongest trained classifier, TF-IDF with LightGBM, reached 72.6% and 63.6%. Always choosing the more common answer would score 56.6% and 51.1%, so class imbalance does not explain the earnings result. TF-IDF represents words and word pairs as weighted counts; LightGBM fits decision trees to that representation.

AUC measures how well the probabilities rank positive answers above negative ones across cutoffs. Jev scored 0.998 for revenue and 0.999 for EPS, compared with LightGBM’s 0.806 and 0.668. All methods’ AUCs appear in the appendix.

Without the number check, Jev’s accuracy was 95.4% on 887 revenue comparisons and 94.9% on 948 EPS comparisons. It scored 97.3% on 1,044 net-loss observations, which had no equivalent check.

Jev received the full release, while the fine-tuned text models received roughly the first 4,000 tokens. Those models were trained as document classifiers with untuned settings. The test did not include a system that extracts and compares the requested figures.

The tested questions offered only yes/no answers, so Jev had to choose even when a release omitted the requested comparison. A research system using these questions would need to handle unavailable comparisons separately.

Trained classifiers assigned 8-K categories more accurately than Jev

Form 8-K gives companies numbered categories for reporting significant events. Each model had to identify the category the company selected when submitting its filing to the SEC. The test covers January-September 2023 filings with one substantive item, in eleven categories. Models received the narrative and relevant exhibits after removing item numbers and category headings.

Tesla’s August 7, 2023 filing reports that Vaibhav Taneja became CFO after Zachary Kirkhorn stepped down. The answer key is Item 5.02, the category for specified director and officer changes. Jev selected it under three question wordings, with probabilities of 0.97-0.99. Both fine-tuned encoders selected it too.

Across 3,006 usable filings, Jev scored 79.3%. TF-IDF with logistic regression reached 89.4%; LightGBM reached 91.3%. The fine-tuned encoders scored about 88%. Training on public labels was more effective at reproducing the filer’s category across these eleven categories.

Jev’s errors concentrate in Item 7.01 (Regulation FD disclosure) and Item 8.01 (other events). Form 8-K’s instructions permit Regulation FD information under either item, with different furnished/filed treatment. Reproducing the company’s choice requires distinguishing categories that can describe the same event. Trained classifiers may learn drafting cues such as furnished/filed language; these runs did not isolate the cues responsible for their advantage.

On the other nine categories, Jev was correct on 98.6% of 1,876 filings, ahead of the strongest trained baseline’s 95.1%. On the 1,130 Item 7.01/8.01 filings, Jev scored 47.3%, versus 77.6% for fine-tuned ModernBERT and 75.5% for SEC-BERT. We selected that split after examining errors and using the true category. A researcher cannot achieve that result by filtering on Jev’s prediction. When Jev predicted one of the nine categories in this mixed feed, it was right 83.1% of the time.

A system could use answer probabilities to reserve uncertain category assignments for manual review. Retaining only the most confident predictions gave Jev 47.9% coverage of the evaluated filings at 99% observed precision; fine-tuned ModernBERT covered 55.5%. The cutoffs were chosen after the outcomes were known, so a live system would need to validate them on new filings.

Trained classifiers predicted further SEC comments more accurately than Jev

SEC staff use comment letters to ask companies to clarify their filings or add disclosure. In this test, Jev and the classifiers had to predict whether the staff would issue further comments after a company responded.

Teladoc’s February 3, 2022 response answers a request to explain a 15.7-year useful life for acquired customer relationships. Teladoc committed to expanding disclosure in its next annual report. The staff closed the review on February 8.

Whether the response fully resolves every issue on its face differs from whether the staff will issue another comment. The closing outcome requires forecasting the staff’s next action. A response can satisfy that process by committing to future disclosure.

On Teladoc, Jev assigned a 0.42 probability to further comments when asked directly about that outcome. Asking whether the staff would be satisfied and close the review implied 0.43. Asking whether the response fully resolved every comment implied 0.76. Fine-tuned ModernBERT and SEC-BERT predicted 0.24 and 0.01, respectively. The observed outcome was no further comments.

The broader test used 2,261 distinct response letters from 2022-2023. The outcome labels mark 925 (40.9%) as drawing substantive further comments within 90 days. A letter closing the review counts as no further comments. The forecast concerns another round of substantive questions within that window, rather than eventual resolution of the filing review.

Jev’s accuracy was 67.0% under the direct “further comments” question, 58.8% under “satisfied” and 41.5% under “fully resolves every comment”. The full-resolution question asks for a more demanding judgment about the response than the future staff action used for scoring.

Always predicting no further comments would score 59.1% accuracy, since that was the more common outcome. AUC measures how well the probabilities rank letters that drew further comments above those that did not. Brier score measures squared probability error, with lower values better. A model can rank the letters correctly while assigning probabilities that overstate the risk of further comments.

The direct Jev question scored AUC 0.713 and Brier 0.2204. Fine-tuned ModernBERT reached 80.6% accuracy, AUC 0.890, and Brier 0.1346. Every trained text classifier beat the direct Jev question on all three metrics.

A hint about investment-company filings improved Jev’s SEC forecasts

Following TypeSafe’s suggestion, we added a new question that stated investment-company filings rarely draw another round of comments. We added it after examining the initial results and testing it on the same letters. Its improvement therefore needs confirmation on new letters.

Jev’s accuracy rose to 67.9%, AUC to 0.831, and Brier improved to 0.1878. The probability improvement was concentrated in investment-company filings. Predictions also changed outside that group, worsening probability error for S-4 and F-4 responses, where further comments were common.

The augmented question ranked outcomes better than the simple form-plus-length comparator, whose AUC was 0.814. That comparator had a lower Brier score of 0.1711. All trained text classifiers beat the augmented Jev question on accuracy, AUC, and Brier.

Comment-letter Brier scores. The direct outcome question improves on the full-resolution question; post hoc domain information improves it further. The trained ModernBERT row still has the lowest error. Lower is better; all rows use 2,261 letters.

Full results for earnings changes, filing categories, and SEC follow-up

The tables use the same Jev-answered documents within each task. Revenue, EPS, and loss have different denominators and checking rules. ModernBERT and SEC-BERT are text encoders; we fine-tune them on the task’s labeled documents. The two zero-shot natural-language-inference (NLI) models instead score candidate answers as hypotheses against the text.

Earnings accuracy (%)

Revenue n = 802; EPS n = 833; net loss n = 1,044. Only revenue and EPS use the number check.

Filing-category accuracy (%)

n = 3,006. Jev averages three question wordings. BART uses plain category descriptions; formal item-title hypotheses scored 26.8%.

Further SEC comments: accuracy and probability error

n = 2,261. Higher AUC and lower Brier are better. We chose to add domain information after examining the results. The form-plus-length comparator uses a different split. Sample majority and constant-rate predictions use evaluated outcomes as diagnostics, rather than prospective forecasts.

Company grouping keeps test labels out of training, with qualifications detailed in the appendix: TF-IDF uses test text when building its vocabulary, and some letters share co-registrant companies across the split.

Jev leads the tested classifiers on earnings; the strongest trained baseline leads across all eleven evaluated 8-K categories and on further SEC comments. Task-specific denominators and validation limits apply.

Classifiers beat Jev on 8-K categories with 400-1,600 training filings

Jev’s category assignments require no task-specific training set. To measure how much supervision would beat them, we trained the classifiers on increasing numbers of labeled filings, excluding the tested companies. Each point averages two fixed evaluation halves, without uncertainty estimates, so the crossing ranges are approximate training budgets.

Across the eleven evaluated 8-K categories, logistic regression and SEC-BERT were below Jev at 400 labeled training documents and above it at 800. ModernBERT crossed between 800 and 1,600. At 1,600, logistic regression and ModernBERT also exceeded Jev’s evaluated-feed reference coverage of 47.9% at empirical 99% precision. SEC-BERT did not.

For comment letters, logistic regression and SEC-BERT exceeded the original Jev question at the smallest tested size, 200 letters. ModernBERT crossed between 200 and 800.

None reached Jev on earnings at the tested 100, 200, or 400 releases drawn. The earnings pools include unlabeled releases, so a 400-release draw supplies only 198-220 revenue labels. These curves compare against Jev’s 95.4% revenue accuracy before the number check.

Fixed company-excluding learning curves. Earnings counts are drawn from documents, including unlabeled releases, and use unfiltered revenue accuracy. Appendix tables give effective label counts. No uncertainty bands or LightGBM curve were produced.

Jev did not produce useful merger forecasts from contract terms alone

A separate experiment asked Jev whether a merger would complete, using the contract’s closing conditions. On 3,106 resolved deals, its AUC was 0.504, close to chance ranking. Forecasts from structured deal summaries depended heavily on the market-implied probability supplied alongside them. The inputs omitted news, regulatory developments, and financing research. The appendix gives the separate and paired sample results.

Earnings decisions were Jev’s strongest financial result

For a researcher extracting earnings-direction features, Jev produced more accurate answers than the tested classifiers without requiring task-specific training examples. NVIDIA’s release illustrates what that decision entails: selecting the requested metric and comparison period among several reported measures.

For reproducing SEC filing categories or predicting further staff comments, labeled examples gave the trained models an advantage. The category learning curves put the cost of matching Jev at roughly 400-1,600 labeled filings, depending on the classifier. The earnings result therefore does not settle the choice for every financial-text task.

These tests compare Jev with document classifiers, including two language models used for zero-shot classification. They do not compare it with other generative language or decision models, and it is unclear how much exposure to historical filings it received during pretraining.

Methods appendix

Which filings were used and how their scoring labels were constructed

Earnings. Start with 2023 Item 2.02 filings and extract the primary narrative plus EX-99 exhibits. The available sample contains 1,391 release bodies. Match each to the earliest qualifying 10-Q/10-K filed from ten days before to 100 days after the release, carrying an 80-100-day quarter ending no more than 100 days earlier. Use US-GAAP XBRL revenue, EPS, and net-income facts from that report and the corresponding prior-year quarter. The matching/tag-precedence rules can select a minor line or the wrong EPS basis. Equal comparison figures, zero net income, and absent facts produce missing labels. XBRL labels are later evidence for scoring, not inputs available to the model at release time.

Of 1,391 releases, 328 have no qualifying report match. Of 1,063 matches, two supply no nonmissing task label, leaving 1,061 releases submitted to Jev; five failed the service’s token limit. The 1,056 usable releases yield 887 revenue labels, 948 EPS labels and 1,044 loss labels. The number check retains 802 revenue and 833 EPS cases. It checks magnitude strings, not table meaning or signs, and the audit found false matches. The 20 audited discrepancies cover revenue/EPS only; no complete corrected key exists. The matching rule selects discrete quarterly facts; performance on unmatched releases and other reporting structures is unknown.

The source’s audit of the 20 remaining revenue/EPS disagreements attributed thirteen to scoring-label errors, four to metric definitions and three to unavailable comparisons. Ten labels used a minor revenue line: nine bank fee lines and one other-revenue line. Another bank case depended on gross revenue versus net interest income plus noninterest income. Three EPS labels lost a loss’s minus sign; three used total EPS where the release headlined continuing-operations EPS. The three unavailable comparisons passed the number check, and Jev’s answer probabilities were 0.57-0.74.

The source audit found no clear wrong Jev answer based on the requested figures in the release. The reported scores nevertheless retain the original labels. We did not compute a complete corrected-label score, and we did not audit agreement cases; the disagreement audit does not establish 100% accuracy.

Majority-class accuracy on the displayed revenue/EPS/loss populations is 56.6% / 51.1% / 54.7%.

8-K. The original random sample of 6,000 entries from EDGAR’s 2023 Form 8-K quarterly indexes yielded 5,704 retained rows, 4,049 single substantive-item reports, and 3,976 rows in eleven classes with at least 30 members. Accession deduplication leaves 3,967 eligible documents in the current training loader. Remove official item numbers/titles and section headings, check for surviving headings, and include EX-99 exhibits. The reported result covers 3,006 usable documents from 3,011 requests; five documents failed, producing fifteen unusable question rows. The budget-limited run processed date-ordered filings from January 3 through September 27, 2023. It excludes multiple substantive items and rare/unlisted categories. The nine-category analysis is a post hoc split by true label.

Letters. Sequence 2022-2023 UPLOAD and CORRESP filings by company. Decode staff letters and classify their content; an UPLOAD closing the review is not a positive outcome. A response is positive if substantive further comments occur within 90 days. Closing letters, silence, and unrelated declinations count as negative. Unreadable or unclassified follow-ups and responses without a full observation window are excluded. The last labeled response date is September 29, 2023. Company-level sequencing can link concurrent reviews, and not all regex classifications have been manually adjudicated.

The corpus contains 18,419 rows and 16,428 distinct readable accessions, with 287 accessions repeated for co-registrants. Restrict to responses between 3,000 and 90,000 characters and take an evenly spaced 2,500-row sample. It resolves to 2,261 unique letters, with 925 positives and no unusable answers. Duplicated answer accessions are pooled by wording. Their stored labels agree in the available outputs. The training loader keeps the first CIK per unique letter and the labels carried by answered rows.

Exact questions and answer options

These are the exact tested Jev instructions. Each earnings question has a true/false answer, with the criteria stated below. The three filing-category questions use the same eleven numbered options. None of the letter questions specify the 90-day scoring horizon.

Revenue up

Is the company’s total revenue for the most recent fiscal quarter reported here higher than its total revenue for the same quarter of the prior year?

True: Revenue for the latest quarter is higher than a year earlier. False: Revenue for the latest quarter is lower than a year earlier.

GAAP diluted EPS up

Is the company’s diluted earnings per share under GAAP for the most recent fiscal quarter reported here higher than for the same quarter of the prior year?

True: GAAP diluted EPS for the latest quarter is higher than a year earlier (a smaller loss per share counts as higher). False: GAAP diluted EPS for the latest quarter is lower than a year earlier.

GAAP net loss

Did the company report a GAAP net loss for the most recent fiscal quarter reported here?

True: The latest quarter’s GAAP net result is a loss. False: The latest quarter’s GAAP net result is a profit.

The post hoc domain run appends this exact text to the direct further-comments question:

Background: when the filing under review is an investment-company form (N-1A, N-2, N-CSR, N-14, N-4, N-CEN), the staff almost never issues a further round of comments.

The added statement is an intervention based on this sample, not an independently validated general fact about the SEC.

Which models were tested, what text they received, and how they were trained

Jev receives full supplied text and typed questions without labeled examples. All measured runs are recorded as jev-1.13.0, September 21-October 7, 2026. Original compact task files lack version fields; the experiment record and October 7 sampled recheck supply the attribution. Of 591 earnings and 456 event answers compared in that recheck, 2 and 5 changed. This sampled agreement does not establish identical responses on the entire corpus.

BART is facebook/bart-large-mnli; zero-shot ModernBERT is MoritzLaurer/ModernBERT-large-zeroshot-v2.0. Candidate answers are hypotheses. Consecutive text windows reserve space for hypotheses within nominal contexts of 1,024 and 8,192 tokens. For each hypothesis, take its maximum entailment logit across the full document’s windows, then softmax over the alternatives. Thus, the two answer hypotheses can obtain their strongest evidence from different windows. BART item-title and plain-description variants are reported separately.

TF-IDF uses words/word pairs, sublinear term frequency, document-frequency limits of 2 and 0.9, and at most 50,000 features. Logistic regression uses C = 10 and at most 2,000 iterations. LightGBM uses 400 trees, learning rate 0.05, 31 leaves, minimum child size 10, and column fraction 0.3. We did not tune these settings for this article. Vocabulary and IDF are fitted before splitting, on the whole task corpus. Only classifier labels are held out.

The fine-tuned general encoder is answerdotai/ModernBERT-large; the finance encoder is SEC-BERT, pretrained on 260,773 10-Ks from 1993-2019. ModernBERT uses two epochs, a learning rate of 2e-5, and a batch size of 2 with eight-step accumulation, for an effective batch of 16. SEC-BERT uses three epochs, a learning rate of 3e-5, and a batch size of 16. Both use weight decay 0.01, a linear schedule, and 10% warm-up. The runs used one RTX 3090.

ModernBERT uses a single nominal 4,096-token window, including special tokens. SEC-BERT processes up to eight 512-token windows, then averages probabilities. Content limits are 4,094 and 4,080 tokens, respectively, using different tokenizers; later text is dropped. Each SEC-BERT window inherits the document label, which may depend on information elsewhere. Earnings models have three outputs and mask missing labels. The limited training and window averaging are consequential design choices, not optimality claims. A one-fold ModernBERT check using five epochs and learning rate 5e-5 did not improve revenue accuracy; it was not a full tuning search.

How companies were separated between training and testing

For earnings and filing categories, companies are divided into five groups. Each classifier trains on four groups and is tested on the fifth, repeating until each group has been tested. This is GroupKFold splitting by CIK, the SEC’s company identifier. The earnings pool includes all 1,391 bodies: 330 have no task label, comprising 328 unmatched releases and the two matched but unlabeled releases. TF-IDF fitting masks missing labels separately for each question; the encoders mask missing output targets. Documents drawn therefore overstate supervision. The tables show the training counts for each split. Splits 1-5 correspond to stored fold IDs 0-4. Every 8-K training document is labeled, with 3,173-3,174 per fold.

Earnings training documents

Earnings training labels by question

Letter text models train on 8,875 unique letters outside the test letters’ first-recorded CIKs. A local audit confirms disjoint recorded CIK sets. Checking all co-registrant associations instead finds 86 letter-company IDs on both sides. These are nominal company-grouped tests, with residual issuer-association overlap for letters and full-corpus TF-IDF preprocessing.

The form-plus-length baseline instead excludes evaluated accessions and estimates outcome rates by subject form and length decile, using a minimum cell size of 20 and form/overall fallbacks. It trains on 14,961 corpus rows, retains co-registrant repetition and shares 1,727 company IDs with the evaluated letters. Its result is a simpler comparator with a different split, not an equally isolated-company model.

Learning curves randomly divide answered documents into two evaluation halves with seed zero. For each half, draw training documents from the other half plus unanswered documents, excluding evaluated recorded CIKs. Points are unweighted means of the two half scores; no repeated-seed uncertainty was measured. Earnings scores use all nonmissing labels, before the number check. We skipped the 800-document earnings draws because only 780 and 777 eligible documents remained. No grouped LightGBM curve was produced.

The eligible earnings pool ranges from 777 to 780 documents. Separate full-data folds draw 1,112-1,113 documents, with 845-853 carrying any label and 701-722 carrying revenue labels. The 400-document curve draws supply 198-220 revenue labels and uses Jev’s unfiltered 95.4% revenue reference, rather than the selected-release 98.5%. For comment letters, the 200-letter logistic-regression margin over Jev was 1.7 accuracy points.

The curve tables use “Logistic” for logistic regression. Training n counts documents used for fitting; earnings draw n also includes documents without labels.

Filing-category learning curve: accuracy (%)

Comment-letter learning curve: accuracy (%)

We did not test the 3,200-document filing point or the 100-, 400-, and 1,600-letter points. ModernBERT and SEC-BERT are fine-tuned in all learning curves.

Earnings learning curves: accuracy (%)

Revenue:

EPS:

Net loss:

Effective supervision in the earnings curves

Each point averages two separate tests, groups A and B (stored half IDs 0 and 1). Documents without the required answer do not train that question.

Curve sizes count documents drawn; only nonmissing earnings labels provide supervision. Full-data fits and curve points are separate experiments. Historical company grouping is not a forward time split. Neither company-size strata nor this design rules out prior exposure to the public filings. No after-training-cutoff test was performed.

How well the earnings probabilities ranked positive and negative answers

These scores use the main accuracy table’s populations: selected revenue and EPS releases, and all labeled net-loss observations. AUCs for Jev and the language-model baselines were recomputed from retained probabilities, with half credit for ties. TF-IDF AUCs come from the saved score table; its per-document predictions were not retained. The constant-score reference has AUC 0.500.

How probabilities were scored and answers selected for high precision

Binary decisions use P(yes) at 0.5; multiclass decisions choose the largest probability. Letter questions phrased positively about resolution are complemented into P(further comments). AUC gives tied probabilities half credit. A strictly increasing recalibration preserves ranking, though mappings that create ties can alter AUC. Brier measures probability error and does not alone establish calibration.

Effect of the letter question

Short row names refer to the exact questions above.

Probability error outside investment-company forms

The full-sample letter accuracy, AUC, and Brier scores are in the main results table. This table adds the Brier scores after excluding fund forms.

The three original letter questions imply mean probabilities of further comments of 0.44, 0.60, and 0.82. Adding domain information lowers the overall mean of the direct question from 0.44 to 0.35. The four sizeable fund-form groups’ Brier ranges change from 0.1003-0.2132 to 0.0131-0.0356, or roughly 0.10-0.21 to 0.013-0.036. For S-4 responses, Brier worsens from 0.2391 to 0.2631; for F-4, from 0.2624 to 0.2923. The pooled improvement from 0.2204 to 0.1878 closes 66% of the gap to the form-plus-length comparator’s 0.1711, while leaving the trained text models ahead.

Brier includes both reliability and resolution effects; a lower Brier score alone does not prove calibration. Pooled results combine different filing-form populations. After examining the original results, we selected the domain-information run and nine-category event analysis.

For coverage, rank the top answer probability for all methods, accept whole ties together, and select the widest evaluated set at or above 99% observed precision. Jev’s separately documented confidence output was not retained in these compact task files. All submitted usable documents receive answers; this metric adds selective acceptance afterward. It is neither class recall nor a prospectively fixed operating threshold. The tables below retain original keys.

Earnings coverage at 99% observed precision (%)

The nine-class accuracy uses filings whose true item is outside 7.01/8.01. Coverage uses all 3,006 filings at 99% observed precision.

Original-key scores do not resolve label ambiguities, omitted comparisons, or missing inputs observed in the data. The evidence supports method comparisons on the selected historical documents. It does not measure live research acceptance, probability portability, or investment returns.

Merger samples and the effect of including market probabilities

The raw closing-conditions run includes 3,106 resolved deals announced in 2005-2026, with 234 breaks (7.5%). AUC is 0.504, and Brier is 0.2151, versus 0.0697 for the constant sample-rate forecast. Extracted deal cards with market-implied probabilities cover 1,956 deals and score AUC 0.601. On 393 paired deals, including 22 breaks, removing the market input lowers AUC from 0.551 to 0.386 and produces a mean completion probability of 0.5006. Probability standard deviation falls from 0.1060 to 0.0321. On 770 paired cards, a separately tested riskier wording changes the mean completion probability by -0.0093. These are different selected samples; their AUCs are not a paired estimate of the gain from adding market information.

The assignment continues after the first assessment. Agents update deal probabilities as filings and press releases arrive. Balyasny’s central Applied AI team supplies infrastructure that investment teams customize around their strategies. Investors retain judgment over the conclusions they use.

My comparison of these accounts examines how funds specify the work they delegate and decide whether to accept the result. Investment expertise enters through data and labels, definitions and instructions, executable tools, and acceptance criteria. Bridgewater provides the broadest technical record in this selection, along with a live investment program.

Public disclosures through October 1, 2026. Sources and dates at the end.

Investment models predate research agents

Hedge funds were using machine learning before conversational LLMs. In a 2019 interview, Man AHL’s Anthony Ledford described ML in daily and intraday strategies, execution algorithms, and smart order routing, including deployed text-based strategies. AQR states that it actively uses ML across its investment process. Its November 2024 portfolio study illustrates learning nonlinear relationships between stock predictors and returns, separately from a managed portfolio’s performance.

Research agents add activities around those models. They can prepare data, investigate documents, write experiments, and revisit an analysis. The distinction helps locate responsibility. An agent that implements a candidate signal still depends on the research criteria used to accept it and the portfolio process used to trade it.

Millennium retains assignments and employee permissions

Millennium reports more than 1,600 personalized assistants, called Digital Twins, in use across the firm. Employees delegate recurring research, meeting preparation, and scheduled work through email. AI Product Manager Diana Meditz gives a specific example. Her assistant reviews overnight AI news and relevant inbox updates, then delivers a morning report without another prompt. Each assistant has its own identity and retained context. It starts with no access, receives permissions system by system, and cannot exceed its employee’s entitlements. The count measures adoption across employees; the firm supplies no measured gain in research quality.

Balyasny’s deal agent revisits an assessment when evidence changes. Millennium’s morning assistant carries a standing assignment into the next scheduled run. A team borrowing either approach has to specify what persists. For deal research, that includes the current assessment and evidence that could change it. For a morning report, it includes the employee’s instructions, relevant sources, and delivery schedule. Retained conversation alone would leave those requirements ambiguous.

Millennium separately announced a project with Anthropic to build, pilot, and optimize a digital risk analyst. It would explain daily changes in risk positions and retain context across questions to support human risk managers. The risk analyst remains a development announcement, separate from the deployed Digital Twins.

D. E. Shaw’s Neil Katz described a similar infrastructure approach in 2024: Assistants, an LLM Gateway, and DocLab for querying documents. Teams could connect models to their own data and software. D. E. Shaw and Balyasny both provide common infrastructure with local customization, enabling investment teams to specify work for their own strategies.

Citadel replicates papers; Man generates signal candidates

Citadel’s Ken Griffin described an agent that reads a finance paper, reproduces its results, and produces out-of-sample results. He reported an average of two to three hours per paper, against a human process taking six to eight weeks. The transcript supplies no paper list, failure rate, or accounting of human review time. Griffin also said the automation had not reduced headcount. Citadel had more problems for its existing people to pursue.

Man’s AlphaGPT starts earlier in research. Separate components propose hypotheses, translate them into code using internal tools and databases, and evaluate candidate signals against statistical, risk, and economic criteria. Man reports its greatest success so far in systematic equity research. Its investment committee reviews hypotheses and economic reasoning, while technology teams review implementation and tests before live use. The account establishes candidates passing internal research thresholds, without attributing live returns to AlphaGPT.

The comparison exposes a shared implementation problem. Citadel starts with a published method that the experiment must preserve. Man starts with a proposed relationship that the generated code must measure. Man explicitly warns that an agent can propose one idea and implement another. Its example asks whether stocks with more buy orders than sell orders tend to outperform. To implement that idea, a researcher would need to specify whether “more” means order counts or submitted share volume, over which interval, and when the information becomes available. A successful backtest using a different definition would still answer a different research question. Man also identifies the multiple-testing risk from producing many candidate signals and describes expanding monitoring infrastructure to maintain oversight quality as signal volume grows.

Man also describes using Claude Skills to express workflows in plain English. Its partnership announcement with Anthropic names AlphaGPT as an existing tool.

Two Sigma lowers the cost of preparing research ideas

Two Sigma’s Ben Wellington explains a different output. An LLM can generate descriptions of companies or events, which researchers can analyze as datasets. He uses CEO microexpressions during earnings calls as an example of an idea whose technical preparation had previously made it expensive to explore. The example illustrates a research possibility, without identifying a deployed predictor. Researchers supply the question and test whether the generated data contains predictive information. Wellington warns that poorly designed automation can make their results increasingly homogeneous.

Wellington reports feature-research insights arriving in days where the work previously took months, without supplying a controlled timing sample. CTO Jeff Wecker describes adapting AI tools to internal platforms. Cheaper preparation can reopen ideas researchers had previously set aside. Citadel’s account similarly describes using existing staff to investigate more problems.

Generated research data introduces a historical-information problem. Two Sigma’s Jin Choi explicitly warns about trusting backtests that predate an LLM’s knowledge cutoff. A model labeling an old earnings call may know events that occurred afterward. For a team adopting the method, the source date, model version, and prompt version belong in the research record. Those records help investigate leakage, although recording them alone cannot remove knowledge already in the model.

Bridgewater uses investor judgment in data, plans and tests

Bridgewater describes AIA Labs’ ambition as an artificial investor capable of the range of activities human investors perform. The firm’s account also describes AI research and tools feeding back into Pure Alpha, with experienced investors supplying feedback. Its resources include macroeconomic data, recorded investment reasoning, and investor expertise. Its public studies show different ways of using those resources.

Investor labels define what is worth reading

Bridgewater and Thinking Machines examined six document-filtering and segmentation tasks on data cleared for public release. Relevance depends on the investor’s question and responsibilities. The researchers first improved prompts by asking experts to articulate distinctions such as financially relevant versus useful to a macro investor. Expert prompting raised frontier-model accuracy from roughly 50% into the mid-to-high 70s.

Training examples supplied further judgment. The team initially trained on non-expert vendor labels, then sent examples where the model disagreed with its training label to investors for correction. Final evaluation used a held-out set. A trained Qwen3-235B model reached 84.7% average accuracy, versus 78.2% for the best prompted frontier comparator. The 13.8-fold saving concerns inference cost per task, excluding annotation, training, and integration. Ablations show that training choices also affect accuracy.

The method uses expert attention to settle contested examples and expert instructions to define the task. The result measures agreement with investor labels on the disclosed tasks. It does not isolate the effect of labels from the training procedure or compare with a closed model fine-tuned on the same data.

PAT turns a research question into inspectable computation

The Pocket Analyst Tool, or PAT, performs exploratory research. Bridgewater’s applied AI team describes an internally deployed analyst used by hundreds of investors. PAT retrieves proprietary documents and series, clarifies the question, and plans dataframe outputs with declared schemas and dependencies. Its execution architecture enforces validation and caches intermediate calculations. User corrections can become human-audited benchmarks and reviewed changes to context or implementation.

PAT makes the intended analysis explicit before generating code. That addresses the same relationship between research intent and implementation raised by Citadel and Man. The analyst can inspect the plan and revise the analysis. The presenters scope PAT as exploratory research, separate from trading. I covered the architecture in How Bridgewater Engineers a Research Agent.

SQL execution can succeed while the calculation is wrong

A text-to-SQL study by researchers at UIUC and Bridgewater AIA Labs, in collaboration with Thinking Machines, shows how successful execution can mislead. The researchers combine expert-checked training data with rewards that examine query behavior.

In the study’s example, a generated query averages line-item prices when the question requires average order totals. The result happens to match because every order in that database has one line item. With multi-item orders, the queries can return different averages. In a pilot training run, 32.8% of positive execution-match rewards went to queries that failed the additional equivalence check. The authors downweighted those rewards using bounded verification of query equivalence. Task definitions therefore enter the training objective as well as the prompt. This is database-query research, with no disclosed internal rollout.

A forecaster can add information while trailing the market

The AIA Forecaster report studies probability forecasts. Multiple agents search news, a supervisor investigates disagreements with further searches, and a statistical correction calibrates the combined forecast. The authors report no statistically significant difference from expert superforecasters on the two ForecastBench sets with those comparisons. That result concerns those tests, not general forecasting equivalence.

On MarketLiquid, the report gives rounded Brier scores of 0.126 for AIA and 0.111 for market consensus, where lower is better. A fitted combination scored 0.106 under leave-one-out evaluation. The benchmark covers 322 events at five forecast dates each, yielding 1,610 questions. These events resolved in April- May 2025. A forecast can contribute information even when it performs worse alone than the forecast already available. This results from combining probabilities before portfolio sizing, costs, and risk constraints.

AIA Macro: returns and investment history

Reuters dates Bridgewater’s AI-strategy work to 2018 and AIA Macro’s inception to late 2023. Bridgewater’s July 2024 statement then described a vehicle with external-client capital that combines tabular learning, reasoning tools, and LLMs. It specified $1.6 billion in AUM raised in nominal terms.

Reuters reports AIA Macro’s 16.4% January-September gain beside Pure Alpha’s 18.4% for the same interval. One unnamed source supplied the AIA figures, including 13.2% annualized since inception and over $4.5 billion in current AUM. Current AUM and the earlier nominal raised-AUM disclosure have different measurement bases. The report does not specify the fee basis, risk exposures, or tool-level return attribution.

The same-period comparison describes two strategy outcomes. According to Bridgewater, Pure Alpha also uses research and tools developed with AIA Labs. It cannot serve as an AI-disabled control. Nor does the public record connect AIA Macro’s return to PAT, the document filter, the SQL model, or this particular forecaster.

What a research team must specify before delegating work

For a team adapting these methods, the work product determines the first check. A document filter needs examples of what this investor would read and what it must not discard. A deal-monitoring agent needs the evidence behind its current probability and rules for revisiting it. A paper-replication or signal agent needs an inspectable link from the stated hypothesis to the variables and code that test it.

Balyasny explains how it tests those dependencies. Flanagan describes evaluating models both alone and within the full agent environment, with users’ tools, files, and requirements. The tests examine numerical mistakes, missed coverage, unsupported conclusions, and retrieval failures. When an economics result improved unexpectedly, the team reran the evaluation and checked the tasks and scoring before accepting it. A team adapting this method can test the research deliverable with its actual dependencies, alongside the model’s isolated performance.

Cheaper preparation and implementation can expand the research agenda. The constraint may then become the team’s ability to distinguish useful results from errors and statistical accidents. A useful adoption test measures reliable research accepted for use, at a total cost that includes expert review. Where the output informs forecasts or portfolio decisions, evaluate its incremental contribution to the existing process.

Sources

Dates below identify publication or announcement, except where a recording, submission or investment event is specified. Dates without a year are in 2026.

• Balyasny: Deal monitoring and shared infrastructure (March 6); deal research and evaluation, Charlie Flanagan interview (September 17).

• Citadel: Ken Griffin interview and transcript (recorded June 2; published July 9).

• Man Group: AlphaGPT’s research workflow (November 13, 2025); Anthropic partnership, naming the existing AlphaGPT (February 11); Man AHL’s earlier ML applications (2019).

• Millennium: Diana Meditz on Digital Twins (September); digital risk analyst development announcement (August). The pages disclose months, without exact days.

• Two Sigma: Feature-research productivity and historical-data risks (January 21); Ben Wellington on generating research datasets (July 9).

• Bridgewater, investor labels and PAT: AIA Labs’ research program; expert-judgment study and original PDF and Thinking Machines’ account (June 30); PAT presentation page (Bridgewater reports recording on May 19) and original video (published July 24).

• Bridgewater, SQL and forecasts: UIUC/AIA Labs text-to-SQL study (August 27); AIA Forecaster technical report, ForecastBench comparisons and MarketLiquid results, Table 7 (first submitted November 10, 2025).

• Bridgewater, investment history and returns: External-capital disclosure (July 1, 2024); Reuters’ first-half account and strategy history; Reuters’ January-September returns (October 1).

• Historical comparisons: AQR’s active use of ML and stock-portfolio study (November 2024); D. E. Shaw’s Neil Katz on shared AI infrastructure (September 10, 2024).

Generating an agent’s practice tasks creates another engineering problem: the tools must work, the requested outcome must be reachable, and the evaluator must recognize it. Otherwise, training can reward an incorrect result or penalize a valid one.

Three published research systems approach that problem differently: Snowflake and UNC’s Agent World Model, Amazon AGI’s AutoGym, and Alibaba’s Qwen team with Georgia Tech’s VHD-Play. Their papers describe environment-generation methods and experiments, rather than each company’s production training pipeline.

What happens in one training episode

A task begins from a saved initial state: documents, database records, inventory, or other objects the agent can inspect and change. The model issues tool calls. Executable software returns observations and records their consequences. The agent’s sequence of actions and observations forms a trajectory; an evaluator scores the attempt using that record and the resulting state.

In reinforcement learning, the training procedure collects multiple attempts and uses their rewards to update model weights. The next batch runs with the updated model. Environment generation is the separate process that supplies the tasks, tools, starting states, and evaluators.

Recorded demonstrations can teach a successful tool-use sequence. An executable environment also lets the model try alternative actions and experience their consequences.

Environment generation produces agent-facing tasks and tools plus a separate evaluator. Training resets the task, collects tool interactions, scores the attempt, and updates model weights.

MLCommons’ September 24 announcement describes this integrated workload for MLPerf Training v6.1’s October submission round: inference, executable software-repair tasks and reinforcement-learning updates, measured by time to a quality target. Results from that round have not yet been reported.

Agent World Model builds database-backed software

Agent World Model, released in February, constructs scenarios and tasks, then SQLite databases, tool interfaces and verification components. Its 1,000 environments execute state changes in code. The released implementation supports resetting the database for another attempt.

One Spotify-like task asks the agent to add a track to a playlist. The verifier can inspect the relevant records before and after the attempt. A successful tool call that changes the wrong playlist leaves evidence in the database.

The reported training setup uses code-augmented model judgment: verification code extracts database evidence, and an LLM judge assesses completion using that evidence and the trajectory. Step-level checks also reward valid tool-call format. The repository offers a code-only verification option, but executable state transitions alone do not make the reported outcome judgment deterministic.

AutoGym derives answers from the constructed environment

AutoGym starts with a solution blueprint, materializes the environment, then executes the blueprint’s derivation against it to compute ground truth. Deterministic checks handle explicit constraints; open-ended outcomes also use process, outcome, and meta-judges. All retained productivity tasks received human validation.

Here, hard means a mean verifier score below 0.30 across eight attempts. In the authors’ seven-tool productivity setup, 217 of 300 AWM-generated tasks survived review; 4% were hard and 90% easy, versus 39% hard for AutoGym’s retained harder configuration. This compares generated task difficulty under one protocol, not overall system performance. Tables 1-2.

Reported generation costs are $2-8 per task, excluding human validation. Learning evidence remains limited to one 8B model trained for 500 GRPO steps. Appendix F and limitations.

VHD-Play separates solving from tool-mediated execution

VHD-Play derives dynamics and a scoring reference from a mathematical mechanism, then generates the scenario and tools. Its outcome reward normalizes achieved utility between a baseline and a solver-derived upper reference, clips it to 0-1, and does so without an LLM judge.

Its five-family diagnostic compares Qwen3.6-35B-A3B before and after 34 training steps:

• Written problem, complete information: 0.962 to 0.992.

• Stateful tools, parameters supplied: 0.231 to 0.875.

• Stateful tools, parameters discovered through interaction: 0.204 to 0.815.

These are mean normalized outcome scores, not success percentages. The starting model struggled with execution even when the parameters were supplied. Table 2.

Of 3,300 admitted environments, 2,200 were assigned to training and 1,100 to evaluation. Marginal generation costs are reconstructed at roughly $0.01-0.03 per admitted environment using public model rates. The upper reference can be unattainable under partial observation, and replay checks vary by family. Sections 3-4 and appendices.

For research systems, the diagnostic suggests an evaluation distinction worth preserving. A model might calculate an optimal allocation from a complete specification yet fail to retrieve the relevant data, maintain state, and commit the right action through an interface. A test that supplies everything in one prompt misses those execution demands.

How construction affects the experiment

Each method places a different requirement on the generated implementation:

• Database-backed generation: state changes and verifier evidence must refer to the same records and task.

• Blueprint-first generation: the materialized documents and tools must support the solution derivation.

• Mechanism-first generation: the tool interface must preserve the dynamics and information restrictions used to compute the scoring reference.

Generation cost also depends on the task contract. The two cost estimates above cover different task families and exclude different parts of the work. A cheap generated record can still require substantial validation before it belongs in an evaluation suite.

A point-in-time data task becomes an agent environment

Consider a proposed retrospective reconstruction task: produce a feature table containing the financial information publicly available on a specified decision date. The agent can inspect the full revision history, including later restatements, but must select only eligible versions. This differs from simulating a historical decision, where future documents should be hidden entirely.

The agent’s tools would let it discover filings, retrieve dated versions, resolve identifiers, inspect revision metadata, and write a feature-table artifact. Define availability in the fixture as the public SEC filing date; a fiscal period’s end and a vendor’s ingestion date are separate timestamps.

The Chapter 4 SEC XBRL fundamentals notebook already illustrates an as-of query using filing dates. Turning it into this agent test would require a versioned fixture, restricted tools, and an independently checked reference table.

The evaluator would inspect values, entity mapping, decision dates, and source-version lineage in the output artifact. Before testing an agent, challenge the evaluator with a later restatement substituted for an original value, a missing observation, and a correct value assigned to the wrong security. Equivalent row ordering should pass.

This checks information reconstruction. Whether those features predict returns requires a separate market-data experiment with held-out periods and implementation costs.

Start with a small collection of such executable tasks. Preserve their initial states and expected outcomes, test the checkers, and use the collection to compare frozen models or configurations. It can support a training decision later, after the evaluation itself is dependable.

Free lesson Thursday; hands-on workshop Saturday

On Thursday, October 1, at 11 am Eastern, the free Can AI agents improve themselves? lightning lesson examines what changes persist between runs and how to evaluate them.

On Saturday, October 3, 10 am-3:30 pm Eastern, Engineering a Multi-Agent Forecasting System applies the evaluation side to a working implementation. We’ll inspect traces, compare agent-count and aggregation choices in the ablation lab, and calculate Brier scores and calibration curves on resolved questions, keeping model weights fixed.

The system recognized a misleading development metric and changed its research policy. But the public leaderboard also guided later choices, so it cannot serve as an untouched final test of the policy change.

For a quantitative researcher, the distinction is familiar. Finding a better strategy and improving the procedure that finds strategies are different achievements. To measure self-improvement, we need to follow the changes from the current result into the agent’s later work.

What gets better?

Self-improvement can reach progressively further into the system producing a result:

• The output improves. The system revises an answer, program, or research result. Evaluation compares the original and revised outputs.

• The agent improves. It retains changes to its memory, tools, code, or model. Evaluation tests whether those changes help on later tasks.

• The improvement process improves. The system revises how it proposes, tests, or selects subsequent changes, then reuses that procedure. Evaluation compares the successors produced by the original and revised procedures under comparable resources.

The experimental standards remain consistent: appropriate controls, protected evaluation, resource accounting, and repeated runs. As the object of improvement expands, the experiment must explain more. A better answer alone cannot show that the agent acquired a reusable skill, or became better at developing skills.

Recursion concerns how a retained change feeds back into subsequent improvement. When an agent revises its improvement procedure, later rounds inherit a different way of producing changes. Reuse establishes that feedback; a controlled comparison tests whether it helps. Acceleration measures progress across cycles, rather than serving as a prerequisite for useful recursive improvement.

How much does the agent control?

Duan and colleagues’ survey, The Last AI Built by Humans, organizes recursive self-improvement around five levels of autonomy. L1 covers executing prescribed improvements, and L2 covers selecting improvement strategies. L3 chooses what experience to learn from, and L4 adapts through deployment feedback. L5 changes the mechanisms governing subsequent improvement. The survey also separates the ability to modify a procedure from evidence that the modification helps under comparable budgets and independent evaluation.

The autonomy levels describe which decisions a system controls. The improvement targets above describe what gets better. An agent can control more of its own development without making better decisions. Both its responsibilities and its measured performance need to be described.

Improving the output

Self-Refine illustrates the simplest case. A model drafts an answer, critiques it, and revises it. The paper reports improvements across seven tasks without additional training. A later task need not inherit any change to the system that produced the revised answer.

Consider a hypothetical factor-research agent that tests 50 variants and selects one. A higher independently validated score for that factor measures an improvement in the output. It leaves open whether the agent will perform better in its next research campaign.

Improving the agent

A lasting improvement requires a change that survives the current task and helps in later work. The change may reside outside the model’s weights.

Reflexion retains linguistic reflections in episodic memory and retrieves them during later trials. Its model weights stay fixed, while the retained reflections can change later behavior. To assess persistent improvement, compare performance with and without those reflections under comparable resources.

RSIAgent makes memory reuse explicit. Its exploration phase writes memory from practice and verification. The reference evaluation resets the environment, freezes the memory, and disables curriculum decisions and memory writeback. The official evaluator’s feedback stays outside the learning agents.

Freezing memory separates learning from evaluation-time reuse. It does not by itself establish transfer to unseen tasks. RSIAgent’s target-conditioned exploration can practice on the target itself. The retained memory also leaves the underlying model and memory-learning mechanism fixed.

To track the agent’s development, identify its mutable boundary: the components it can edit. Record persistence, whether a change survives, and inheritance, whether a later process uses it. Each retained artifact needs a version and a record of the processes that loaded it. The model, evaluator, and selection rules should also be identified, including which remain fixed.

Suppose the factor agent also writes a revised search policy. Loading that policy in the next campaign establishes inheritance. Whether it improves the campaign remains an experimental question.

Isolating the contribution of a retained change

Causal attribution concerns what caused the improvement. Transfer tests where it holds. Evaluator independence requires keeping the final test outside the feedback used during selection.

For the factor agent, compare the original and revised policies using the same model, data access, and candidate budget. Give both access to the accumulated research archive. Otherwise, the revised policy may benefit from knowing which ideas already failed while the original policy repeats them.

An ablation removes a candidate cause while keeping the rest of the experiment comparable. A memory ablation might remove retained reflections or replace them with irrelevant ones. A policy ablation restores the earlier procedure. State what each comparison isolates.

Resources belong in the comparison. RE-Bench, which evaluates AI research and engineering work, compares performance at different time budgets. Equal iteration counts can hide differences in token use, tool calls, or human assistance. For a research agent, count candidate strategies and backtests as well as model expenditure. Report the cost of producing inherited state separately from the cost of using it.

Repeat each condition and report uncertainty across runs. Shared tasks and paired seeds can reduce comparison noise, but repeated runs do not replace evaluation on new tasks. Preserve failed candidates and rejected updates; reporting only the winning run hides how much search produced it.

Transfer asks whether the retained change helps beyond its selection setting. That might mean new tasks from the same distribution, another domain, or a later market period. A later random seed tests repeatability under stochastic variation, not cross-domain transfer.

RRSI, by Peng Xia and colleagues, supplies a concrete transfer test. It evolves an agent’s harness, the prompts, tools, and control logic around a frozen model. The method limits proposed edits, screens benchmark-specific changes, and prunes changes with insufficient benefit. In its workspace ablation, unregularized evolution scores 92.8 on the optimization set versus RRSI’s 90.5. Their average scores on three unseen benchmarks are 40.3 and 43.6, respectively. RRSI uses fewer policy tokens than unregularized evolution, but more than the starting harness. The result supports transfer from a designer-specified improvement procedure; it does not establish that the system autonomously improved that procedure.

RRSI trades a lower optimization-set score for better results on unseen benchmarks. Token counts measure inference per trial, not the full search budget. Source: RRSI v1, Table 2.

For the factor agent, select the policy on development data, freeze it, and compare policies on a later period. Record whether performance persists across instruments or market conditions. A gain confined to one regime leaves its usefulness in other regimes untested.

The final test must also remain outside the feedback used to select the policy. If an agent repeatedly queries a nominal test grader and adapts to its scores, the grader participates in development. Indirect feedback through a verifier or a human selector can create the same problem.

Anthropic’s Automated Weak-to-Strong Researcher provides a documented example. Nine agents worked for five days with unlimited submissions to an outcome grader. Anthropic reports that this effectively turned the test set into a validation set. Agents exploited the access through seed selection and, in one case, probing uncertain examples with both labels.

A sealed final benchmark or prospective outcomes can limit that exposure. An independent review team helps only if its evidence and feedback are protected too. Changing the people who score a reused dataset does not restore its independence. Record who could query each evaluator, what feedback they received, and when selection stopped.

Improving the improvement process

Meta-improvement means improving the procedure that produces later changes. An agent might revise its experiment-selection rule, then use that rule to direct subsequent research. Measuring the effect requires comparing the research produced by the original and revised procedures.

One strong design compares three conditions under matched resources:

1. The original procedure with its original evidence.

2. The revised procedure with accumulated evidence.

3. The original procedure with that same accumulated evidence.

The third condition helps separate a procedural improvement from access to a better archive. For the factor agent, it asks whether a new experiment-selection rule helps after both policies know which factors were already tried. Other controlled interventions can isolate the same effect.

Hyperagents tests a changed improvement procedure directly. Its transfer experiment freezes evolved meta-agents and lets them generate agents for Olympiad math grading. After 50 iterations, the reported imp@50, the test-score gain of the best validation-selected descendant over its starting agent, is 0.630 (confidence interval 0.540-0.630). The initial-meta-agent comparison has a gain of zero. This supports transfer of improvement ability under the experiment’s fixed iteration budget.

A separate 200-iteration experiment allows the meta-agent to keep evolving. Starting from transferred agents yields 0.640 versus 0.610 from the initial agent, a difference the authors report as statistically insignificant. These are different protocols, not successive observations on one learning curve. Faster learning after transfer does not by itself establish a sustained increase in the rate of improvement across cycles.

A-Evolve’s inherited policy bundles a standing hyperparameter recipe with search instructions. Its baseline worker controls noise against a fixed training substrate, but does not isolate the effect of the revised policy given the same accumulated evidence. The reported four-round campaign supports policy inheritance and adaptive search; attributing its gains specifically to meta-improvement needs a further controlled comparison.

Measuring acceleration

A better improvement procedure can yield a one-time gain or continuing gains that diminish as the remaining problems get harder. Recursion describes the feedback into later improvement; acceleration measures whether progress per unit of resources increases across cycles.

Measuring acceleration requires a sequence of comparable cycles, with each cycle’s gain related to its resource use. Specify the performance measure and resource unit before interpreting the trend. Report raw scores alongside normalized gains, and account for changes in models, tools, task difficulty, and human intervention.

The measurement scale also matters. Bounded scores leave less headroom as performance rises. Transforming the scale can change the apparent curvature, so a headroom or log-odds transformation needs justification. Neither a flattening score curve nor a favorable transformation alone settles whether the improvement process is accelerating.

The experiments discussed here support different forms of retained improvement and procedure transfer. They do not establish a sustained increase in resource-adjusted improvement rate across comparable recursive cycles.

Comparing the experiments

• Self-Refine

  Retained change: None required across tasks.
  Demonstrated result: Better current outputs.
  Remaining question: Does a later task benefit from retained state?

• RSIAgent

  Retained change: Frozen exploration memory.
  Demonstrated result: Later execution uses learned memory.
  Remaining question: What transfers beyond target-conditioned practice?

• RRSI

  Retained change: Agent harness.
  Demonstrated result: Gains on unseen benchmarks.
  Remaining question: Does the improvement procedure itself improve?

• Hyperagents

  Retained change: Task-agent and meta-agent code.
  Demonstrated result: Procedure transfer at a fixed iteration budget.
  Remaining question: Does improvement per resource unit accelerate?

• A-Evolve-Training

  Retained change: Recipe and search policy.
  Demonstrated result: Inherited policy revision.
  Remaining question: Does the revised procedure help under matched evidence?

The comparison summarizes the specific experiments described above. It does not assign a general capability level to each system.

What to report

Before optimization, specify whether the experiment measures changes in the output, the agent, or its improvement process. Define the performance measure, fixed components, resource budget, and protected final test. A useful run record covers:

1. Mutable boundary: what could change, what stayed fixed, and who could intervene.

2. Persistence: the retained artifacts and when each was frozen.

3. Inheritance: the later runs that loaded those artifacts and the decisions they affected.

4. Causal attribution: ablations, resources by condition, repeated runs, and uncertainty estimates.

5. Transfer: tasks, domains, or periods excluded from selecting the change, with results reported separately.

6. Evaluator independence: all feedback available during search, query counts, access controls, and the final evaluation procedure.

7. Meta-improvement: the comparison isolating the revised procedure from accumulated evidence and extra resources.

8. Progress across cycles: scores, resource use, and inherited changes, including unsuccessful updates. A study of acceleration also compares the rate of improvement across those cycles.

A study of output quality need not demonstrate meta-improvement. Reporting which parts of the system were evaluated keeps a useful result interpretable without implying that every part improved.

Measure a forecasting agent

On Thursday, October 1 at 11:00 a.m. Eastern, my Lightning Lesson, Can AI agents improve themselves?, maps the improvement loop onto a forecasting agent and examines how to measure changes in its performance and research process.

For hands-on practice with an auditable multi-agent architecture, Engineering a Multi-Agent Forecasting System on Saturday, October 3 compares configuration profiles on the same questions. You will toggle the supervisor, change aggregation methods, and examine Brier scores and calibration on resolved questions.

Today, multiple systems claim different degrees of RSI success using different approaches, for example, by revising an answer before returning it, storing lessons for the next attempt, or evolving executable programs against a score. Stronger claims involve systems that edit agent code, update model weights, revise the research policy used by a later round, or change the evaluator between selection periods. Each result requires different evidence and has different implications.

Current experiments support real progress: agents can persistently modify memory, code, research policies, harnesses, and weights. Some of these changes produce downstream performance gains. Recent experiments also change the procedure that produces later improvements. However, public evidence has yet to establish a fully autonomous self-accelerating successor process.

A fully autonomous successor process would control proposal, evaluation, selection, persistence, and modification of the machinery that produces its next successor. It would test final capability on evidence withheld from every adaptive step and report the compute, samples, tools, and human input used along the way. Self-acceleration requires one more result. Inherited changes must increase the rate of later improvement under comparable external inputs. No paper below demonstrates this entire process.

For quantitative researchers, the evaluation problem is already immediate. An agent can generate more factor expressions, model configurations, and backtests than a human team can inspect. Repeated access to a score can reward research progress, additional search, leakage, or adaptation to the evaluator. Persistent memory can preserve the error as easily as the insight.

The reading path therefore follows four questions: what changed, what persisted, what stayed fixed, and which evidence remained unavailable during selection?

How self-improvement became more recursive

The first dividing line is persistence. Revising one answer can improve the current output while leaving the next attempt unchanged. Memory, code, or weights can carry an improvement into later work. The stronger recursive cases also change part of the procedure that proposes, tests, or selects the next change.

Selection and final evaluation need separate labels. A search metric directly selects candidates. An external metric becomes part of the adaptive loop when later rounds respond to it. The strongest test remains unavailable until the system and its learned state have been frozen.

From intelligence explosion to formal self-rewrite

The modern argument traces back to I. J. Good’s 1965 intelligence-explosion argument. If designing intelligent machines is an intellectual activity, Good reasoned, a sufficiently capable machine could help design a more capable successor. The argument did not specify how a system should decide that a proposed self-modification is an improvement.

Jürgen Schmidhuber’s Goedel Machine (2003) supplied a formal answer. A utility-maximizing program searches for a self-rewrite and executes it only after proving that the rewrite has higher expected utility than continuing the search. The permitted rewrite can include the proof searcher itself.

Modern systems replace proof with experiment. They propose changes, run an evaluator, retain promising results, and repeat. This makes self-improvement practical, but places much of the epistemic burden on evaluation. A system can become better at optimizing a benchmark without improving the broader capability the benchmark is meant to measure.

From revision to persistent experience

Self-Refine (2023) provides a useful lower bound. The same language model generates an answer, critiques it, and revises it. The paper reports an average absolute improvement of about 20 percentage points across seven tasks, but nothing needs to persist after a task ends.

Reflexion (2023) crosses that boundary. It stores natural-language reflections from earlier attempts and retrieves them in later trials while model weights remain fixed. Useful lessons can compound across trials, and misleading reflections can trap later behavior in a local optimum.

RSIAgent (2026) also adapts the experience used to construct persistent memory. Its final evaluation freezes that memory, resets the environment, and disables curriculum decisions and writeback. The system therefore adapts future context without adapting the fixed mechanisms that select practice, verify outcomes, and update memory. The survey The Last AI Built by Humans (2026) offers a useful taxonomy for such degrees of autonomy, but does not independently validate the systems it classifies.

From better artifacts to better improvers

Self-Taught Optimizer (2023) starts with an “improver” program that uses a language model to improve other programs, then asks that improver to improve itself. The model stays fixed, but the program that proposes later improvements becomes mutable. Promptbreeder (2023) explores a related mechanism by evolving both task prompts and the mutation prompts that generate them.

FunSearch (2024) and AlphaEvolve (2025) follow a different route. Models propose executable programs, automated evaluators score them, and successful programs become material for later generations. FunSearch pairs a frozen model with a fixed evaluator and program database. AlphaEvolve extends that pattern across mathematics, algorithm design, and production systems. In both cases, the models, evaluators, and outer evolutionary procedure remain externally specified. Andrej Karpathy’s autoresearch (2026) makes a compact version of this workflow inspectable, but it is not a controlled study.

The Darwin Goedel Machine (2025) moves the mutable boundary from a task program to agent code. Selected descendants enter an archive and can help produce later descendants. The paper reports gains from 20.0 to 50.0 percent on its SWE-bench evaluation and from 14.2 to 30.7 percent on Polyglot. Foundation models, archive operations, parent selection, and benchmark validation remain outside the recursive relation.

Hyperagents (2026) places the task agent and meta-agent inside one editable program. Its transfer experiment freezes an evolved meta-agent and tests whether it can produce better task agents in a different domain under a fixed iteration budget. The result tests whether a changed improvement procedure transfers, while tasks, budgets, validation, and outer selection remain fixed.

Figure 1. Selected systems differ in the artifact they retain and in the distance between selection and final evaluation. Freezing learned state, transferring an improver, and withholding the final test support progressively stronger claims.

Autonomous research exposes the evaluation path

Automating a research cycle does not establish that the research process is improving itself. The AI Scientist (2024), for example, automates idea generation, experiments, paper writing, and simulated review under a fixed workflow. Evidence for recursive improvement requires a later research round to inherit a changed research procedure.

A-Evolve-Training (2026) reports four autonomous post-training rounds on a 30B model with eight workers per round. A collector aggregates results, and a meta-agent revises the research policy for the next round. The final model scored 0.86 on the public leaderboard against 0.87 for the top human submission and ranked eighth among roughly 4,000 entries at the reported snapshot.

The collector fetched that leaderboard between rounds, and the meta-agent used the development-to-leaderboard gap to redirect later search. The leaderboard was therefore an adaptive selection signal, not an untouched final test. The campaign shows inherited policy revision under a fixed substrate, constitution, objective, worker structure, and evaluation harness.

SIA: Self-Improving AI with Harness & Weight Updates (2026) alternates changes to the task harness and model weights, but its inspected LawBench test grader also optimizes. Red Queen Goedel Machine (2026) changes the evaluator between epochs while holding it fixed within an epoch. Both expose the same requirement: an independent criterion is needed to show that optimizing or changing an evaluator improved the intended objective.

Research comparisons also depend on budget and evaluator access. RE-Bench (2024) shows that relative human-agent performance changes with the time horizon. Anthropic’s Automated Weak-to-Strong Researcher (2026) used nine agents for 800 cumulative hours and allowed repeated queries to the remote score. Its headline result demonstrates productive automated search, but part of the nominal test signal became validation.

OpenAI’s internal report (2026) records 3.1 agent workdays per human workday by mid-August. Researchers wrote more code and ran more experiments while available compute also increased. High-level planning remained a small fraction of agent activity, and more than half of successful four-to-eight-hour tasks involved at least one human intervention. These measurements establish greater automation of research execution without isolating its effect on model progress.

Finance turns evaluation into a statistical problem

AQuA (2026) contains separate systems for symbolic-factor search and model-configuration search. Each retains validated evidence and uses it to guide later proposals. The sandboxes fix data splits, features, labels, evaluator logic, and allowed actions. Because the experiments do not include a compute-matched condition without persistent memory, later gains can reflect retained evidence, additional search, or both.

Agentic Empirical Asset Pricing (2026) makes a valuable methodological distinction. An output backtest evaluates one factor or strategy selected after a search. A system backtest reruns the full discovery process at historical decision dates and asks whether that process could repeatedly produce usable outputs.

Its SEADS implementation retains attempt logs, reflector lessons, and an admitted-factor library. Component ablations show that these mechanisms affect performance, but the rolling comparison does not isolate persistence from other system differences. Mutation retries add unmatched compute, historical runs use a current model, headline Sharpe is gross of costs, and most cross-system differences are not reliably distinguishable at the reported sample size.

The Self-Driving Portfolio (2026) describes a meta-agent that compares forecasts with realized returns and can revise prompts, skills, and Python code. The authors report insufficient out-of-sample evidence to determine whether those revisions improve later allocation. The paper specifies an update mechanism without establishing its downstream effect.

Markets add non-stationarity, multiple testing, revised-data leakage, and transaction costs to every evaluation path. An adaptive research system needs an independent final test and a record of failed trials. Otherwise, it can become a more efficient optimizer of the historical score it repeatedly sees.

Economic acceleration is a separate claim

The Economics of Recursive Self-Improvement (2026) models feedback between research capability and later improvement. Its calibration depends on assumptions about research production, compute, substitution, transfer, and diminishing returns. The model identifies conditions that could support sustained acceleration, but does not measure a universal empirical growth law.

Numerical takeoff probabilities require a separate analysis. They depend on which research tasks become automatable, how human judgment and compute enter the production function, and whether improvement procedures transfer to the systems that create later models.

What current evidence establishes

Current systems can:

• retain changes in memory, code, research policies, and weights beyond a single attempt;

• improve executable artifacts under automated evaluation;

• modify parts of the machinery that produces later improvements and transfer a modified procedure in bounded experiments; and

• automate bounded research loops on human-selected, outcome-gradable problems.

Public evidence does not yet establish:

• repeated construction and independent validation of more capable foundation model successors;

• improvement of the full outer loop, including agenda selection, objectives, and final evaluation;

• sustained transfer beyond the optimization environment; or

• acceleration caused by inherited system changes after accounting for human labor, compute, data, and task selection.

The next issue will turn these distinctions into an audit for self-improvement claims.

A compact reading path

Eight sources provide the shortest route through the technical progression:

1. I. J. Good, “Speculations Concerning the First Ultraintelligent Machine” (1965) for the intelligence-explosion argument.

2. Goedel Machines (2003) for the formal self-rewrite problem.

3. Reflexion (2023) for persistent linguistic memory without weight updates.

4. Self-Taught Optimizer (2023) for a program that improves its own improvement procedure while the model stays fixed.

5. AlphaEvolve (2025) for evaluator-driven evolution of executable programs.

6. Darwin Goedel Machine (2025) for inherited changes to agent code.

7. Hyperagents (2026) for an editable meta-agent and transfer test.

8. A-Evolve-Training (2026) for inherited research-policy revision across post-training rounds.

For quantitative research, add AQuA (2026) for persistent experimental evidence and Agentic Empirical Asset Pricing (2026) for rolling evaluation of the discovery system itself.

Learn how to inspect the system

On Thursday, October 1 at 11:00 AM ET, I am teaching a free 30-minute lightning lesson: Can AI agents improve themselves?. We will map current systems by what they change, what they keep fixed, and what their experiments establish.

The Agent Engineering workshop follows on Saturday, October 3. Participants build and evaluate an auditable multi-agent forecasting system with traces, ablations, probabilistic scores, and calibration checks.

The largest gain came from risk-metrics-calculation. The skill covers drawdown, value at risk, and other standard risk measures. It took ten tasks from 70.0% to 100.0% and reduced token use by 34.8%.

The ML4T skills are now public

We built the ML4T skill collection around procedures that can fail silently, producing wrong results rather than throwing errors. Today, we are making all 61 skills public on GitHub under the Apache 2.0 license. They cover the end-to-end research process, from point-in-time data and feature engineering to validation, backtesting, portfolio construction, production, and agent engineering.

The ten-task result for the risk-metrics-calculation skill does not establish a general effect. It supports a useful hypothesis: skills help when an agent does not reliably know the required procedure. Quantitative research adds a second possibility. A skill may help more when the environment does not expose a procedural mistake. A wrong risk calculation still returns a number.

What an agent skill contains

A skill is a Markdown file with a name, a routing description, and a procedure. The agent reads the description first and loads the full procedure when it judges the skill relevant. Anthropic calls the approach progressive disclosure. The company introduced Agent Skills in October 2025 and published the format as an open standard in December.

Reusable procedures predate the current file format by decades. Korf’s 1985 macro-operators gave search algorithms reusable sequences of moves. Sutton, Precup, and Singh’s 1999 options framework did the same for extended actions in reinforcement learning. Researchers can now inspect, share, and test a procedure in a standard, readable file.

Why two benchmarks report different gains

SkillsBench 1.1 contains 87 tasks across eight domains. Benchmark authors wrote a relevant skill for each task and used deterministic checks to score the result. Across 18 model/harness combinations, the average pass rate rose from 33.9% to 50.5%, a gain of 16.6 points.

SWE-Skills-Bench tests a different setting. It evaluates public skills in repositories that their authors did not target. The benchmark generates each requirement from the skill, then selects a repository with the relevant technology. A fixed prompt generates the pytest verifier. The resulting test runs deterministically, although a model wrote it.

SkillsBench asks how much a task-specific skill can help. SWE-Skills-Bench asks how well a public skill transfers to a new repository. Skill authorship, task mix, model, harness, baseline, and scoring method all differ.

The starting pass rate also matters. SWE-Skills-Bench began at 89.8%, leaving 10.2 percentage points available to improve. Its 1.2-point gain closes about 12% of that gap. SkillsBench began at 33.9% and closes about 25% of the available gap. The apparent fourteen-fold difference falls to roughly twofold after this adjustment. The studies remain too different for a direct comparison.

Results also vary within the same benchmark. In SkillsBench 1.1, average gains range from 9.7 points in Mathematics and OR to 28.8 in Natural Science. Thirteen of the 87 tasks get worse with a skill. In SWE-Skills-Bench, broad or version-specific guidance sometimes distracts the agent from the immediate task.

NVIDIA reports larger gains on a third type of evaluation. Its SkillEvaluator benchmark covers more than 300 skills for over 30 NVIDIA products. Average correctness rose from 46 to 87 on a 0-100 rubric. NVIDIA writes the skills, owns the product documentation, and builds the evaluation. The post reports no confidence intervals, and most skills use one attempt per task.

Quantitative research has silent failures

Software agents receive feedback while they work. A compiler rejects a type error, a test fails, or a service does not start. The agent can use the signal to revise its work.

Many research errors produce valid code and plausible output:

• Cross-validation leakage. Training labels or feature windows overlap the test period. The backtest runs and the estimated Sharpe ratio can rise.

• Pre-split standardization. A scaler fitted on the full dataset puts future information into the training data. No runtime error identifies the leak.

• Current index membership. Applying today’s constituents to earlier dates excludes many firms that later left the index. The resulting backtest contains survivorship and constituent-selection bias.

• The wrong cost base. Charging costs on the position instead of the trade overstates costs in a 4% to 5% rebalance. Other denominator errors can understate them. The tearsheet still renders.

A researcher has to state the relevant invariant and test whether it held.

Skills, libraries, and verifiers serve different roles

A skill gives the agent a procedure before it starts. A library provides a tested implementation while the agent works. A verifier checks the process or result. Repeated failures may require all three.

A skill instructs before execution, a library constrains implementation, and a verifier detects violations.

For example, a skill can tell the agent to fit a scaler inside each training fold. A library can expose an API that accepts fold-local data. A verifier can record which rows reached fit and reject any call that included test data.

How we designed the ML4T skills

We applied the benchmark findings directly:

• Narrow scope. Each skill covers one research procedure and stays within 120 lines. Short files reduce irrelevant context.

• Explicit routing. Every description states when the agent should load the skill, so the agent can select it before loading the full procedure.

• Concrete failure examples. Every skill shows a plausible wrong approach and a corrected one. The examples focus on stable research invariants rather than fixed API versions or parameter values.

• Library integration. Fifty skills include a Production Implementation section that shows the relevant ML4T library. CI can check the imports and function signatures against the library source.

• Instruction-only skill directories. Each directory contains one SKILL.md, with no executable hooks or companion scripts.

The collection includes procedures for leakage, lookahead bias, point-in-time data, non-stationarity, feature validation, backtest costs, portfolio risk, and live-trading controls. It also includes end-to-end research workflows and skills for building reliable research agents.

How we are evaluating the collection

Our March pilot tested five paired tasks, with one run per condition. All ten runs passed. Runs with skills used 1,253,830 tokens, compared with 1,212,535 without skills. The pilot established a cost baseline and showed that the next benchmark needs harder tasks and stronger verifiers.

The next evaluation starts from silent research failures. Each task will require the relevant procedure. Deterministic checks will inspect data splits, training inputs, or point-in-time records where possible. Research decisions that resist a deterministic rule will need a judge calibrated against human labels.

We will run repeated skill and no-skill trials on the same tasks. The report will show where each skill improves correctness, how consistently it does so, and what it costs in tokens. The results will also show when a library constraint or verifier is more effective than written guidance.

Get the skills

Inspect any public skill before installing it. One marketplace study analyzed 31,132 skills and estimated that 26.1% contained at least one vulnerability. Executable scripts increased the observed risk, but harmful instructions can also live in Markdown.

The ML4T skills are public under Apache 2.0:

git clone https://github.com/ml4t/skills.git ~/.ml4t-skills
~/.ml4t-skills/scripts/install.sh

Install the collection, inspect the files, and use the skills that fit your research workflow. We will publish the evaluation results as we measure each skill’s effect on correctness, consistency, and cost.

Consider an agent asked to screen a US equity universe for companies with material exposure to data-center power demand and recommend two for further underwriting. The system has to interpret “material exposure,” establish the universe and period, and identify companies and sources. It then retrieves documents and data, extracts comparable evidence, calculates relevant quantities, resolves contradictions, decides when it has enough evidence, and synthesizes the result.

A failure anywhere in that sequence can change the recommendation. The agent can search the wrong period, miss a filing, or retrieve the right evidence but exclude it from working state. It can confuse reported and adjusted figures, calculate a ratio with inconsistent denominators, or stop before investigating a contradiction. Its conclusion may fail to reflect the evidence it collected.

The language model is one component. The useful unit for evaluation is the running research system: models, prompts, orchestration, tools, data sources, retrieval, working state, computation, runtime policies, and the mechanisms that turn all of these into analysis or decision support.

This system view changes the evaluation problem. The questions become whether each part of the research process works reliably, whether failures can be localized, and whether an observed failure points toward a controlled next experiment.

What a financial research agent actually does

Implementations differ. Some systems use one model repeatedly inside a tool loop. Others route work among specialized agents for research, financial analysis, valuation, review, and report generation. The distinction matters less for evaluation than the functions the system has to perform.

A useful functional decomposition has six stages.

The first is task interpretation and planning. The system translates a research request into an operational task: universe, period, subquestions, required evidence, analytical operations, and a stopping condition. A request such as “compare the economics of these companies” requires decisions about peer definitions, business segments, time periods, accounting measures, and which economic differences matter.

The second is information acquisition. The agent searches filings, earnings materials, transcripts, market data, company disclosures, economic series, industry sources, internal research, or other approved data. It also has to find the right document, parse it, identify the relevant passage or table, and respect the date when information became available.

The third is an evidence workspace. Useful research requires more than appending retrieved text to a context window. The system needs a representation of what it has established: extracted facts, source provenance, dates, unresolved questions, conflicting observations, intermediate calculations, and hypotheses that require additional evidence.

The fourth is computation and tool use. Financial research routinely requires deterministic operations that should be inspectable outside the language model: ratios, growth rates, valuation calculations, time-series transformations, reconciliations, screening operations, table extraction, and sometimes code execution. The inputs, transformation, and output should be preserved wherever practical.

The fifth is analysis and judgment. The system compares evidence, reconciles discrepancies, considers alternative explanations, applies the relevant financial concepts, and decides what the evidence supports. This stage may send the agent back for another filing, another calculation, or evidence on a competing hypothesis.

The sixth is verification and synthesis. Before returning the result, the system checks coverage of the original request, connects claims to sources, surfaces relevant uncertainty or disagreement, and produces the report, recommendation, estimate, or other decision support.

The resulting system is iterative. Analysis creates new research questions. Calculations expose missing inputs. Conflicting sources require another retrieval pass. The orchestrator selects the next operation and decides when to stop.

A financial research agent runs an iterative workflow over shared state and configuration. Instrumentation spans every stage so the team can evaluate components, trajectories, and repeated runs.

The figure is a functional model rather than a prescription for one implementation. A recent survey of deep research agents describes planning, multi-hop retrieval, iterative tool use, code execution, memory, and structured report generation across static and dynamic workflows and single-agent and multi-agent systems. The open Finance Agent v2 harness provides one concrete financial example, with web search, SEC EDGAR access, page parsing, stored-information retrieval, price history, and a calculator.

The architecture provides an evaluation map because each function introduces a different class of failure.

Evaluate the system where it can fail

A single aggregate score cannot distinguish these failures.

• Task interpretation: Verify the universe, period, definitions, and requested output with coverage checks and explicit task decomposition. Otherwise, the agent may screen the wrong companies or answer only part of the request.

• Retrieval: Measure required-source recall, retrieval success, and date validity. A missed filing or transcript leaves downstream components without required evidence.

• Evidence selection and state: Inspect lineage, workspace contents, and provenance coverage. A relevant fact may be retrieved and then discarded or buried.

• Calculation: Recompute results independently and test formulas, units, and tolerances. Correct revenue figures can still produce the wrong growth rate.

• Analysis and judgment: Use counterfactual and sensitivity tests. A material risk disclosure should affect the risk assessment in a defensible direction.

• Orchestration: Analyze trajectories, tool errors, and stopping behavior. Repeated searches can consume the budget while a missing calculation remains undone.

• Synthesis: Grade required criteria and check citations. Correct research can still omit a requested company or material qualification.

• Repeated execution: Run repeated trials and record failure frequency and decision variance. Similar tasks should succeed consistently.

• Evaluator: Compare automated scores with independent finance-expert judgments and adjudicate consequential disagreements.

The metric depends on the task. A retrieval test can measure whether required sources or facts were recovered. A valuation task can reproduce calculations and impose numerical tolerances. A company screen can test coverage of the eligible universe and verify inclusion criteria. A forecast can be evaluated with proper scoring rules and calibration. A long report can be decomposed into required claims. A recommendation can be subjected to scenario or evidence-sensitivity tests.

These operations require different metrics. They share one engineering requirement: observability.

Instrument the research process before evaluating it

A research team cannot diagnose a system from the final answer alone.

For each run, preserve enough information to reconstruct what happened. Record the model and version, system and task prompts, tool definitions, query date, source access rules, searches, retrieved documents, extracted passages, API data, and calculations with their inputs. Also record errors, time and token budgets, tool-call limits, stopping reason, total cost, and final output.

This does not require exposing a model’s private internal reasoning. It requires recording the system’s observable actions and artifacts.

Suppose an agent’s report omits Intel’s decline in notebook volume while explaining its revenue slowdown. The system may never have found the relevant filing. It may have retrieved the filing but failed to identify the passage. It may have extracted the fact but failed to retain it in the evidence passed to analysis. The analysis may have had the evidence and given it no weight.

The final report looks the same in all four cases. The next experiment should be different.

AgentBoard provides useful methodological precedent. Its authors argue that final success rates reveal little about intermediate agent behavior, then add a progress-rate metric over intermediate states or annotated subgoals and trajectory analysis. Its environments cover embodied, game, web, and tool tasks rather than financial research. The engineering implication still applies: intermediate observations distinguish systems that receive the same final score after materially different runs.

Use report-level grading for what it can measure

FrontierFinance, released in August 2026 by seven Samaya AI researchers, addresses one part of the architecture: evaluating whether a long financial research report covers the facts, calculations, and qualifications required by the task.

The benchmark contains 220 open-ended finance research tasks spanning six parts of the investor workflow. The authors decompose them into 11,543 binary rubrics written and reviewed by finance professionals.

One released question asks what caused Intel’s revenue slowdown. The rubric separately checks reported revenue for 2023 and 2024, notebook and desktop volumes, demand explanations, pricing, and other evidence needed for the causal account. Must-have labels identify especially consequential criteria.

The public questions average 52.5 rubrics, with a median of 27 and a range from 3 to 475. Each report receives the fraction of its criteria that it satisfies, and the benchmark averages those scores by question. A screening problem containing several hundred checks therefore does not dominate a smaller company-research question because it has more rubric items.

This decomposition has support outside finance. CheckEval, an EMNLP 2025 study across 12 evaluator models, reports stronger correlation with human judgments, higher evaluator agreement, lower variance, and more traceable decisions from checklist-based evaluation.

DeepResearch Bench makes a related separation. Its RACE framework evaluates report quality against task-specific criteria and a reference report. Its FACT framework separately examines factual abundance and citation trustworthiness. The benchmark supplies methodological support for keeping report quality and information acquisition in separate scores. It does not provide evidence about financial-agent performance.

Criterion-level grading tells the team which required claims or calculations are missing. It does not identify why they are missing.

The model and the agent are different units of comparison

The system view also changes how leaderboard results should be interpreted.

FrontierFinance compares models using native web search, models inside the open Finance Agent v2 harness, and Samaya’s in-house system, which uses its own models, tools, retrieval engines, and financial data index.

The strongest reported native-search configuration satisfies 33.0% of all criteria. The leading Finance Agent v2 configuration reaches 49.2%. Samaya’s high-effort system reaches 56.0%.

Those results show that system configuration affects performance. They provide limited evidence about which component caused the difference.

Samaya’s 6.8-point advantage over the strongest Finance Agent v2 configuration combines changes in models, retrieval, tools, and data infrastructure. A team deciding whether to change its retriever, tool interface, model, context construction, or stopping policy needs controlled comparisons within its own system.

The configuration therefore belongs in the result. Record the model and version, prompts, tools, data providers, retrieval settings, inference parameters, orchestration rules, context limits, tool-call budgets, query date, evaluation date, and cost.

Finance adds an important temporal requirement. A system answering a historical investment question must exclude information that became available afterward. FrontierFinance gives every system the query date in its prompt, while Samaya also applies that date as a retrieval cutoff to its data index. These mechanisms provide different controls, and a model may have absorbed later information during training.

Diagnose the trajectory, not the tool-call count

Execution traces become useful once a report misses a criterion.

FrontierFinance provides a simple example. Claude Fable 5 leads the Finance Agent v2 group at 49.2% while averaging 16.6 tool calls per question. GPT 5.6 Sol averages 46.3 tool calls and scores 46.8%. More activity does not imply more complete research.

The benchmark also examines failed page retrievals. It classifies a URL as originating from parametric knowledge when its domain did not previously appear in the run’s search history. Those direct attempts have higher access-error rates than search-discovered URLs. Invalid addresses and pages that block crawlers consume the same finite execution budget as useful research.

A trajectory diagnostic should ask what the actions accomplished. Did searches add relevant evidence? Did the system return repeatedly to the same sources? Did tool failures trigger a sensible alternative? Was a calculation attempted once the required inputs were available? Did the agent identify unresolved questions? Did it stop because the requested evidence had been collected or because the run budget expired?

For the Intel example, suppose the trace shows that the filing and notebook-volume decline were both retrieved before the report was written. Another retrieval query is unlikely to fix the omission. The failure has moved downstream.

Retrieval and evidence use are separate tests

A source appearing in a report does not establish that its contents affected the analysis.

RAGChecker separates retriever properties such as claim recall and context precision from generator properties such as context utilization, faithfulness, and sensitivity to irrelevant information. Its experiments cover general retrieval-augmented generation, so the metrics require validation on financial research tasks.

Lost in the Middle shows that models’ use of relevant information can depend on where it appears in a long context. Access to the document does not imply uniform use of its contents. The experiments use multi-document question answering and key-value retrieval rather than investment research, but they identify a component behavior worth testing.

A new finance-specific preprint makes the distinction more consequential. Reading Is Not Using holds a focal risk disclosure constant while expanding unrelated filing context from 2,000 to 128,000 tokens. The authors report that direct retrieval of the disclosure remains strong while its influence on a downstream judgment approaches their experimental noise floor.

In their experiments, placing a targeted structured extraction immediately beside the decision restores more of that influence. Chunk-then-aggregate processing and instructions to reason further do not.

The paper is explicitly a very preliminary draft. Its primary experiments use constructed disclosures, its tests on actual filings are exploratory and capability-dependent, and a public replication package is still pending. The result is best treated as a hypothesis for evaluation.

First determine whether the agent can retrieve and identify the relevant evidence. Then change one material fact under controlled conditions and measure whether the downstream estimate, ranking, recommendation, or risk assessment moves in the expected direction.

A system can pass the first test and fail the second.

Test the same system more than once

Agent trajectories are stochastic. Search results vary, models sample different actions, sources become temporarily inaccessible, and an early choice can redirect the rest of a long run.

One successful execution provides weak evidence of reliability. Run important evaluation tasks repeatedly and measure the distribution of results. At minimum, record task success, criterion coverage, critical failures, cost, tool-call count, and whether the final decision changes.

τ-bench introduced pass^k to estimate the chance that an agent succeeds on all k repeated trials of the same task. Its retail and airline customer-service domains are far removed from investment research. The reliability principle applies directly: average single-run success can conceal a system that fails intermittently under the same underlying task.

Suppose an agent recommends Company A in seven runs and Company B in three despite having access to the same source set and evaluation date. The variation may reflect a close call supported by the evidence. It may instead come from one run missing a filing, another using a different denominator, and a third stopping before completing the peer comparison. The evaluation should reveal which case occurred.

This also changes how small benchmark differences should be interpreted. A one-run comparison between two configurations provides less information than repeated runs that estimate failure frequency and outcome variance on the same tasks.

Validate the evaluator too

Once long reports are decomposed into criteria, another system still has to determine whether each criterion was satisfied.

FrontierFinance uses majority vote across GPT 5.4, Gemini 3.1 Pro, and Claude Sonnet 4.6. The authors report that this three-model panel closely matched a nine-model committee in preliminary experiments. The paper does not report agreement with finance experts on the released tasks.

That leaves one component of the evaluation system unvalidated against the intended professional standard.

Model-based graders have documented failure modes. The MT-Bench and Chatbot Arena study, for example, found position and verbosity biases and weaknesses on some reasoning tasks. Its pairwise evaluation setup differs from FrontierFinance’s binary pointwise grading, so those findings do not establish that FrontierFinance has the same biases.

The appropriate test is local. Sample tasks across research types and difficulty levels. Have finance professionals grade the same criteria independently. Measure agreement, inspect disagreements that would change a deployment decision, and revise ambiguous rubrics or grading prompts before relying on the automated evaluator at scale.

The grader is another component of the system.

Define the evaluation contract before the run

These tests become easier to manage when the research team specifies them before comparing systems.

• Task and decision: The research task, universe, output, and decision it informs.

• Required evidence and behavior: Required claims, sources, calculations, tolerances, coverage, temporal rules, and sensitivity tests.

• System configuration: Models, prompts, tools, data, retrieval, orchestration, budgets, dates, versions, and cost.

• Component observations: Retrieval, evidence state, calculation, analysis, trajectory, synthesis, repeatability, and grader results.

• Follow-up experiment: One component change, the controls held fixed, and the result that would change the decision.

Return to the data-center-power screen.

Before the run, define the equity universe and what qualifies as material exposure. Specify the evidence required to advance a company, the allowed source hierarchy, the relevant date, and calculations that must reconcile. Decide which omissions invalidate the result.

Then instrument the system. If a company is missing because its filing was never discovered, test retrieval. If the filing was found but the relevant segment disclosure was not extracted, test document processing. If the correct inputs were preserved but the exposure calculation was wrong, test computation. If the evidence was available and correctly calculated but the ranking barely responds when the material input changes, test analysis. If identical runs produce different selections because the agent stops at different points, test orchestration and repeatability.

Choose one intervention that addresses the observed failure and hold the rest of the configuration fixed.

A benchmark score describes how a configuration performs on a defined task set. An instrumented evaluation identifies what to change.

FrontierFinance makes one difficult part of that process unusually concrete: it turns long, open-ended financial reports into thousands of inspectable claim-level criteria. The broader evaluation begins with task interpretation and continues through evidence retrieval, computation, judgment, synthesis, repeated runs, and validation of the evaluator.

My free 30-minute Lightning Lesson on Wednesday, September 2, Managing Your Strategy Research Process, covers the adjacent problem in empirical ML and strategy research. We will identify the decision a result informs, diagnose the constraint exposed by the evidence, and select a controlled experiment that could change a proceed, revise, monitor, or stop decision.

Register for the September 2 Lightning Lesson

We turned that scaffolding into libraries while writing the third edition of Machine Learning for Trading because nine end-to-end case studies need a single implementation of these decisions rather than nine. This week the last of them reached a stable release. Six packages now cover the workflow from raw market data through live trading.

The six ML4T libraries across the research workflow

That is the release. The rest of this piece is about why the timing turned out to matter more than we expected.

Coding agents changed the argument for domain libraries

For most of the time we were building these, the case was ordinary. Shared, tested infrastructure beats six private copies of the same fragile code, the same way it does in any other field.

Coding agents made the case sharper, and in a direction that surprised us. The obvious prediction was that agents reduce the value of libraries: if a model can write a backtest in thirty seconds, why maintain one? What actually happens is the reverse.

An agent will write you that backtest. It will be clean, it will run, and it will produce a Sharpe ratio. What it will not tell you is that it filled orders at a price nobody could have traded at, that it scored the model on data containing the answer, or that the number it reported is the best of forty variants it tried and discarded. The code being correct Python says nothing about whether the experiment was sound.

The deeper problem is continuity. When a person writes that code, the assumptions live in their head, and they carry them from one experiment to the next. When an agent generates it, nothing carries them. The next run regenerates the file from scratch and may resolve a dozen small questions differently: how missing assets are handled, how ties are broken, which lag count goes into a standard error. Two results you intended to compare are then answering slightly different questions, and nothing in either output says so.

So the division of labor we settled on is this. The agent decides which experiment is worth running. The library decides what a fill is. Speed of generation is exactly why the second half has to be fixed: an agent can produce forty variants in an afternoon, and forty variants are only informative if they differ in the way you intended and in no other way.

What each library decides for you

The point of each package is not that it has functions for a topic. It is that it settles a specific question the same way every time, so that two experiments remain comparable.

• ml4t-data settles what the data looked like at the moment of the decision. It covers more than 20 market, macro, factor, and prediction-market sources, stores them locally as Parquet with metadata, and handles incremental updates, gap detection, and validation. Getting this wrong is the single most common way a backtest ends up trading on information that did not exist yet.

• ml4t-engineer settles how a feature or a label is constructed, and where the boundary sits between what a model may see and what it is predicting. Its registry holds 120 features, 60 of them validated against TA-Lib to a tolerance of 1e-6.

• ml4t-models settles what the model is actually fitted to. Financial data arrives as a panel of many assets over time, and whether you are predicting the cross-section on a given day or a single asset’s path changes the model contract. The library makes that explicit rather than leaving it to a reshape.

• ml4t-diagnostic settles how a result is scored: the lag count in a standard error when returns overlap, the purge and embargo that stop a validation split from leaking, the trial count that goes into a multiple-testing correction. These are the adjustments that separate a real edge from a well-dressed one.

• ml4t-backtest settles when an order fills, at what price, and what it costs, along with account rules, risk limits, and an exported record of the run.

• ml4t-live settles what happens between a validated strategy and a funded account: shadow and paper modes, risk limits, reconciliation against the broker, and execution journals. Its adapters for Alpaca, Interactive Brokers, DataBento, and CCXT sit outside the stable contract and require an explicit experimental opt-in.

All six are MIT-licensed and on PyPI, with documentation at ml4trading.io/docs.

Putting it to work: an agent that runs experiments

Chapter 24 of the third edition builds a research operator on top of these libraries. It is worth describing plainly, because the architecture is the point rather than the code.

The operator is a single Python module. It gives a language model ten general-purpose tools, none of them financial: run a shell command, read and write files, read Parquet, query a database, search for a methodology document, and declare the task finished. It contains no sequence of financial calculations at all.

Three things sit outside it.

The first is a corpus of about sixty short methodology notes, each describing one method such as walk-forward validation or a deflated Sharpe ratio, and naming the library function that implements it. The model retrieves these at run time. The methodology can be improved without touching the operator.

The second is the libraries, which the model calls through the shell like any other user.

The third is a research registry, which is simply a database of previous experiments: what configuration was trained, what predictions it produced, what backtest was run on them, and what the metrics were. This is what makes a new experiment comparable to an old one, because the agent can look up exactly how the earlier run was configured instead of guessing.

The agent decides which experiment to run; the libraries decide how it is computed

That last piece turns out to matter more than it sounds. When I asked the operator whether an ensemble could improve one of the book’s ETF case studies, the specification it recovered from the registry was four fields: score_weighted_top_k, top_k=20, monthly rebalancing, long-only. Those four fields are the entire reason its result could be compared with the baseline at all.

Two runs

The ensemble run combined the strongest predictions from three model families and reran the cost-aware backtest under the baseline’s recovered configuration. Mean information coefficient, the rank correlation between predictions and subsequent returns, rose from 0.052 to 0.065. Portfolio Sharpe fell from 0.92 to 0.56, with three of eight validation folds showing negative IC.

The agent recommended against adopting the ensemble, and identified why. Its fold-level instability, amplified by an allocator that concentrates on the highest-scoring names, is what costs portfolio performance even as rank correlation improves. That is one of the case study’s own teaching points, reached in 39 turns without being pointed at it. The comparison runs on the validation window, since the registry holds holdout predictions for the baseline alone, and the chapter notebook records that scope against the original run artifact.

A second run tested how much of a US firm-characteristics strategy depended on its smallest names. Filtering the existing validation predictions to the top three market-cap quartiles cut Sharpe from 4.27 to 2.24, lowered IC from 0.074 to 0.048, and deepened maximum drawdown from 15% to 52%. Turnover was nearly unchanged, 1.77 against 1.80, because the filter screens signals without retraining. That makes it a sensitivity screen rather than a capacity estimate: it shows the result leans heavily on small caps, and it does not separate that from reduced breadth, nor say what a capacity-constrained implementation would earn.

Neither run produced a stronger strategy. Both produced a specified experiment, a recorded configuration, and a result that the next iteration can build against, which is most of what research actually consists of.

Where the tools stop

The libraries constrain what gets computed. They do not constrain what gets concluded from it. Which evaluation window supports which claim, whether a specification answers the question that was actually asked, and whether a result justifies a decision are review judgments, and no package boundary reaches them. That is why the operator records the evaluation window, the fold-level statistics, and every generated script rather than only the final recommendation.

There is also a security boundary worth stating plainly, because it is easy to get wrong. The operator restricts file edits to a designated output directory, but that is a guardrail for a cooperative model rather than a sandbox. Its shell tool runs commands with the host user’s privileges, so a confused or jailbroken model can write outside that directory or reach the network. The published notebook replays a recorded trace by default and executes nothing the model supplies. Running it live against an external model needs real isolation: a container, dropped privileges, read-only mounts, and no network egress unless you mean to allow it.

An unattended system needs more still: deterministic seeds, contamination tracking, explicit approval points, and limits on what may act downstream of a research note.

What this adds up to

A research agent is only as trustworthy as the tools it calls. Give it a shell and a prompt and it will reimplement finance from memory, differently each time. Give it versioned libraries that have already settled how a fill works, where a validation boundary sits, and how a trial count enters a correction, and its output becomes something a reviewer can check and a second run can reproduce.

That is what these six libraries are for, and it is why finishing them mattered more in 2026 than it would have in 2023.

All six are on PyPI now, and the Chapter 24 operator is public.

McManus sets the scope before the demo runs:

PAT is not about how we trade. It’s really about performing deep exploratory research, enabling our investors to go after questions that they never would have had the bandwidth to pursue before.

Asked in the comments what the return has been, Weight replied that exact numbers are hard, then named time saved on analyses, “expected multiple man-years in first few months”, and investors trying things they would otherwise have dropped, which he marks as value-add rather than efficiency gain.

Their roadmap is a set of discrete sub-agents covering a research cycle: perceiving what is happening, forming questions, investigating, synthesizing, and folding what was learned back into shared memory. PAT is the investigation piece. Its outputs land in the same database its inputs came from, so one analysis becomes an input to the next, and human and agent work accumulate in one place.

PAT’s pipeline: clarify, plan, generate, analyze dependencies, execute, validate, with a Teach loop feeding corrections back into context and harness

Domain inspection took retrieval from 50% to 90%

PAT searches a time series database holding tens of millions of series, some external like the price of oil, some internal like their own 12-month-forward inflation estimate. The search agent starts with what you would expect. Ran:

The search agents use traditional search techniques like RAG, re-ranking, et cetera, but the thing we found to be a huge difference-maker was layering on an element of human-like inspection.

The inspection is what a researcher does without narrating it. You do not accept a series because its name matched. You look at the frequency, you look at the currency, and you check whether the values agree with what you already believe about the world.

So embedding this sort of reasoning into our search agent is something that got us up from roughly 50% accuracy all the way to 90%.

Fifty to ninety, from encoding domain judgment as search logic. Both numbers are Bridgewater’s own; in the evaluation, the talk does not describe how many queries, what counted as correct, or whether the two measurements share a test set. The direction is the transferable part. The reflex when retrieval underperforms is to reach for a better embedding model or a reranker. Their gain came from writing down what an analyst checks before trusting a series.

The dataframe plan is an intermediate representation

PAT resolves ambiguity with the user first, then plans. The planning phase names every dataframe the analysis will produce, the schema of each one, and how they connect. Ran calls the cost deliberate, and it buys something specific.

Because every task declares its output schema up front, code generation for all of them runs in parallel. A visualization task at the end of the plan already knows the shape of data produced by a loading task whose code does not exist yet.

So this lets an analysis with three data frames or a medium-sized one with something like 30 data frames take roughly the same amount of time to generate code in both scenarios.

Their benchmark names its comparison. For the same context and the same plan, PAT generates code about four times faster than Claude Code, and a 20-task plan costs what a three-task plan costs.

The structure has a published analog. LLMCompiler (Kim et al., ICML 2024) plans a directed acyclic graph of tasks with their dependencies and dispatches them to a parallel executor, reporting up to 3.7x lower latency and roughly 9% better accuracy than ReAct on its benchmarks. LLMCompiler’s graph is over function calls. Bridgewater’s is over dataframe schemas, which makes the plan a domain-specific intermediate representation: code generation lowers it into Python, and the unit being compiled is an entire analysis. What the schema buys is that downstream work can start before upstream implementations exist.

Weight pushes the framing further. The plan is a natural language Python project, not the to-do list a coding agent keeps, and two agents compiling the same task should produce code that computes the same values.

Validation is mandatory, not discretionary

Weight’s background is compiler theory. A compiler lowers user code to something executable, and it treats determinism and correctness as requirements rather than aspirations. A coding agent lowers a plan to Python. He argues for building the second the way the industry already knows how to build the first.

Concretely: after generation, static analysis builds the dependency graph, and validation agents run in parallel across its layers. A five-task plan collapses to three layers, a twenty-task plan to four or five. The orchestration is ordinary Python.

The main point I want you to take away from this part of the slides is that we enforce correctness in the architecture. Again, no agentic orchestration. This is regular Python code, so the guardrails are really hard, and the agents cannot forget to validate. They are forced to validate.

Their phrasing is worth reading narrowly, because the narrow version is the useful one. Ordinary Python control flow guarantees that every generated task passes through the prescribed validation stages, which removes the failure mode of an agent deciding this particular output does not need checking. It guarantees nothing about whether the validators test the right properties, whether the expected values are right, or whether a task and its validator share a common-mode error. The architecture makes validation happen; whether the validation is sufficient is a question for the test suite.

There is evidence behind putting the check outside the agent’s discretion. Huang et al. (ICLR 2024) tested intrinsic self-correction, where a model revises its own answer using nothing but its own judgment, and found that reasoning performance fails to improve and, in their experiments, frequently degrades. They frame that setting as the one that matters in practice, since high-quality external feedback is often unavailable. PAT does inspect its own output, which Ran compares to a junior analyst double-checking work before handing it over, and that inspection at least has computed values and rendered charts to work against. The paper’s finding applies to the case the architecture removes: judgment as the only check.

The result:

The result is that when we run our test suite on any plan, 95% of the time, the code that comes out is exactly the same for two different agents. So it’s essentially a deterministic coding agent.

“Exactly the same” is doing unexamined work here. Identical source, identical computed values, and semantically equivalent code are three different claims, and the talk names neither which one it measured nor how large the suite is. Read as high reproducibility on their internal suite, it still buys something. Evaluation survives nondeterminism; it just costs more runs: when two runs of the same plan can disagree, attributing a change to your prompt means averaging over enough runs to separate the effect from generation variance. Their phrase for the alternative is “vibes-based or LLM-as-judge evals.”

That aside is fairer to judges than it reads. Zheng et al. (2023), the work that popularized the method, report agreement with human preferences above 80% and document in the same paper that judges exhibit position bias, verbosity bias, and self-enhancement bias, the last being a model’s preference for its own output. Where a property can be checked by an executable test, the test is the more direct instrument. Open-ended qualities still need a judge, with that drift priced in.

Incremental execution on a runtime they own

PAT does not invoke its own generated code through a terminal. Bridgewater runs it for the model, injecting caching annotations by static analysis so intermediates are never recomputed. Their second benchmark takes the last chart in a plan and changes its name. Claude Code reruns the lot; PAT re-executes essentially instantly. Small edits to a finished analysis stop carrying the cost of a full iteration, so an investor can keep adjusting one while still thinking about it.

This is available to Bridgewater because they own the execution environment. An agent shelling out to a terminal cannot inject caching into a runtime it does not control, which is the same point as treating the harness as context plus tools.

The transferable idea is incremental execution over an explicit dependency graph, and it arrives with the problem every build system has. A renamed chart is the easy case. The graph tells you what to recompute only if the cache key captures code changes, data versions, parameters, and the relevant environment; get invalidation wrong and stale intermediates survive an edit that should have killed them.

Observed failures become regression benchmarks

A user who thinks PAT should have done better presses a Teach button, and an agent reads the conversation for behavioral mistakes, context gaps, or steering that could have been anticipated. Then, in order: it writes a benchmark expected to fail, confirming the bad behavior reproduces. It edits the context repositories or the harness until that benchmark passes. It verifies the rest of the suite still passes, so the fix cannot silently break an earlier one. It opens a pull request, which arrives in Slack for a human.

This is test-driven development pointed at agent behavior. Writing the failing test first is what ties the change to the behavior a user actually reported. Who defines the expected behavior decides whether the loop is worth anything, because a conversation can reveal a user’s preference as readily as an objective error. The human holds that step twice: the user can revise the proposed lesson before it becomes a benchmark, and a reviewer decides whether the pull request changes PAT for everyone. Agents also mine completed conversations for the same material without being asked, and a correction reaches every investor at the firm rather than only the one who hit the problem.

One PAT per investor, differing by context and tools

Different investors at Bridgewater see different things. Someone with access to firm positions needs a PAT that knows them; an analyst without that access needs a PAT that cannot leak them. Ran describes the resolution in one line:

Practically speaking, this uniqueness is just a function of what context and what tools each person’s PAT has.

Every investor runs their own PAT, and the difference between any two is which context is loaded and which tools are exposed. That is a clean account of personalization, and it is not a security boundary. The data and tool layers still have to enforce each investor’s permissions, because a context-assembly rule that is too broad, or a tool that reaches further than intended, exposes data whether or not the model was told to respect the limit.

What the team says it learned

Weight leaves the audience with three things: specialize agents instead of building generic, powerful ones; benchmark narrow workflows heavily and hill climb them; and treat agentic coding as a compiler problem. The first comes with a warning.

We don’t really believe in generic, powerful agents. They make really cool demos. I’m sure many of us have given them. But it’s really hard to make that a daily workflow that you can depend on.

What does not carry over

Bridgewater has been writing down the causal logic behind every trade since a 1980 bond system on a yellow legal pad, and codifying it into machine-readable form ever since. McManus is direct about the advantage:

We didn’t have to go back and write down everything for agents. It was already there for us to draw upon.

Fifty years of codified investment logic, tens of millions of modeled series, and millions of documents arriving at thousands a day. Few teams can reproduce that, and the talk says plainly that it is the foundation the rest sits on. The second advantage is how the work is staffed. McManus describes multi-archetype teams with investors, technologists and scientists sitting side by side: “Investors bring the context and the domain-specific expertise. Technologists bring the architectural capability. And scientists bring the rigor.”

Five things carry over without a hedge fund’s archive: encoding domain inspection into retrieval, planning in enough detail that generation parallelizes, enforcing validation structurally so it cannot be skipped, converting observed mistakes into failing benchmarks, and treating the harness as context plus tools. Each still needs schemas, metadata, and access rules of your own, which is real work. None of them needs fifty years of it.

The unanswered point-in-time test

The highest-rated comment on the talk proposes a test: reconstruct a past thesis “using only information available on that date,” which would “separate fluent retrieval from genuine research discipline.” That is point-in-time correctness, and it decides whether a research system is measuring anything at all. Nothing in the talk says how PAT handles it.

The same firm’s AIA Labs has published on exactly that problem for a different system. The AIA Forecaster technical report treats foreknowledge bias, information reaching the model from past its intended cutoff, as a first-class evaluation target. They built an LLM judge that flags search results implying knowledge beyond the cutoff, hand-audited 502 sampled traces to calibrate it, and estimate that about 1.65% of search results carry some leakage. Bounding the worst case moves their headline Brier score from 0.1159 to 0.1166. They also built MarketNightly, a live benchmark drawing only on active prediction markets, where foreknowledge is impossible by construction rather than merely detected.

The forecasting results are worth stating precisely, because the direction changes with the benchmark. On ForecastBench, the AIA Forecaster is statistically indistinguishable from expert superforecasters, scoring 0.1076 against their 0.1110 on one variant and 0.1099 against 0.1152 on the other, where lower is better. On MarketLiquid, a harder benchmark sourced from liquid prediction markets, it trails the market consensus, 0.1258 against 0.1106; a regression ensemble of the two reaches 0.106 and beats both. Matching superforecasters on one benchmark and losing to a liquid market on a harder one both appear in the same paper. That is the standard of reporting to want before believing an agent result.

On Saturday, August 1, I teach this architecture by building it: a forecasting agent end-to-end, working from my own implementation of the AIA Forecaster above. Prompt, context, harness, loop. Typed tool contracts, agent state and replay, point-in-time research agents running in parallel, a supervisor that reconciles their disagreement, and evaluation with proper scoring. 10:00 am to 3:30 pm ET, live. Details and registration.

Antonio Gulli, Senior Engineering Director in Google’s Office of the CTO, and author of the excellent book Agentic Design Patterns (among many others) wrote the foreword: “I definitely recommend this book if you want to understand finance in the modern AI and agentic world. It’s a definitive guide.”

I am walking through the new edition in a 30-minute session next Thursday, July 30 at 11:00 a.m. ET: how the 27 chapters map to the research workflow, how the nine case studies and the companion code fit together, and which route through the material fits which goal. Register here.

Here is what is new.

The research process, front and center

The second edition had a workflow. The third puts it front and center.

The ML4T research workflow: data acquisition, feature engineering, model training and selection, strategy simulation, portfolio construction, cost and risk analysis, and live deployment, with an iterate-and-redeploy loop back to data.

The stages before modeling now get real treatment: how to turn a trading idea into a specification with an economic hypothesis, a universe, a horizon, and constraints; how to define the learning task, because the label choice determines what a model can learn; how to build financial features, model-based features, and text features without leaking the answer.

The stages after modeling are separated into four dedicated chapters. The second edition covered backtesting as historical simulation. The third splits the decisions that produce a simulation worth running into strategy simulation, portfolio construction, transaction costs, and risk management, then adds a fifth chapter that analyzes the nine case studies side by side to show how different design choices behave.

Nine cross-asset case studies

Each case study carries a research question from raw data through labels, features, models, backtests, costs, and risk to a deployment assessment. They are complete research projects, organized as pipelines instead of isolated demonstrations of one method.

US firm characteristics, a US equities panel, S&P 500 equities plus options analytics, CME futures, ETFs, crypto perpetuals, S&P 500 options, FX pairs, and NASDAQ-100 intraday microstructure. The studies range from 15-minute intraday data to monthly equity panels.

All nine follow the same sequence of research phases, while the model families, features, and horizons vary where the market requires it. That shared phase order is what makes it possible to see where market structure, data availability, and trading costs actually differ across asset classes.

The new chapters

Everything below is either new to this edition or rebuilt from the ground up. Each links to its chapter guide.

• 5. Synthetic Financial Data for the situation where you do not have enough history to test what you want to test.

• 6. Strategy Research Framework, turning a trading idea into a testable specification.

• 9. Model-Based Feature Extraction, features that are themselves model output, with the leakage discipline that requires.

• 13. Deep Learning for Time Series, greatly expanded to the architectures that now matter.

• 15. Causal Machine Learning, in this book for the first time. Most of what a researcher wants to know about a strategy is a causal question, and this chapter treats it as one.

• 16-19. Strategy Simulation, Portfolio Construction, Transaction Costs, Risk Management, four chapters where the second edition had one on backtesting.

• 21. Reinforcement Learning for Execution and Hedging. The second edition trained a deep Q-learning agent to trade the stock market. The third shifts the emphasis to optimal execution, market making, and deep hedging.

• 22. RAG for Financial Research, retrieval over filings, research, and news.

• 23. Knowledge Graphs, for the questions where relational structure earns its cost.

• 24. Autonomous Agents, for both forecasting and research.

• 25. Live Trading Systems and 26. MLOps and Governance, carrying the workflow into production, monitoring, and the two sources of live failure.

20,000 stars

The companion repository passed 20,000 stars this month, with more than 5,400 forks. It went through GitHub Trending in June alongside karpathy/nn-zero-to-hero and microsoft/ai-agents-for-beginners.

That repository is where the book actually lives. Everything below is in it.

Six open Python libraries

The workflow as installable code, one package per stage, each documented and usable on its own:

• ml4t-data (repo), unified market-data acquisition from 19+ providers behind one interface

• ml4t-engineer (repo), features, labels, alternative bars, and leakage-safe dataset preparation

• ml4t-models (repo), finance-native latent factors, stochastic discount factors, direct prediction, and portfolio learning

• ml4t-diagnostic (repo), feature validation, strategy diagnostics, and the Deflated Sharpe Ratio

• ml4t-backtest (repo), event-driven backtesting with realistic execution

• ml4t-live (repo), production trading with broker integrations

What makes them finance-native is specific: point-in-time data contracts, financial labels, purged and embargoed validation, transaction-cost models, and broker parity.

459 notebooks

Every chapter and every case study is executable, and all of it goes live today in the companion repository.

The data stack has moved from pandas to Polars, which cuts memory use and execution time in the larger research pipelines.

112 primers and 61 agent skills

The primers hold the prerequisites and specialized extensions that would otherwise interrupt the chapters, so the book can assume a concept and link to its explanation.

The agent skills encode repeatable research tasks together with their checks, covering lookahead bias, data leakage, walk-forward validation with purging and embargo, cost models, position sizing, and production workflows. They are for readers running a coding agent inside their research loop.

Courses

Live cohorts and workshops that apply the process to your own research with feedback, for readers who want more than a description.

Where to start

Choosing a market: open the case study closest to what you trade and follow it through the workflow chapters.

Building infrastructure: start with ml4t-data and ml4t-backtest. Both are usable without adopting the rest of the stack.

Here for what changed since 2020: causal machine learning, the four strategy-design chapters, then reinforcement learning through live trading.

Or come to the free reader’s guide on July 30, and I will walk you through it.

The book is available from Amazon and Packt. The code is on GitHub, and everything else is at ml4trading.io.

The full chapter map

Every chapter has a guide on the site.

1. The Process Is Your Edge

2. The Financial Data Universe

3. Market Microstructure

4. Fundamental and Alternative Data

5. Synthetic Financial Data

6. Strategy Research Framework

7. Defining the Learning Task

8. Financial Feature Engineering

9. Model-Based Feature Extraction

10. Text Feature Engineering

11. The ML Pipeline

12. Gradient Boosting and Advanced Tabular Models

13. Deep Learning for Time Series

14. Latent Factor Models

15. Causal Machine Learning

16. Strategy Simulation

17. Portfolio Construction

18. Transaction Costs

19. Risk Management

20. Strategy Synthesis

21. Reinforcement Learning for Execution and Hedging

22. RAG for Financial Research

23. Knowledge Graphs

24. Autonomous Agents

25. Live Trading Systems

26. MLOps and Governance

27. The Systematic Edge

My thanks to Antonio Gulli for the generous foreword, to the editors, reviewers, and technical contributors, and to the readers of the second edition, whose questions, issues, and pull requests shaped what this one became.

• the agent loop that writes or edits code,

• the harness loop that decides whether the work is acceptable, and

• the workflow loop that decides what work should exist, who reviews it, and what state survives into the next run.

The argument was that the workflow loop is where the engineer stays. The issue closed on a design rule — decide what the agent may change, define how the harness checks it, preserve the evidence needed for review, and keep the research claim outside the automation boundary — and then stopped, because a reading list describes a loop without handing you one.

The three loops from the last issue. The agent loop writes code; the harness loop checks whether the work is acceptable; the workflow loop decides whether the work should exist, who reviews it, and how state persists. This release is one implementation of that outer loop.

coding-agent-toolkit is now public: seven host-neutral steps that take a piece of work from a rough idea to a merged pull request, on both Claude Code and OpenAI Codex. It is the specific workflow loop I run daily, released now because the shape has stopped moving. The agent does the work; the toolkit keeps the state.

A concrete case for what “keeps the state” buys you. An agent starts a data-loader fix, patches the imports, and gets three tests green before it runs out of context. The next morning, a fresh session — maybe on the other host — picks up the branch. With nothing durable on disk, it re-reads the repo, guesses at what was in flight, and quietly widens a loader fix into a refactor of the ingestion layer. With a spec, a plan, an open issue, and a handoff snapshot to read first, it resumes the same scoped task: finish the loader fix, keep the diff reviewable, and stop. The workflow loop is what makes the second outcome the default.

What The Seven Steps Do

Five steps carry a piece of work forward — align, plan, plan-issues, next-issue, ship — and two, handoff and continue, carry state across the gaps between sessions. Each writes one file the next step reads, so state lives on disk rather than in a conversation, and each exists to head off a specific way delegated work goes wrong.

The five delivery steps, the file or GitHub object each writes, and the two transition steps that carry state across boundaries. The file on disk is the contract that the next step reads.

• align → spec.md. Interrogates a rough request into a verifiable end-state, one question at a time, before any code is written — so the agent is not optimizing toward an implicit, private definition of done.

• plan → plan.md. Breaks the spec into issue-sized chunks with dependencies, so one request does not arrive as a single sprawling change that no one can review.

• plan-issues → a GitHub milestone and one issue per chunk. Dry-run by default; --apply writes. The work becomes reviewable units under version control, not lines in a chat transcript.

• next-issue → a branch, an implementation, tests, and a PR. Takes the lowest-numbered open issue in the active milestone, so the agent works the next scoped unit instead of whatever is convenient.

• ship → a squash-merge and a closed milestone. Runs only after verifying that every issue has a commit with a closing footer, so nothing merges while leaving the work graph half-open.

• handoff / continue → a durable session transition. handoff writes a snapshot; continue replays it and flags drift — so the next session does not trust stale prose, and drift does not stay invisible until it corrupts the next task.

Six of the seven ship as a Claude skill and a mirrored Codex prompt. plan is the exception: it delegates to each host’s native plan mode and a capture hook, because re-implementing plan mode as a shell command would discard the structured output the host already produces.

Closes #N in a PR body is the signal that bubbles state up: a merged PR closes its issue, and ship closes the milestone once every issue is closed by a commit footer. GitHub is the projection; the file each step writes to disk is the contract the next step — or the next agent — reads, rather than re-deriving where things stand.

The same steps apply to non-code work when the deliverable consists of reviewable files. Closes #N closes an issue about a chapter draft or a research report as cleanly as one about a bug fix.

How It Answers The Design Rule

The last issue’s rule had four clauses. The toolkit maps onto three directly; the harness clause it leaves to your existing tests and review.

• Decide what the agent may change. align produces a spec with acceptance criteria before the agent touches a file. The scope is written down and reviewable in advance, so “the agent wrote something” and “the agent did the agreed work” become separate, checkable claims.

• Preserve the evidence needed for review. The chain produces a milestone, issues, branches, and a merged PR — a trail a human reads, rather than an agent transcript to reconstruct afterward. The diff arrives scoped to one issue, tied to a stated spec, with a closing footer.

• Keep the research claim outside the automation boundary. The last issue’s sharpest example was a data-loader test made to pass by dropping the tickers that had stopped returning history: the suite goes green, and the backtest quietly runs on survivors only. The code is correct; the research claim has changed. A host-neutral workflow layer does not prevent that edit by itself — domain checks do that: a frozen universe, a survivorship guard, an invariant test.

What the workflow changes is visibility. The edit arrives as a scoped PR against a spec that stated what the loader was for, so a reviewer sees a change to the ticker universe instead of a green checkmark, and the domain checks have somewhere to run. The same reading list noted that many agent failures are workflow failures — abandoned reviews, duplicate work, task mismatches — rather than syntax errors, and that this is the layer this works on.

The toolkit does not make the model write better code, nor does it replace tests, domain invariants, data checks, cost models, or review. It makes the agent work legible enough that those checks — and a human — can still own the result.

Cross-Host State

The same workflow runs on Claude Code and OpenAI Codex because there is no orchestrator between them. The cross-host primitive is a shared durable state under .workspace/, read natively by both.

Swapping hosts mid-feature works because the next session — whichever host it runs on — picks up where the last one left off by reading what the last one wrote. There is deliberately no “execute as the other host” command; the file is the swap.

Session continuity has one refinement worth calling out. A prose handoff goes stale silently: the next session cannot tell whether what it describes is still true. handoff pairs the summary with a fenced block of read-only commands, each carrying an inline # expect: <value>. continue runs each command, compares the output to the expected value, and flags every divergence. Drift is information: a repo one commit ahead is fine; a milestone that has since closed may matter. continue surfaces suggested next steps and does not act on them.

This release supersedes claude-code-toolkit, the broader, Claude-only predecessor — a wide collection of patterns and plugins. This one is narrower: seven steps, two hosts, one job. It is a sibling, not a competitor, to code review: if you run roborev, open reviews on the current branch surface when a session starts, and stay silent when it is not installed. One reviews code, one drives work, and they compose.

Installing It

Claude Code. Copy or symlink skills/ into your project’s .claude/skills/, or install the workflow plugin from the coding-agent-plugins marketplace, which mirrors the same steps. The marketplace is Claude-side; the toolkit itself is cross-host.

OpenAI Codex. Point your prompts directory at codex/prompts/, or copy the files in.

Then invoke the steps in order on a new piece of work, and handoff / continue at the boundaries of anything long-running. The SKILL.md in each step directory is that step’s authoritative documentation; there is no separate runtime to install.

The toolkit is pre-1.0 and MIT-licensed. The step contracts are stable enough for daily use, but their shape may still shift in response to friction encountered in real work. Issues and PRs are welcome. Repository: github.com/stefan-jansen/coding-agent-toolkit.

Where This Sits In The ML4T Work

The near-term value of coding agents for ML4T readers is the bounded, checkable work: environment repair, notebook maintenance, data-loader fixes, test scaffolding, CI triage, reproducible experiment plumbing. The workflow loop is what keeps that work reviewable as you delegate more of it, and what draws the line the last issue insisted on — between the changes you hand to the loop and the ones that decide whether a result is real.

Today I am teaching a free Maven Lightning Lesson, Getting stuff done with coding agents (4:00 PM UTC), which walks through this chain end-to-end: turning an ambiguous goal into a scoped work unit, keeping state across sessions, and using review gates so a human still owns the result.

For the hands-on version, I am running a workshop, Loop Engineering: Reliable Work From Coding Agents, on July 18 (10 am–2 pm EDT). It builds what this issue describes, on your own keyboard: turn a vague goal into a verifiable spec, project the plan onto GitHub issues, and write a handoff that survives the session boundary. It also puts you in front of the green-but-wrong failure — the suite passes while the result is corrupted — and has you write a verification skill the agent cannot satisfy by quietly dropping the tickers that stopped reporting.

The same division of labor — what you hand to the loop, what stays with you because it decides whether the result is real — runs through the Machine Learning for Trading: From Research to Production cohort across nine case studies; the current summer cohort is full, with a waitlist for the next.

“Loop engineering” names a real shift in practice: developers spend less time on single prompts and more time designing the system around the agent — context, tools, state, branches, tests, permissions, review, and stop rules. That is worth naming.

But a serious reading list cannot stop with the current discourse. The current labels give new names to problems that control theory, planning, and human-automation research have studied for decades: bounded search, partial observability, shared state, reusable actions, delegation, verification, and supervisory control. The newer work matters because LLMs made these problems operational in software repositories, where the agent can change files and the environment can push back through tests, logs, compilers, benchmarks, diffs, and code review.

This list separates six kinds of sources:

1. practitioner pieces that explain why “loops” became the current label;

2. older control and agent foundations that give the label depth;

3. LLM-agent primitives that explain the mechanism;

4. coding-agent systems and harness papers;

5. empirical studies of adoption, review, refactoring, context, and test quality;

6. benchmarks and safety work that measure task completion or risk.

That separation matters. A benchmark is not the same as a field study. A product essay is not the same as evidence. A context-file study is not the same as a repository-level benchmark. Mixing them into one bucket makes the field look thinner than it is.

Match your question to the bucket, and note what that kind of source can and cannot settle:

Prompt quality still matters, but the engineering surface expands as agents gain context, tools, state, permissions, verification, and persistence.

From Prompting To Loop Engineering: The Current Discourse

Read these first to understand the vocabulary. Then move on.

• Addy Osmani, “Loop Engineering”. The term’s current anchor. Osmani frames loop engineering as designing the system that prompts, monitors, and resumes the agent. The useful primitives are automations, worktrees, skills, plugins or connectors, subagents, and external memory. Read it for language, not for empirical proof.

• Andrew Ng on three product-building loops (The Batch #359, June 2026). The highest-profile statement of the frame, published as the naming wave crested. Ng separates three loops by timescale and who acts: an agentic coding loop (minutes), a developer-feedback loop (tens of minutes to hours), and an external-feedback loop (days to weeks). That is a different cut than Figure 2 below, which separates loops by what each one validates — evidence, not contradiction, that “loops” already names several things. Ng also traces the phrase’s spread to Boris Cherny (Claude Code) and Peter Steinberger (OpenClaw).

• Armin Ronacher, “The Coming Loop”. The skeptical counterpart. Ronacher separates the model’s tool-result cycle from the external harness that restarts, redirects, or escalates work. Read it for comprehension debt: the risk that code becomes easier for machines to change than for humans to understand.

• Simon Willison, “Designing agentic loops”. A cleaner earlier statement of the loop idea: agents run tools in a loop to achieve a goal. The value is that it predates the June 2026 naming wave.

• Birgitta Böckeler, “Harness engineering for coding agent users”. Published on martinfowler.com. Note the pieces around the model: guides, sensors, tests, feedback, and the boundaries that make delegated work reviewable.

• Geoffrey Huntley, “everything is a ralph loop”. A reminder that the external while-loop around a coding agent existed before the current name. The practice predates the label.

The agent loop writes code; the harness loop checks whether the work is acceptable; the workflow loop decides whether the work should exist, who reviews it, and how state persists.

Older Agent And Control Foundations

This is the layer the previous draft underdeveloped. These sources are not ornaments. They are the reason the current vocabulary makes sense.

• Newell and Simon, Human Problem Solving. The starting point for problem solving as search through a structured problem space under bounded computation. Coding agents explore repo states, command outputs, and patch candidates under similar limits.

• Hart, Nilsson, and Raphael, “A Formal Basis for the Heuristic Determination of Minimum Cost Paths”. The A* paper. It belongs here because loop design is partly about spending search budget intelligently: which branch, which tool call, which retry, which stop condition.

• Fikes and Nilsson, “STRIPS”. Preconditions, operators, and effects. Modern tool calls look different, but safe agent design still asks what must be true before an action runs and what the action changes.

• Ghallab, Nau, and Traverso, Automated Planning: Theory and Practice. A broad synthesis of planning, decomposition, actions, goals, temporal structure, and uncertainty. It gives a broader frame than “prompt a model to make a plan.”

• Kaelbling, Littman, and Cassandra, “Planning and Acting in Partially Observable Stochastic Domains”. A coding agent never sees the full system. Files, logs, tests, prior runs, and issue text are observations. Context engineering is better belief-state maintenance, not larger prompts.

• Erman, Hayes-Roth, Lesser, and Reddy, “The Hearsay-II Speech-Understanding System”. The canonical blackboard architecture. Multiple specialized knowledge sources coordinate through shared state. Modern versions include issue trackers, state files, memory logs, CI summaries, and review artifacts.

• Rao and Georgeff, “BDI Agents: From Theory to Practice”. The belief-desire-intention model is still useful for separating what an agent knows, what it is trying to achieve, and what course of action it has committed to.

• Sutton, Precup, and Singh, “Between MDPs and Semi-MDPs”. The options framework gives the clean lineage for skills: reusable, temporally extended actions that can be selected like primitives.

• Horvitz, “Principles of Mixed-Initiative User Interfaces”. Good agent systems need rules for initiative: when the system proceeds, asks, pauses, escalates, or hands control back.

• Parasuraman and Riley, “Humans and Automation: Use, Misuse, Disuse, Abuse”. The automation-trust paper to read before giving agents more authority. Coding-agent failures often come from overtrust, under-specification, or weak supervision rather than local syntax errors.

• Sheridan, Telerobotics, Automation, and Supervisory Control. A durable treatment of human supervision over semi-autonomous systems. It is directly relevant to agent loops that operate while a human reviews only the artifacts.

LLM-Agent Primitives

This layer is closer to the earlier agent canon, but here the emphasis is on what each source adds to coding-agent loops.

• WebGPT. Browse, gather evidence, cite. It is an early template for language models acting through tools rather than answering from memory.

• MRKL. Modular routing across tools and symbolic systems. It foreshadows the connector/plugin layer.

• ReAct. The atom of the modern agent loop: reason, act, observe.

• Toolformer. Tool use as a learnable behavior, not only an inference-time wrapper.

• Reflexion. Actor, evaluator, self-reflection, and memory. Coding agents need the same separation between making a change and judging it.

• Tree of Thoughts. Branch, score, and backtrack. Maps onto parallel worktrees, alternate patches, and explicit search over candidate plans.

• Voyager. Executable skills that accumulate over time. It is the clearest ancestor for practical skill libraries.

• CodeAct. Executable code as the action language. This is central to why coding agents feel different from generic chat agents.

• SWE-agent. The agent-computer interface becomes a first-class variable. The environment and harness shape the observed capability.

• AI Agents That Matter. Cost, holdout validity, benchmark shortcuts, and complexity discipline. Read this before believing any agent leaderboard.

Coding-Agent Systems, Harnesses, And Configuration

This group is about the engineering layer between the model and the repo.

• SWE-bench. The most widely cited benchmark for repository-level issue resolution. It belongs in this section as the benchmark that shaped coding-agent systems, not as a field study about adoption.

• SWE-agent. Also belongs here because it showed that the agent-computer interface changes performance. The same model can behave differently under a different scaffold.

• CodeAgent. Early repo-level, tool-integrated code generation. Useful for seeing how tool use entered repository work before the current loop vocabulary.

• SICA: A Self-Improving Coding Agent. A coding agent modifies its own codebase and improves from 17 to 53 percent on a random SWE-bench Verified subset. Read it as a harness/self-improvement result, not as proof of open-ended autonomy.

• Code as Agent Harness. A synthesis of the idea that code is not only the agent output; it is also part of the harness that structures reasoning, action, and verification.

• Configuring Agentic AI Coding Tools. A practical taxonomy of configuration mechanisms across coding tools.

• A Dataset of Agentic AI Coding Tool Configurations. Shows what configuration patterns appear in real repositories, rather than only in vendor docs.

• Agent READMEs. Descriptive evidence on context files in the wild: what teams put in them, how they vary, and why they behave more like configuration than prose.

• Evaluating AGENTS.md. The corrective to context-file enthusiasm. Repository instructions can reduce success and increase cost if they add irrelevant or distracting guidance.

Empirical Studies Of Use, Review, And Maintenance

This section is distinct from benchmarks. These studies look at adoption, pull requests, review, refactoring, context use, and test quality in practice.

• Agentic Much?. A GitHub-scale adoption study. The current arXiv version reports 22.20 to 28.66 percent adoption across 128,018 GitHub projects. It measures traces of use, not net value.

• AIDev. A dataset of agent-authored pull requests — 932,791 PRs across 116,211 repositories, produced by five agents. Keep it separate from companion studies that analyze acceptance or failure.

• The Rise of AI Teammates in SE 3.0. Uses an earlier AIDev slice to study agentic PR behavior. Read for the acceptance gap and the difference between speed and accepted contribution.

• Where Do AI Coding Agents Fail?. A failure study of agent-authored PRs. It is valuable because many failures are workflow failures: abandoned review, duplicate work, CI/test failures, or task mismatch.

• Agentic Refactoring. Refactoring is not one thing. Agents can make local structural changes while leaving deeper design issues intact.

• Are Coding Agents Generating Over-Mocked Tests?. Test quality matters because shallow verification is easy to satisfy. Agent commits change tests more often than non-agent commits (23 versus 13 percent) and add mocks more often (36 versus 26 percent).

• ContextBench. Process-oriented evidence that finding relevant code and using it correctly are different skills.

• SWE-ContextBench. Prior experience can help, but only when retrieved and summarized correctly. This is the right paper to read before building memory-heavy loops.

• Coding Agents Are Effective Long-Context Processors. Supports the idea that agents can externalize long-context processing through filesystems and tools. Do not generalize it to all long-horizon maintenance.

Benchmarks, Stress Tests, And Safety

Benchmarks are not field studies. They measure task performance under a protocol. They are useful when the protocol is visible.

Read benchmark scores as protocol results. A benchmark can say what an agent did under a task set, scaffold, tool policy, and evaluation rule. It does not by itself establish production value, review cost, or research validity.

• SWE-bench. Real GitHub issues as repository-level tasks. Essential, but easy to overread once leaderboards saturate or scaffolds diverge.

• SWE-PolyBench. Multi-language repository-level evaluation. It helps counter Python-only benchmark bias.

• RExBench. Research-extension tasks in real repositories. A useful ceiling check for hard, novel work.

• SaaSBench. Long-horizon enterprise SaaS engineering. Its main lesson is integration: failures often happen before agents reach deep business logic.

• RedCode. Risky code execution and generation. This belongs in the core path because coding agents operate tools, shells, files, and sometimes networks.

• OWASP Top 10 for LLM Applications. The security frame for prompt injection, supply chain risk, sensitive data, and excessive agency.

When reading a benchmark score, ask: which task set, which date, which model, which scaffold, which tools, which permissions, and which evaluation protocol? A model score without the harness is not a product capability claim.

What This Means For ML4T Work

For ML4T readers, the near-term value of coding agents is practical: environment repair, notebook maintenance, data-loader fixes, test scaffolding, documentation updates, CI triage, and reproducible experiment plumbing. These tasks are useful because they are bounded and checkable. To make the work more reliable, we published 60+ ML4T agent skills.

The risky step is letting an agent mutate the research claim. In trading and ML, the dangerous errors are often leakage, wrong timestamps, stale data assumptions, broken evaluation windows, accidental survivorship, weak baselines, or transaction-cost assumptions that no longer match the text.

The failure is rarely a syntax error. A more typical case: an agent makes a failing data-loader test pass by dropping the tickers that stopped returning history. The suite goes green, and the backtest quietly runs on survivors only. The code is correct; the research claim has changed.

For an ML4T notebook, loader, or backtest change, an agent-generated patch should record:

• data source and snapshot date;

• timestamp convention and decision-time availability;

• label definition;

• train/test split or walk-forward window;

• baseline comparison;

• transaction-cost assumption where relevant;

• exact command to reproduce the result;

• files changed and why;

• tests added or modified;

• reviewer note on whether the research claim changed.

Do not delegate changes that redefine the target variable, change the backtest window, select the best run after repeated iteration, add data that would not have been known at decision time, or rewrite the published claim to fit the new output. Those choices belong to the research process, not to the coding loop.

On Tuesday, July 7, I am teaching a free Maven Lightning Lesson, Getting stuff done with coding agents. The lesson is the practical companion to this reading list: how to turn an ambiguous goal into a scoped work unit, keep state across sessions, use review gates, and make agent work legible enough that a human can still own the result.

The same boundary sits at the center of Machine Learning for Trading: From Research to Production, the cohort course I am running this summer: it works nine case studies through that exact division of labor — what you can hand to the loop, and what stays with you because it decides whether the result is real. Watch the video overview.

Compressed Reading Path

If the full list is too much, start here:

1. Osmani, Ronacher, Willison, Böckeler, and Huntley for the current loop and harness debate.

2. Newell/Simon, POMDPs, Hearsay-II, options, mixed initiative, automation misuse, and supervisory control for the older design problems.

3. ReAct, Reflexion, Voyager, CodeAct, SWE-agent, and AI Agents That Matter for the LLM-agent mechanism.

4. Evaluating AGENTS.md, Agentic Much?, AIDev, Where Do AI Coding Agents Fail?, Agentic Refactoring, Over-mocked Tests, and ContextBench for field evidence.

5. SWE-bench, RExBench, SaaSBench, SWE-PolyBench, and RedCode for benchmark and safety constraints.

The practical result is a design rule: decide what the agent may change, define how the harness checks it, preserve the evidence needed for review, and keep the research claim outside the automation boundary.

Here is the kind of problem that workflow has to handle.

One of those nine case studies runs the ML4T workflow on thirty CME futures, daily, from 2011 through 2025. The model is a gradient-boosted ranker on a five-day forward return, traded as a long-short book — long the top-ranked contracts, short the bottom, inverse-volatility weighted across both sides. The label that feeds it is built on a ratio-adjusted continuous price series: back-adjusting the price level at each contract roll keeps the roll gap from registering as a price move, so it never lands in the forward returns the model trains on. Leave the series unadjusted and every roll embeds that artifact. Both decisions matter, and so does the validation around them: the result is measured on a holdout never touched during model selection, and against a long-only equal-weight basket of the same contracts.

Put together, they hold up. The validation Sharpe — the inverse-vol book under a 5% trailing-stop overlay — is 1.36, and on the 2024–2025 holdout it comes in at 1.11 — close enough that the profile carries, though the window is short enough that the gap between the two is not statistically resolved. The robust part is the signal itself: a daily information coefficient near 0.03 that clears the 5% significance threshold on both windows (on autocorrelation-robust standard errors, since the five-day labels overlap), and a result that survives the cost model — a Sharpe near 1.0 net of a tick-implied trading cost of about 10 basis points of notional per leg (modeled, not realized fills), against a breakeven near 30 basis points per leg, a roughly threefold cushion. It sits above the long-only basket too (about 0.6 in validation, 0.8 on the holdout), though a long-short book against a long-only directional basket is a floor rather than a risk-matched comparison. In this market the data construction shapes the result just as much as the model family does: get the roll handling wrong and even the cleanest model trains on a price path you could never have held. The case works because the data, the model, the cost model, and the holdout each carry their weight, and the workflow is built to show which stage carries how much.

Validation (left) and a holdout the model never saw (right), each against a long-only equal-weight basket of the same thirty contracts. Validation Sharpe 1.36; holdout 1.11 — the profile carries across the split.

None of those decisions is glamorous on its own: most of a strategy’s fate is settled in the handoffs between stages, not in the choice of model. We ran this loop nine times, and the new cohort is built so you run it too: you walk away with the workflow and one case you took all the way through yourself.

That workflow is the same on every case: a feasibility check, labels and features, a ladder of models, backtesting, portfolio construction, cost and risk analysis, an out-of-sample holdout, the deployment path, and the monitoring after. Running it nine times is what makes the handoffs visible: the decision that carries each case moves from one market to the next, so you learn the pattern instead of one example. Every run is logged, and you keep the notebooks, the run registry, and the deployment scaffold.

The comparative lab

The pipeline is fixed; what it has to handle changes with the market. ETFs anchor the set as a shared reference — daily frequency, an inspectable universe, a cost model you can reason about, an allocator you can deploy — and the other eight show how the same workflow adapts when the market does.

The two S&P rows describe different problems on the same underlying: one uses options data as side information to forecast equities; the other trades the options directly — the same instrument posed as two different trading problems. Each setting shows where an edge comes from, and how much of it survives once costs and capacity are real.

What the workflow forces you to confront

A working pipeline, stage by stage, in the same numbered sequence on every case — one notebook per stage, mapped to the chapter that teaches it:

• Feasibility — confirm the universe and its cost structure can support a strategy before you model anything; the straddle case is a feasibility problem before it is a modeling one.

• Labels and features — forward returns at the horizon you actually trade, walk-forward splits, and features from momentum and carry to model-based ones: walk-forward GARCH, HMM regimes, particle-filtered stochastic volatility.

• A model ladder — linear baselines, Optuna-tuned gradient boosting, neural tabular and sequence models, latent-factor methods, and double machine learning for the causal question.

• Strategy construction — backtest, covariance cleaning with random-matrix methods (following Paleologo’s treatment), an instrument-appropriate cost model, and risk overlays.

• Validation — an out-of-sample holdout scored with deflated performance measures and multiple-testing corrections.

• Production — a deployment loop, then drift monitoring once it is running.

The work that decides the outcome lives in the handoffs between these stages, and one principle keeps reappearing: use only data you would have known at decision time. In the CME opener it was building the label on a roll-adjusted series, so the model never learns from a continuation no one could have traded. In NASDAQ microstructure it is bar-close timestamps — whether an order-book feature was observable when the 15-minute bar closed. In the firm-characteristics panel it is reporting lags and restatements, aligning each fundamental field to the date it was actually known. Different markets, same decision, and it separates a tradable result from a notebook artifact — which you only learn by working the case.

The record, and the right to deploy

Model work is where people expect the course to get technical. It does, and the discipline that makes it pay off is being able to prove, six weeks later, exactly what you ran.

Every training run, prediction set, causal estimate, and backtest is written to a run log: a SQLite registry where each entry is content-addressed by a hash of its full configuration, with its artifacts and results recorded against that hash. A hyperparameter sweep becomes an auditable archive you can query rather than a folder of guesses. That is what makes a model comparison honest: the synthesis step reads the registry to recover how many configurations you effectively tried, then scores each candidate against that count — deflated and probabilistic Sharpe ratios, multiple-testing corrections, the probability of backtest overfitting, and a block-bootstrap comparison against an equal-weight benchmark. (The deflated-Sharpe and overfitting measures are López de Prado’s.) Your best result is judged against the number of attempts it took to find it, and then it meets an out-of-sample holdout it has never touched.

When validation and the holdout agree, you have evidence to carry the strategy forward: paper trading, limited deployment, or further monitoring. The firm-characteristics case is the cleanest version of that agreement: a monthly cross-sectional ranker over roughly 2,500 US stocks whose validation Sharpe of 2.75 carries to 1.77 on a holdout the model never saw, with its drawdown profile holding intact alongside the Sharpe — the cleanest such transfer of the nine. When validation and the holdout disagree instead, the course teaches you to read why — whether the prediction degraded, the portfolio translation lost it, or the regime changed. Where a causal question is well defined, a double-ML treatment-effect estimate with a refutation test asks whether the relationship survives a different specification — a separate question from whether the strategy is tradable after costs. That diagnostic ability is what lets you tell the strategies worth more of your time from the ones that are not.

Through deployment, and after

A research result is not the finish line. The deployment chapters run the strategy as a live loop: a unified framework that shares feature and signal code between research and live paths so they cannot silently diverge, a verification step that checks that parity, an order state machine, broker integrations for paper and live trading, and runtime safety controls. In the cohort, deployment means standing up that loop and its controls; committing capital stays your decision. The monitoring chapters cover what happens once it is running: drift detection on features and predictions, circuit breakers, safe model rollout, and a feature store so the live system reads the same features it was trained on. This is running code — the ETF, crypto-funding, and FX cases each have a working deployment loop.

In the cohort

You take one case study — or your own data, if it fits the scope — and run the arc end to end: a clean baseline, a hypothesis about its binding constraint, two or three targeted cycles, each change tested against an out-of-sample holdout under the same protocol. Some changes hold up; some do not, and the cohort is built around learning to tell them apart. What you produce is a research package and the verdict it earns — advance, revise, or retire — scored against a rubric you see on day one.

Coding agents work inside that same protocol. The third edition ships an ml4t-skills companion — 61 skills mapping specific failure modes (leakage, lookahead, purging, cost models, capacity, deployment) to checks an agent runs in your editor, browsable now with a free account at https://www.ml4trading.io/skills/. Agents speed up loaders, tests, and ablations; the skills keep that speed inside the research protocol so the code stays correct as it gets faster.

Logistics

The first cohort starts Monday, July 6, 2026, and the first live session is Thursday, July 9, at 4 PM UTC. The format is eight weekly 90-minute cohort sessions on Thursdays — twelve hours of live class — plus four biweekly 60-minute office-hours sessions, and a final 30-minute one-on-one discussion with me about your project. Every session is recorded and posted within a day, so nothing is lost if you miss a live hour. Between sessions, support runs through a dedicated Discord server, so questions on your case do not have to wait for the next session.

The book publishes in the coming weeks: chapters one through ten are already public, with the case-study code rolling out after, and cohort members get everything they need from day one. Several cases run on free, open data; the intraday microstructure work uses institutional AlgoSeek data, provided to every cohort member along with the rest of the course material.

The best fit is a reader who can already build a complete strategy and wants to get better at the research itself. By the end you have a complete case-study research package — the workflow, the diagnosis and experiments behind your result, your out-of-sample evidence, and the means to repeat all of it on your own data.

See course page and overview video for more details.

The first release covered the data side: market data, microstructure, alternative data, synthetic data, and the data/ package. This release adds the next layer: the trading setup, labels, features, diagnostics, validation checks, and point-in-time inputs.

Here, model-ready evidence means something practical: the target, feature timing, validation split, and search history are explicit enough to inspect before model selection starts. A lot can go wrong before an estimator enters the workflow: the label can use the wrong horizon, the feature can belong to the wrong decision time, the cross-validation split can mismatch the trading cadence, or the result can come from too many uncounted trials.

Across the five chapters, the release adds 43 notebooks:

• Chapter 6: Strategy Research Framework — 2 notebooks

• Chapter 7: Defining the Learning Task — 9 notebooks

• Chapter 8: Financial Feature Engineering — 8 notebooks

• Chapter 9: Model-Based Feature Extraction — 15 notebooks

• Chapter 10: Text Feature Engineering — 9 notebooks

The release also updates the case-study directory with shared utilities and each case study’s config/setup.yaml: universe, costs, walk-forward CV, and labels.

We will work through the same research-to-production workflow as in the live Machine Learning for Trading: From Research to Production course, which starts Monday, July 6.

Part II connects the data chapters to the model chapters: research setup, labels, features, validation, and model-ready inputs.

Where to start

If you want the shortest path through the release:

• Start with 06_strategy_definition/01_cv_foundations for walk-forward validation, purging, embargo, and nested CV.

• Start with 07_defining_the_learning_task/03_label_methods if the prediction target is still unclear.

• Start with 08_financial_features/05_feature_selection if the feature set is too broad.

• Start with 09_model_based_features/11_hmm_regimes if the regime state is part of the signal story.

• Start with 10_text_feature_engineering/09_filing_text_signals if you need point-in-time NLP alignment.

Chapter 6: Strategy Research Framework

Chapter 6 defines a trading strategy as an executable decision process, not just a signal or model. It specifies the market, universe, decision schedule, tradability rules, score-to-position mapping, costs, constraints, evaluation metrics, and run logging.

Chapter 6 separates the live trading loop from the research loop: one runs the strategy on a fixed cadence, the other improves the pipeline under a fixed setup and evaluation protocol.

The chapter also introduces the nine case studies used throughout the book, enabling strategy research to be compared across asset classes, trading cadences, and market structures.

Notebook highlights:

• 01_cv_foundations builds walk-forward cross-validation from first principles, including decision-time admissibility, purging, embargo, calendar-aware splits, nested walk-forward validation, and combinatorial purged CV.

• 02_case_study_overview summarizes the nine case studies, including datasets, asset classes, constraints, evaluation protocols, and prediction coverage across time.

Chapter 7: Defining the Learning Task

Chapter 7 turns raw but validated data into a learning problem. It builds split-aware preprocessing, label engineering, feature-label evaluation, multiple-testing checks, and lightweight causal falsification.

Chapter 7 uses the triple-barrier method to label an event by whichever barrier is hit first: the profit target, the stop-loss, or the maximum holding period.

The useful move is that labels are treated as trading decisions, not just target columns. Horizon, overlap, resolution time, and implied trading intensity all contribute to the target definition.

Notebook highlights:

• 01_data_quality_diagnostics checks the ML4T datasets for index integrity, missing values, duplicates, outliers, and temporal gaps.

• 02_preprocessing_pipeline implements split-aware preprocessing, including a SplitAwarePreprocessor method that fits scalers and encoders on training data only.

• 03_label_methods implements fixed-horizon, percentile, triple-barrier, ATR-based, trend-scanning, meta-label, and sequential-bootstrap labels on ETF data.

• 05_signal_evaluation and 06_ic_inference use Information Coefficients, quantile returns, signal decay, HAC standard errors, and block bootstrap to evaluate single-factor signals.

• 07_multiple_testing covers factor-zoo selection bias, Benjamini-Hochberg FDR control, complexity-aware corrections, the Deflated Sharpe Ratio, and the Probability of Backtest Overfitting.

• 08_causal_sanity_checks uses timing placebos, shared-driver checks, and VIX regime heterogeneity tests to ask whether a signal story survives basic falsification.

• 10_ml4t_library_ecosystem shows the ml4t-data, ml4t-engineer, and ml4t-diagnostic libraries used across Chapters 7-12.

Chapter 8: Financial Feature Engineering

Chapter 8 moves from a trading idea to a documented feature specification: what the feature is supposed to capture, which horizon it belongs to, which reference frame it uses, and how it should be lagged.

Chapter 8 organizes feature families by data requirement and role: direct signals, state variables, feasibility inputs, and context.

The chapter builds direct financial features from prices, volume, liquidity, microstructure, term structure, cross-asset relationships, options markets, fundamentals, macro data, calendars, and events. It also includes feature selection and sensitivity checks, so it is not just a catalog of indicators.

Notebook highlights:

• 01_price_volume_features builds multi-horizon returns, trend features, volatility estimators, volume-liquidity features, ranks, and z-scores from ETF data.

• 02_microstructure_features computes Kyle lambda, Amihud illiquidity, Roll spread, and order-flow imbalance from NASDAQ ITCH data.

• 03_structural_cross_instrument_features builds carry and roll-yield features from CME futures, rolling beta and lead-lag features from ETFs, and options-implied features from S&P 500 options.

• 04_fundamentals_macro_calendar builds value, quality, and growth features from point-in-time financial statements, plus yield-curve, credit-spread, VIX, and calendar features.

• 05_feature_selection reduces a large ETF feature set using IC ranking, correlation deduplication, hierarchical clustering, BH-FDR filtering, bootstrap stability, and LightGBM importance.

• 06_robustness_sensitivity sweeps momentum parameters, checks IC by VIX regime, and builds signal-times-state interaction features.

• 07_event_studies implements event studies around ETF momentum breakouts, including normal-return estimation and CAAR confidence bands.

• case_study_feature_summary compares feature results across case studies, including IC distributions, FDR survival rates, and feature-family effectiveness by asset class.

Chapter 9: Model-Based Feature Extraction

Chapter 9 turns fitted models into features. These are not the final supervised models being compared later. They are inputs: filtered states, residuals, volatility estimates, uncertainty summaries, regime probabilities, and panel-level relationships.

Chapter 9 turns fitted models into features, including filtered regime probabilities from Hidden Markov Models without using future observations.

A fitted model can compress structure that raw rolling features miss: latent state, changing volatility, structural breaks, uncertainty, cycles, and pairwise relationships. The point-in-time distinction matters here: filtered regime probabilities use only information available up to the decision time, whereas smoothed probabilities incorporate future observations and would leak.

Notebook highlights:

• 01_visual_diagnostics, 02_structural_breaks, and 03_fractional_differencing cover stationarity tests, ACF/PACF analysis, rolling diagnostics, break detection, and fixed-width fractional differencing.

• 04_kalman_filter extracts level, slope, innovation, and uncertainty features, and estimates time-varying hedge ratios for pairs trading.

• 05_spectral_features uses wavelets, rolling FFT features, Welch power spectral density, and spectral heatmaps.

• 07_arima_features, 08_garch_volatility, and 09_har_rough_volatility turn time-series models into residual, forecast, uncertainty, conditional volatility, persistence, leverage, and rough volatility features.

• 10_uncertainty_features uses PyMC stochastic-volatility models and ARIMA forecast uncertainty to build uncertainty inputs.

• 11_hmm_regimes, 12_wasserstein_regimes, and 13_regime_as_feature compare parametric and distribution-based regime features, including the difference between filtered and smoothed probabilities.

• 14_panel_features builds cointegration, Kalman hedge ratio, Ornstein-Uhlenbeck half-life, ranking, sector-relative, and universe aggregation features.

• case_study_temporal_summary compares model-based feature inventories and incremental IC contributions across case studies.

Chapter 10: Text Feature Engineering

Chapter 10 treats financial NLP as feature engineering. It starts with lexicons, bag-of-words, TF-IDF, and static embeddings, then moves through sequential models, Transformers, domain adaptation, fine-tuning, and text-derived trading signals.

Chapter 10 moves from dictionaries and bag-of-words to contextual Transformer representations such as BERT and FinBERT.

The practical focus is timestamp-safe text features: when the text was available, which entity or asset it maps to, how the signal is aggregated, and how it is evaluated against future returns.

Notebook highlights:

• 01_word2vec_training trains Word2Vec on Financial PhraseBank text and evaluates embeddings with similarity, analogies, and t-SNE.

• 02_asset_embeddings applies Word2Vec to SEC 13F holdings, treating portfolios as sentences and stocks as words.

• 03_sentiment_evolution compares TF-IDF, GloVe, and FinBERT on Financial PhraseBank and shows why distribution shift matters.

• 04_bert_finetuning fine-tunes FinBERT, DeBERTa-v3, and ModernBERT for financial sentiment classification.

• 05_financial_ner_finetuning fine-tunes a FinBERT-based named-entity recognizer for companies, amounts, and dates.

• 07_news_return_signals builds news surprise and sentiment factors from FNSPID news and evaluates them against S&P 500 forward returns.

• 08_text_feature_evaluation evaluates text-derived alpha signals with IC, ICIR, t-statistics, and quintile spreads across 1-day, 5-day, and 20-day horizons.

• 09_filing_text_signals builds 10-Q MD&A sentiment and narrative-change signals and joins them to S&P 500 returns with point-in-time alignment.

These notebooks prepare research inputs and diagnostics. They do not claim that a feature survives full strategy simulation, transaction costs, portfolio construction, risk controls, or live deployment; those checks come in later releases.

Learn the workflow live

The full live-course and workshop schedule is on the ML4T courses page.

The Machine Learning for Trading: From Research to Production course starts Monday, July 6. The course uses the same workflow in live work: start with a strategy idea, build a baseline, diagnose what failed, and decide what to try next without quietly reusing the holdout.

The repository gives you the code. The course adds review and pace.

You can watch or star the ML4T repository to follow the staged release. The next releases move into model comparison, simulation, portfolio construction, transaction costs, risk, and live deployment.

A panel of returns. A table of fundamentals. A calendar of releases. A cross-section of firm characteristics. A feature matrix that tries to preserve what the model could have known at the decision time.

That remains the center of gravity. The third edition still treats data design, labels, baselines, backtests, costs, and monitoring as the hard parts of the workflow. Forecasting agents add an evidence-processing layer to that workflow.

The layer is specific: it can process current unstructured evidence on the fly and turn the result into a structured forecast artifact.

For finance, that evidence might be a central bank statement, company guidance, a filing, a research note, macro commentary, or market chatter that has not yet become a stable feature history. For sports, it might be squad news, injuries, lineup changes, travel, form, or match reports. The domains differ. The pattern is the same: some relevant information arrives as text, search results, or messy context before it arrives as a clean panel.

This Saturday, June 27, we are building this pattern in a live Maven workshop: Building Multi-Agent Forecasting Systems. The workshop uses an AIA-style multi-agent forecaster inspired by Bridgewater AIA Labs: research agents gather evidence; an aggregation layer combines their probabilities; a supervisor checks the result; calibration adjusts the final number; and every run is stored so it can be scored later.

The system should read evidence that a conventional model may miss, produce a probability, and then have that probability compete with a baseline.

The missing input is often not another price series

In a clean modeling setup, the training panel defines the world. If the feature is not in the panel, the model cannot use it. That discipline is important because it prevents hindsight from leaking into the experiment.

But real decisions often happen before the relevant information has been converted into a feature. A company may update guidance, a regulator may change tone, a central bank may revise its language, or a prediction market may move before the event has resolved. Some of that evidence would be hard to reconstruct across twenty years of history.

The same issue appears outside finance. In the World Cup forecaster, a statistical model can estimate the probabilities of a win, draw, or loss from historical match data. Match-specific information is a different kind of input: squad updates, recent form narratives, injury reports, expected lineups, conditions, and tournament context.

An agent can put that evidence into the workflow without pretending it has already become a clean historical feature.

The agent can search, read, summarize, compare sources, reason over contradictions, and emit a probability with a trace of what it used. That output is not automatically correct. It is a candidate input.

The test is whether the candidate input improves the forecast record, the diagnosis, or the decision process when compared with a baseline.

Web search is the easy version

The simplest forecasting agent uses a web search.

Give the agent a binary question, a cutoff date, a search tool, and a prompt that asks for evidence, uncertainty, and a probability. Let several agents search independently. Aggregate their probabilities. Ask a supervisor to check the disagreement. Store the trace. Score the result when the event resolves.

That already teaches most of the system design: resolvable questions, cutoff-aware retrieval, source metadata, schema-bound outputs, aggregation policy, calibration, and baseline scoring.

The current search is convenient but not replayable. Search rankings change. Pages move. Articles update. Snippets may reflect information unavailable as of the forecast date. Date filters help, but they do not create a clean historical archive. If you want to tune the system rather than merely demo it, a point-in-time evidence archive becomes much more attractive.

For finance, that might mean timestamped news, filings, transcripts, macro releases, prediction-market snapshots, and internal research notes. For sports, it might mean dated match previews, squad announcements, injury updates, odds, and official lineups. The important feature is not that all of this is text. It is that the system can know what was available at the time.

Once the evidence is archived point-in-time, the agent becomes easier to improve:

• You can backtest retrieval policies without letting the agent read the future;

• You can compare prompts, tools, and agent counts on the same historical evidence;

• You can run prompt or program optimization against resolved outcomes;

• You can measure whether a supervisor helps or merely adds cost;

• You can decide whether debate improves calibration or only produces longer traces.

A live demo can run on the current search. A system that improves over time needs evidence that it can replay.

Two practical ways to use the forecast

Two deployment patterns cover most practical use cases.

The first is to build the agent in isolation as an additional forecast. It reads evidence, produces a probability, and gets compared with a market price, a statistical baseline, or a human forecast. If it provides incremental information, the output can be ensembled with the baseline or used as a feature in a downstream model.

That is the restrained interpretation of the AIA Forecaster result. On ForecastBench, the report finds that performance is statistically indistinguishable from that of human superforecasters. On MarketLiquid, the paper evaluates 322 liquid prediction-market questions at five forecast dates each, producing 1,610 forecast instances from April 2 to May 23, 2025. The agent alone trails the market consensus there. The practical result is that combining the agent forecast with market consensus beats either source alone on that benchmark.

That is a realistic role: independent, inspectable evidence processing that can add information when combined with a strong baseline.

The second pattern is to enrich the agent with structured inputs from the start. It receives a model prior, market odds, recent structured data, and the current evidence it should inspect.

That is what we are doing in the World Cup agent: model the prior first, current evidence second, and a separate forecast surface throughout.

The World Cup app illustrates this second pattern. A Poisson-style team-strength model provides the prior; the agent reads the current football context and publishes its own win/draw/loss forecast alongside the model. In one pre-match validation run, changing the prompt increased the agent’s mean maximum swing from the model by 3.5 percentage points across eight group-opener fixtures. The goal was to keep the model prior visible while allowing the agent to move when supplied evidence justified it.

If the agent simply rubber-stamps the model, it adds little. If it makes large moves without evidence, it adds noise. Evidence-driven deviations are the cases to track and evaluate.

The AIA-style architecture

Chapter 24 of Machine Learning for Trading uses forecasting because the output can be scored. A forecast has a timestamp, a probability, a resolution rule, and an eventual outcome.

For prediction-market questions, the target is usually a binary event. For match forecasts, the outcomes are mutually exclusive. The architecture is similar, but the scoring layer changes.

The architecture we use in the workshop follows the same practical sequence:

An AIA-style forecasting system turns a question into research-agent traces, an aggregate forecast, supervisor review, calibration, and a persisted probability that can be scored later.

1. Define the forecast target and cutoff policy.

2. Run several research agents that search and reason independently under the same cutoff policy.

3. Aggregate their probabilities under an explicit rule.

4. Optionally run a debate or specialist review stage when the question set warrants the cost.

5. Let a supervisor inspect disagreements and perform clarifying searches when needed.

6. Calibrate the final probability.

7. Persist the run: question, inputs, traces, configuration, token usage, probabilities, and later scores.

This is a read-only forecasting system. It can pull questions, prices, search results, and evidence. It does not place trades or bets.

That boundary makes the system easier to evaluate. A read-only agent can be wrong without also creating an execution problem. The first product of the system is a probability and a record. Action comes later, if the probability survives comparison with the rest of the workflow.

What has to be measured

Forecasting agents invite a familiar mistake: treating a good explanation as evidence of a good forecast.

The explanation may be useful. It may help diagnose what the system saw. It may reveal a stale source, a missing counterargument, or a brittle prompt. But the forecast has to be judged by scoring rules and baselines.

Calibration remaps probabilities based on past forecast errors. It cannot add evidence the agent missed, and it should be evaluated on a disjoint set of resolved forecasts.

At a minimum, the evaluation should ask:

• Does the agent improve Brier score, log score, calibration, or sharpness relative to the baseline?

• Does the improvement survive time-disjoint evaluation?

• Does the agent add information when combined with market consensus or a statistical model?

• Which component actually helps: search, more agents, debate, supervisor, calibration, or a better evidence archive?

• Does the benefit justify the cost and latency?

A negative result still tells you which component not to keep. An agent may fail to beat a baseline on its own and still improve an ensemble or help diagnose model blind spots. If it only produces plausible prose, it should not survive the ablation.

This is also why point-in-time evidence matters. Without it, a backtest can become a story about leakage. With it, the same architecture can become a repeatable experiment: the same question, the same cutoff, the same archive, different prompt or tool policy, scored after resolution.

What we will build on Saturday

The workshop is built around a working AIA-style forecaster.

Students run a CLI and dashboard, inspect individual agent traces, change profiles, compare single-agent and multi-agent runs, look at aggregation and calibration, and see how forecast runs are stored for later scoring. The deterministic replay path works without API keys. Live model and search providers are optional upgrades.

The day is deliberately practical:

• start from a resolvable question;

• run one research agent and inspect its evidence trail;

• move from one agent to several;

• aggregate and calibrate the probabilities;

• store the run so it can be evaluated later;

• discuss how the same pattern generalizes from prediction markets to finance, sports, and other decision workflows.

The ML4T link is the evaluation habit: preserve the timestamp, compare with a baseline, and remove components that do not improve the record. Forecasting agents make a new class of inputs available to that workflow.

The live workshop is this Saturday: Building Multi-Agent Forecasting Systems.

The free Lightning Lesson page is here: Build Multi-Agent Systems You Can Audit.

In the workshop, a component earns its place only if the stored runs show that it improves the forecast record, the diagnosis, or the evaluation against the baseline.

Chapter 1, The Process Is Your Edge, sets up the workflow: define the research problem, keep exploration separate from confirmation, and treat live degradation as something the process has to handle. Chapters 2-5 then make the data foundation inspectable.

Data infrastructure supports strategy research and live deployment. This release provides access to the book's coverage of the infrastructure layer: the data and chapter notebooks that later case studies build on.

This release also arrives as we start teaching the ML4T workflow live. The free From trading idea to validated strategy Lightning Lesson runs Wednesday, June 24, at 16:00 UTC / noon ET; the full Machine Learning for Trading: From Research to Production course starts July 6. The course is organized around this loop: start with a research question, build the data contract, cross the evidence boundary only when the experiment deserves it, and carefully track the live feedback loop.

In trading research, few things are more important than the information contained in data, and how to handle it properly: clocks, sessions, adjustments, revisions, identifiers, contract rolls, venue rules, licensing limits, and the question of what a strategy could have known at the time of the decision. A later model result only means something once those choices are visible.

A point-in-time accounting example from Chapter 4. The final database view shows the same $100M revenue value across the year; the lower track shows what was actually available after the original filing and later restatements.

This release adds the first five chapter directories and the central data/ package that later case-study work builds on:

• Chapter 1: The Process Is Your Edge

• Chapter 2: The Financial Data Universe

• Chapter 3: Market Microstructure

• Chapter 4: Fundamental and Alternative Data

• Chapter 5: Synthetic Financial Data

• The data download package

Chapter 1 adds the workflow frame. The four data chapters contain 62 notebooks. The data/ directory adds the catalog, download scripts, loaders, configs, and access notes that turn the chapter examples into something a reader can inspect and run. The implementation detail worth checking first: loaders return Polars DataFrames through a consistent API, and missing data raises setup instructions with explicit download guidance.

What the data layer contains

The top-level data catalog currently lists 31 dataset entries. Financial data does not fit cleanly into one giant downloadable bundle. Some sources can be fetched without an API key. Some need a free key. Some are paid. Some require a manual provider download. Some reduced licensed packages are still being prepared for hosting.

The catalog groups entries around the constraints a research workflow inherits: daily market data, microstructure feeds, options data, fundamentals, positioning, macro series, prediction markets, on-chain metrics, news, and text. That structure makes setup expectations and redistribution limits visible before a reader spends time trying to reproduce a notebook.

The synthetic-data chapter takes a different angle. It asks how to create alternative histories for robustness analysis when one realized market path is not enough. The notebooks start with classical simulation and move through TimeGAN, Tail-GAN, Sig-CWGAN, GT-GAN, Diffusion-TS, LLM-based tabular generation, and differentially private GAN training. The diagnostics focus on stylized facts, dependence structure, downstream task utility, and privacy constraints.

Chapters 2-5 in one pass

Chapter 2 gives the map. It covers the financial data universe by asset class and source type, then moves into due diligence and storage. The practical thread is that provider data already contains research choices: corporate actions, survivorship, contract construction, point-in-time availability, storage format, and update policy.

Chapter 3 gets closer to the tape. Market data is the output of market design: sessions, order types, venue rules, visible liquidity, and timestamp conventions. The notebooks parse NASDAQ TotalView-ITCH, reconstruct limit order books, validate trade-direction classification, compare bar-sampling rules, and show why the sampling clock changes the return distribution a model sees.

A Chapter 3 notebook turns the same AAPL trading day into one-minute bars, 10,000-share volume bars, and tick imbalance bars. The choice of sampling clock changes the observations a model receives before feature engineering begins.

Chapter 4 turns point-in-time correctness into implementation work. It covers SEC EDGAR and XBRL data, Form 4 insider transactions, 13F holdings, entity resolution, macro release alignment, CFTC futures positioning, on-chain fundamentals, prediction markets, and the extraction of filing text for later NLP work. Financial information must be aligned with the time it became available, as well as the fiscal period or event it describes.

Chapter 5 handles synthetic data with a practical standard: fidelity, utility, and privacy. Generated histories are useful when they stress robustness claims and privacy constraints; extra sample count alone cannot make a backtest more convincing.

From dataset to case study

The third-edition case studies begin with a market, a universe, a decision schedule, and a data definition. The case-study table in the main README is already public; the detailed case-study notebooks will be included in later release batches.

The futures path shows the point. A CME futures example starts with contracts, sessions, UTC timestamps, roll rules, continuous-series construction, and positioning data that arrives on its own schedule. A term-structure signal built on that data depends on each of those choices before any model sees a feature matrix.

The firm-characteristics case has a different failure mode. A monthly cross-sectional result only means something if the accounting data is lagged and aligned to a point-in-time before labels and features are built. The README lists the remaining nine case studies, including the distinct S&P 500 equity-plus-options and options-only tracks.

That is why the data release comes before the model chapters. Later chapters can ask whether a model improves a score. This release sets up the prior check: whether the score was computed on a dataset whose timing, universe, costs, and availability rules would have held up against the actual research problem.

How to inspect the release

A narrow first pass works better than browsing every notebook.

Start with data/README.md. It explains the dataset catalog, access categories, loader names, storage tiers, and the quick-start commands. The repo expects the data path in a root .env file:

# In the repository root .env file
ML4T_DATA_PATH=/path/to/your/data

# Then download free datasets
uv run python data/download_all.py --free-only

Then pick one path.

If you want the lightest local setup, start with ETFs, factors, and crypto. The minimum tier is about 70 MB. Adding macro and FX data still leaves the standard tier around 75 MB, assuming the free API keys are configured. Adding the broad US-equities dataset takes the footprint to roughly 740 MB. Adding CME futures brings it to roughly 825 MB. The full setup, including the larger microstructure and reduced licensed packages, is closer to 7 GB.

That separation matters because a reproducible notebook is only reproducible within the data rights and access assumptions it declares.

A practical first pass is:

• open the data/ README;

• run one free dataset download;

• open the matching chapter notebook;

• inspect the loader output and canonical timestamp/entity columns;

• trace that dataset into the case-study table in the main README;

• then move to the Chapter 6 feasibility notebook when that batch lands.

The next ML4T Insights issues will follow the same sequence: data first, then strategy definition, labels, features, model comparison, backtesting, costs, risk, and deployment.

For the walkthrough

The June 24 Lightning Lesson starts from this data contract: choose a market, define the decision problem, build a first baseline, diagnose what failed, and make the next iteration measurable without quietly reusing the holdout.

The July 6 cohort uses the same case-study structure as the book and repo, with sequencing, feedback, and live review.

The repo gives readers the artifacts. The course adds the working cadence: turn the artifacts into a baseline, read what the baseline says, and decide what deserves the next experiment.

You can watch or star the repo to follow the staged release. The lesson is a short live walkthrough of why the release starts with the data contract before moving to models.

Star history for the public ML4T repository, showing steady growth from 2019 through 2026. The third-edition rebuild starts from an existing reader base that has accumulated over several years.

The third-edition README is now live on the main branch. The older material is preserved on the first-edition and second-edition branches. The new code will be rolled out in stages over the next few weeks leading up to the July launch.

That staged release has a practical reason. A large trading-ML repo can easily turn into a larger notebook archive: more models, more datasets, more examples, more ways to get lost. The third-edition rebuild is organized around a different question:

When a forecast looks promising, what has to be checked before it becomes a trading decision?

The third-edition workflow separates strategy research from evaluation with an evidence boundary, then closes the loop through deployment, monitoring, and the decision to retrain, pause, or retire a strategy.

That is also the question behind next Wednesday’s free Lightning Lesson “From trading idea to validated strategy”. The repo is the public resource; the lesson is the short guided walkthrough.

The work after the forecast

A familiar research sequence goes like this. A model produces a plausible return forecast. The validation IC improves. A backtest looks better than the baseline. Then the harder checks begin.

Was the asset universe fixed before the test? Did the label line up with what was knowable at the time of the decision? Were overlapping labels handled correctly? How many variants were tried before this one survived? Did the result clear the cost model? Did turnover make the signal unusable? Does the portfolio rule amplify a weak forecast or diversify it? What happens after position limits, drawdown rules, and execution assumptions are applied?

The book calls one part of this the evidence boundary: the line between exploration and confirmation. On one side, you search, tune, compare labels, test feature families, and improve the baseline. On the other side, you spend a holdout on a result that was already specified well enough to deserve the test.

That boundary changes the decision a result can support. A strategy can meet a naive Sharpe threshold and still fail once the number of variants tested is taken into account. A feature can look predictive until its timestamp is checked. A model can improve rank correlation and still lose once costs and turnover are factored in.

At that point, the right response is to return to exploration, narrow the claim, or stop. The holdout has already answered the question it was allowed to answer.

That is the practical center of the third-edition workflow.

Three ways into the repo

The new README is the route map for the staged code release. It shows three useful ways to enter the material as the directories fill in.

First, enter by workflow stage. If the problem is data reliability, start with the financial data layer. If the problem is label design, feature triage, or leakage, start with research design and feature engineering. If the problem is model comparison, start with the model chapters. If the backtest looks too clean to believe, start with portfolio construction, costs, risk, and strategy synthesis. If you are working with RAG, knowledge graphs, or agents, start with the advanced-AI chapters. If the issue is live operation, start with deployment, monitoring, and the retrain-pause-retire loop.

Second, enter by case study. The case-study page lists nine studies across seven asset classes, 168 notebooks, and six pipeline stages. Each one puts pressure on a different part of the process:

• ETFs are a cleaner daily cross-asset setting for following the full loop.

• Crypto perpetuals introduce funding rates, shorter decision intervals, and a market structure that differs sharply from equities.

• NASDAQ-100 microstructure makes bar construction, order flow, and intraday costs part of the modeling problem.

• S&P 500 equity plus options uses implied-volatility information to improve equity selection.

• US firm characteristics revisit the canonical monthly cross-sectional factor problem, where point-in-time data discipline does much of the work.

• FX pairs expose the limits of a small cross-section when shared macro and dollar factors are present.

• CME futures force rolls, term structure, and sector structure into the foreground.

• S&P 500 options test whether an options-only strategy can survive labels, hedging, and instrument-specific costs.

• The broad US equities panel asks whether weak individual signals become useful when scaled across thousands of names.

Third, enter by library layer. The current library page lists six workflow-aligned packages: ml4t-data, ml4t-engineer, ml4t-models, ml4t-diagnostic, ml4t-backtest, and ml4t-live. A reader can use a single layer without adopting the entire stack. A data team may only need acquisition and storage. A researcher may only need labels, features, and diagnostics. A course participant may start with a baseline and then decide whether the next change belongs in data, labels, validation, costs, allocation, or execution.

What to inspect first

The first release wave is the data and research foundation. That is less fashionable than agents or deep learning, but it is where many strategy problems start.

The data chapter spans market, fundamental, and alternative data; corporate actions; futures and continuous contracts; options; crypto; FX; point-in-time validation; survivorship; provider comparison; incremental updates; and storage benchmarks.

The microstructure chapter moves from raw exchange messages to feature-ready bars: NASDAQ ITCH parsing, limit-order-book reconstruction, order lifecycle analysis, TAQ-style examples, Lee-Ready validation, alternative bar sampling, and intraday jump detection.

The synthetic-data chapter is another natural deep dive. Since the second edition, the topic has moved well past “can a GAN produce realistic-looking prices?” The current problem is more practical: whether a generator preserves the structure a downstream research task needs, without leaking the original data or creating a world that is easier than the real one. The chapter covers classical simulation, TimeGAN, Tail-GAN, Sig-CWGAN, diffusion-based generators, LLM-based tabular generation, and privacy-aware variants.

If you open the repo as the release unfolds, a reasonable first inspection path is:

• pick one case study instead of browsing every notebook;

• read the data construction before the model notebook;

• locate the label definition, horizon, and decision timestamp;

• check the validation split, purge, embargo, and holdout rule;

• inspect cost assumptions before performance statistics;

• treat the first backtest as a diagnostic, not evidence;

• look for the handoff from forecast output to portfolio rule, risk limit, and deployment decision.

That is also the editorial path for the next set of ML4T Insights issues. The useful issues will turn the release into resources readers can keep: a financial-data failure-mode map, a guide to when tick data changes the research problem, an update on synthetic financial data after GANs and diffusion, a case-study access map, and a checklist for when a backtest should count as evidence.

For the first time: live course and workshop

The repo is the public resource. The course and workshop are the guided version of the same workflow. They aim to help you improve your ML4T research and trading practice, not just repeat what’s in the book.

On Wednesday, June 24, there are two free Maven Lightning Lessons:

• From trading idea to validated strategy at 16:00 UTC

• Build Multi-Agent Systems You Can Audit at 15:00 UTC

The first lesson is the closest companion to this release. It walks through the same transition from idea to baseline, from baseline to diagnosis, and from diagnosis to a cleaner second version without quietly reusing the holdout.

The second lesson connects to the agent material in the later chapters and to the forecasting-agent workshop. It uses the same discipline in a narrower setting: define the task, preserve the trace, compare against a baseline, and score the output.

The full Machine Learning for Trading: From Research to Production course runs July 6 to August 29. It is the live cohort version of the workflow: choose a case-study path, build a baseline, diagnose failure modes, iterate honestly, and defend the result against a published rubric.

The Building Multi-Agent Forecasting Systems workshop runs June 27. It is the hands-on route through the agent side: multi-agent forecasting, trace capture, ablations, and scoring.

Neither course is required to use the public repo. The reason to take one is sequencing, feedback, and time spent applying the workflow with other people.

Start with the map

The practical next step is to watch or star the repo if you want to follow the code release as it unfolds.

The useful test for this release is simple: can a reader open the repo, pick a market, and understand what has to be checked before a model result becomes strategy evidence?

Machine Learning for Trading, 3rd Edition on GitHub

Morocco reached the semifinal in 2022. Costa Rica topped a 2014 group that included Uruguay, Italy, and England. Germany won the tournament in 2014, then went out in the group stage in 2018 and again in 2022. The tournament has a long record of embarrassing anyone too sure about it.

That is exactly what makes it a fun public forecasting lab. The questions are the ones everyone already has an opinion on, the answers come back within hours, and once the match is over, a forecast has very little room to talk its way out of a bad call.

By the morning of June 16, the World Cup Forecast Agent had already built a record worth picking apart. Spain was a 75.6% favorite in the model and an 80.0% favorite in the agent’s call; Cape Verde held Spain to 0-0. Saudi Arabia-Uruguay and Iran-New Zealand ran the same way: the agent nudged the favorite higher, and both matches finished level.

That is the reason for publishing the lab while the tournament is still underway. Every pre-match number is locked, the trace behind the agent’s adjustment is stored, and the call is scored once the result is in.

Before kickoff, the traces can read sensibly. A few hours later, the scoreboard may have a different opinion.

A June 16 fixture strip from the live app. Each row shows the model call and the agent call before kickoff, so readers can see the adjustment before the result arrives.

See below for a free lightning lesson about building multi-agent forecasting systems.

The model gives the prior

The baseline starts with a familiar statistical idea: estimate each team’s attacking and defensive strength from international results, weight those results by competition and recency, then simulate the tournament bracket.

As of the current public run, the model’s leading champion probabilities are:

• Spain: 14.2%

• Brazil: 12.9%

• Argentina: 11.8%

• England: 9.3%

• France: 6.5%

• Portugal: 6.5%

• Germany: 5.0%

• Colombia: 4.6%

Treat those as a dated snapshot. They are where the forecast starts, not where it ends.

The live forecast table keeps model champion probabilities alongside Polymarket and Kalshi prices, rather than blending them into a single consensus number.

The baseline is useful because it is systematic. It is fit on more than 47,000 international results, checked in walk-forward tests, and straightforward to score after the fact. The forecast table sets the model’s champion odds next to Polymarket and Kalshi prices without averaging the three, which keeps the disagreement on the page rather than dissolving it into a single tidy consensus figure.

The baseline is also restricted to variables we can reconstruct over time. Injuries, lineup news, weather, travel, squad value, current form: a model could use all of it if each input were a clean, point-in-time history. Most of it does not. Some arrives as prose; some is patchy; and for many variables, there is no decades-long archive of comparable snapshots from the past. Pour today’s headline into a model that was never trained on yesterday’s headlines, and you have introduced a new bias, not improved the prior.

That gap is the agent’s job. It can work with current, partly structured information without demanding a 50-year feature history. In production, the next step would be to rebuild richer point-in-time datasets and tune prompts, tools, and aggregation rules against historical cases. This app runs the lighter version: hold the quantitative prior steady, let the agent make a separate context adjustment, and score that adjustment on its own.

The agent calls the matchup

Before each match, the agent receives the model’s win/draw/loss probabilities, expected goals, recent form, head-to-head history, availability signals, key players, match conditions, a relative squad-value signal, and, when available, recent search results.

The part closest to Bridgewater AIA Labs’ AIA Forecaster is the research step. The agent first decides what is actually worth checking for a fixture: a late fitness scare, a suspension, a venue condition, a question about one specific player. It then runs a brief Tavily search, retains the sources relevant to the question, and files the query trace alongside the forecast.

The squad-value signal is kept relative by design: rank within the 48-team field, percentile, value ratio, and marquee names rather than raw euro totals. That gives the agent a current-talent check against a prior built from results, without pretending a transfer-market price tag is a starting eleven. It is a serviceable proxy that still carries the usual biases around age, league, club brand, and the transfer market itself.

The trace is what turns “the agent looked into the match” into something a reader can examine. For the June 16 France-Senegal call, the agent ran three searches, kept nine sources, and moved France from 50.3% to 61.7%. The source list included a piece flagging William Saliba as doubtful and a later one that walked some of that worry back. The agent is reading a noisy evidence stream, not the truth; the trace lets a reader see what it leaned on and judge whether the move was earned.

The forecast itself comes from a small ensemble of language model calls. Each one returns win/draw/loss probabilities, a scoreline distribution, and a factor-by-factor account of its reasoning. The system aggregates those passes into a single forecast and publishes it as the agent’s call, set beside the statistical model rather than folded into it.

A single agent call card from the live app. The card exposes the model-to-agent shift, likely score, short rationale, source count, and grading status.

What the live record adds

A tournament forecast table is easy to publish and easy to forget by the weekend. The World Cup Forecast Agent pulls apart the pieces such tables usually blend:

• the model prior

• the market’s outside view

• the match’s current context

• the agent’s reading of it

• the eventual score

A market disagreement is not an injury note. A model probability, a research summary, and a language model’s explanation are three different objects. Holding them apart makes the forecast inspectable and scoreable.

That turns the World Cup into a gentle version of a harder ML4T problem. Many forecasting workflows begin with a quantitative prior and then have to wrap it in messy, recent, and largely unstructured information. The question that matters is whether the agent contributes a distinct, reviewable adjustment around the model or just restates it in livelier language.

Football makes that concrete. The match is played. The call can be checked. The scoreboard does the rest.

The live app is already showing its shape. As of the June 16 morning snapshot, 16 calls had been graded, with agent and model tied at 37.5% top-pick accuracy. The model’s live Brier score was 0.7271 against 0.704 for the naive-prior floor. Lower is better, so the model had not yet pulled clear of the floor, which itself was hitting 43.8% on the same 16 matches. Sixteen games is far too few to render a verdict, but the machinery is already doing its work: publish the call, score the result, let the record stack up.

The scorecard uses multiclass Brier for match outcomes, with signed per-call deltas showing whether each agent adjustment helped or hurt the model’s score. By the end of the tournament, the bar should be higher than “the agent sounded informed.” The adjustment has to improve the scoring record, sharpen the diagnosis, or both.

Why this belongs with the forecasting-agent workshop

The World Cup build is intentionally lightweight. It is the playful cousin of Bridgewater AIA Labs’ AIA Forecaster: start from a prior, pull current evidence, run independent forecast passes, aggregate them, store the trace, and score the call once it resolves.

The differences are worth being explicit about. The AIA technical report includes supervisor reconciliation and statistical calibration that this app does not claim to reproduce. AIA’s result is also more interesting than the usual “agent beats the market” story: on a liquid-markets benchmark, the agent underperformed market consensus on its own, while an ensemble of AIA forecasts and that same consensus beat the consensus alone. That is why this app keeps model, market, and agent in view at the same time.

Inside ML4T, this is the second live forecasting-agent surface. The ML4T Agent Lab tracks AIA-style questions across Polymarket and Kalshi; the World Cup app brings the same discipline, logged priors, sourced adjustments, and scored outcomes, to a tournament anyone can follow.

The workshop pushes these choices in a less-forgiving setting: independent research, reconciliation, calibration, ablations, and replayable traces. The World Cup app is the easy way in. You can pull up a match, disagree with the number, and look at exactly what the agent checked before kickoff.

How to read the live app

There are a few good places to start. The model view carries champion odds, match probabilities, team ratings, and market comparisons. The agent view holds the upcoming and resolved calls: the model-to-agent shift, the factor trace, the searches and sources, the date each was generated. The scorecard is where the misses surface. Once a result lands, every pre-match call is graded against both the outcome and the model.

It will be wrong, and often. Football is low-scoring, noisy, and decided by small events that can swamp reasonable priors. That is most of the reason to publish the record at all.

Watch the matches, argue with the numbers, and check back as the scoreboard fills in.

World Cup Forecast Agent live app

Learn how to build multi-agent forecasting systems

If you want to see how the architecture travels beyond football, next week’s free Maven Lightning Lesson walks through the design of auditable multi-agent systems: Build Multi-Agent Systems You Can Audit.

The full hands-on workshop is here: Building Multi-Agent Forecasting Systems.

The accompanying notebooks test where this architecture helps, where it adds cost, and where a simpler baseline would have been enough.

One example is the adversarial-debate notebook. A bull-bear debate ran for three rounds on a Fed-rate question. The bull ended at 0.78, the bear at 0.35, and the gap stayed fixed at 0.43 across all three rounds. The debate added more than 6,000 tokens and moved the blended forecast by one percentage point.

The result provides a practical lesson for multi-agent systems: the extra-agent stage has to justify its cost.

Multi-agent systems are easy to make look sophisticated. Add several model calls, assign roles to them, ask them to debate, and place a supervisor at the end. The harder question is whether the added structure produces independent evidence, actionable disagreement, better reconciliation, or a better-scored output than a simpler baseline.

Forecasting is a good lab for that question because the output is hard to hide. A probability can be timestamped, compared with a market price or base rate, calibrated, ablated, and scored after the event resolves.

Chapter 24’s full multi-agent forecasting pipeline: specialist research agents feed aggregation, optional debate, supervisor reconciliation, a forecast artifact, and audit artifacts. The question for this issue is which stages earn their cost.

The realistic standard

Bridgewater AIA Labs’ AIA Forecaster technical report is one of the clearest public blueprints for this pattern.

The report describes a multi-agent forecasting system with agentic search, supervisor reconciliation, and statistical calibration. It reports performance that is statistically indistinguishable from human superforecasters on ForecastBench, but it also reports a more sobering result on liquid prediction-market questions: AIA Forecaster underperforms the market consensus on its own. The interesting result is the ensemble. On that harder benchmark, combining market prices with AIA forecasts performs better than either source alone.

The standard for agent systems in this setting is narrower than market replacement. The agent has to add information that survives comparison with a strong baseline.

The Chapter 24 implementation starts with a single ReAct research loop, then adds tool contracts, explicit state, multi-agent forecasting, evaluation, governance, and a separate ML4T research operator. This issue focuses on the forecasting system because it most clearly exposes the multi-agent design.

The pipeline

The forecasting agent example starts with a question that can be resolved - on a prediction market, at a later stage.

“Will inflation be high?” is not enough. A resolved example from the Lightning Lesson is precise enough to score: “Will US core PCE for April 2026 exceed 0.3% month over month in the BEA Personal Income and Outlays release scheduled for May 28, 2026?” The BEA release reported core PCE at 0.2% month over month, so the event resolved “no.”

After the question is fixed, several research agents work independently. Each agent searches for evidence, records what it finds, reasons about the result, and returns a probability with supporting evidence. In the AIA design, these agents are parallel evidence-gathering runs rather than role-playing characters.

The system then combines the probabilities. The simple mean is the reference point. The median and trimmed mean reduce the influence of outliers. Extremization pushes an aggregate away from 0.5 when forecasters are treated as sufficiently independent.

One diversity adjustment used in the Chapter 24 notebooks is:

Here, D quantifies how far the aggregate can deviate from 0.5 after accounting for dependence among forecasters. With three independent forecasters, D = 1.73. At pairwise correlation 0.7, it falls to 1.12. Three correlated agents are only marginally more informative than a single agent.

A supervisor can inspect disagreement after aggregation. The supervisor policy is narrow: identify the source of disagreement, run clarifying searches when evidence is missing, and override the aggregate only when the new evidence clears a stated confidence threshold. Otherwise the aggregate remains the forecast.

Calibration comes last. It can correct systematic probability bias only when estimated on resolved forecasts outside the run being scored. It cannot repair a contaminated question, a weak evidence process, or three agents that all missed the same fact.

The stored forecast record should contain the question, resolution source, cutoff, evidence references, agent forecasts, aggregate, supervisor action, calibration setting, model configuration, market or base-rate baseline, final probability, outcome, and score.

What the notebooks show

The Chapter 24 notebooks include weak results rather than only clean demonstrations to show the reality of agent system development (notebooks will be released closer to publication, late June/early July).

In the notebook research_agent, two identical research agents working on the same Fed-rate question returned 0.75 and 0.72. In the notebook multi_agent_research, three agents on a pinned Fed-hike question all returned 0.25, even though their search paths differed slightly. Temperature alone did not create an informative panel.

In the notebook aggregation_math, the diversity factor makes the cost of correlation explicit. Adding agents helps only if they bring different information. If they share the same model, tools, retrieval sources, and priors, the panel can appear broad while behaving like a single repeated forecast.

In adversarial_debate, debate lengthened the trace without narrowing the gap. The notebook shows why debate needs an ablation rather than just a transcript. If the debate does not move beliefs, reveal missing evidence, or improve resolved-outcome scores, it is theater with a token bill.

In evaluation_and_governance, the evaluation panel compares the pipeline with simpler variants using Brier score, log score, expected calibration error, sharpness, ablations, and a market baseline. The panel is small and carries contamination caveats, so it illustrates the scoring workflow rather than a performance claim.

The design rule is simple: keep a component only when it improves the forecast record, the diagnosis, or the evaluation.

What adjacent work adds

The relevant papers add design lessons rather than slogans.

Bridgewater’s AIA Forecaster supplies the forecasting blueprint: agentic search, multiple independent forecasts, supervisor reconciliation, calibration, and comparison with market consensus.

AlphaAgents matters here less for its reported portfolio results than for its role-structured design: fundamental, sentiment, and valuation agents with debate as the reconciliation mechanism.

Recent work on scaling agent systems makes the general point explicit: more agents can help on decomposable tasks and hurt on sequential ones. Coordination overhead, redundancy, and error amplification are design variables.

Where the pattern fits

The workshop uses forecasting because the work is measurable. That does not make forecasting a universal test case for agents.

Sequential execution tasks usually need a workflow, not a panel of agents. A deterministic data-cleaning step, for example, should not become a debate among models. Evidence-gathering tasks may require one strong research agent at first, but several later. Multi-agent design becomes more plausible when the task has competing interpretations, missing evidence, and a decision artifact that can be compared with a baseline.

Forecasting gives that problem a disciplined form. The artifact is a probability. The baseline may be a market price, base rate, consensus forecast, or simpler model. The outcome eventually resolves. The system can be wrong in a measurable way.

That is why this topic belongs in ML4T. It is close enough to trading and research to matter, but constrained enough to evaluate.

What we are building

The companion forecasting-agent codebase turns the AIA pattern into a runnable system, including a CLI, configuration profiles, read-only prediction-market connectors, SQLite persistence, a Streamlit dashboard, deterministic replay mode, local and hosted model profiles, and ablation settings.

The free Lightning Lesson on Thursday, June 18, 2026, walks through this architecture: independent research agents, aggregation, supervisor reconciliation, calibration, and scoring. The practical goal is to inspect, test, ablate, and score the system rather than trust the final paragraph.

The full hands-on workshop, Building Multi-Agent Forecasting Systems, runs Saturday, June 27, 2026, 10am-5pm ET. Participants work with config profiles, replay traces, aggregation settings, supervisor policy, calibration diagnostics, and resolved-outcome scores. The guaranteed path uses deterministic replay, so the exercises do not depend on API keys or live network access. The connectors are read-only.

Use five questions to judge the result:

• Did independent agents produce independent evidence?

• Did aggregation improve on the simple mean?

• Did the supervisor find missing evidence or merely override?

• Did calibration improve reliability without hiding a weak evidence process?

• Did the system improve on a market price, base rate, or simpler model?

Tomorrow’s issue uses the World Cup Forecast Lab as a public forecasting surface. Later issues go deeper into AIA Forecaster, multi-agent diversity, and the ML4T research operator.

Register for the free Lightning Lesson: Build Multi-Agent Systems You Can Audit

Register for the workshop: Building Multi-Agent Forecasting Systems