In March, SWE-Skills-Bench tested whether skills improve the quality of agent work on software repositories. The authors evaluated 49 skills across 565 task instances. The average pass rate rose from 89.8% without skills to 91.0% with them. 39 skills produced no improvement, three made results worse, and seven improved them.
The largest gain came from risk-metrics-calculation. The skill covers drawdown, value at risk, and other standard risk measures. It took ten tasks from 70.0% to 100.0% and reduced token use by 34.8%.
The ML4T skills are now public
We built the ML4T skill collection around procedures that can fail silently, producing wrong results rather than throwing errors. Today, we are making all 61 skills public on GitHub under the Apache 2.0 license. They cover the end-to-end research process, from point-in-time data and feature engineering to validation, backtesting, portfolio construction, production, and agent engineering.
Join us for a free 30-minute intro on Sep 2, “Managing your Strategy Research Process”.
I am also teaching on September 9, How to Engineer a Multi-Agent System, covering task decomposition, deterministic harnesses, and the decisions that still require human approval.
The ten-task result for the risk-metrics-calculation skill does not establish a general effect. It supports a useful hypothesis: skills help when an agent does not reliably know the required procedure. Quantitative research adds a second possibility. A skill may help more when the environment does not expose a procedural mistake. A wrong risk calculation still returns a number.
What an agent skill contains
A skill is a Markdown file with a name, a routing description, and a procedure. The agent reads the description first and loads the full procedure when it judges the skill relevant. Anthropic calls the approach progressive disclosure. The company introduced Agent Skills in October 2025 and published the format as an open standard in December.
Reusable procedures predate the current file format by decades. Korf’s 1985 macro-operators gave search algorithms reusable sequences of moves. Sutton, Precup, and Singh’s 1999 options framework did the same for extended actions in reinforcement learning. Researchers can now inspect, share, and test a procedure in a standard, readable file.
Why two benchmarks report different gains
SkillsBench 1.1 contains 87 tasks across eight domains. Benchmark authors wrote a relevant skill for each task and used deterministic checks to score the result. Across 18 model/harness combinations, the average pass rate rose from 33.9% to 50.5%, a gain of 16.6 points.
SWE-Skills-Bench tests a different setting. It evaluates public skills in repositories that their authors did not target. The benchmark generates each requirement from the skill, then selects a repository with the relevant technology. A fixed prompt generates the pytest verifier. The resulting test runs deterministically, although a model wrote it.
SkillsBench asks how much a task-specific skill can help. SWE-Skills-Bench asks how well a public skill transfers to a new repository. Skill authorship, task mix, model, harness, baseline, and scoring method all differ.
The starting pass rate also matters. SWE-Skills-Bench began at 89.8%, leaving 10.2 percentage points available to improve. Its 1.2-point gain closes about 12% of that gap. SkillsBench began at 33.9% and closes about 25% of the available gap. The apparent fourteen-fold difference falls to roughly twofold after this adjustment. The studies remain too different for a direct comparison.
Results also vary within the same benchmark. In SkillsBench 1.1, average gains range from 9.7 points in Mathematics and OR to 28.8 in Natural Science. Thirteen of the 87 tasks get worse with a skill. In SWE-Skills-Bench, broad or version-specific guidance sometimes distracts the agent from the immediate task.
NVIDIA reports larger gains on a third type of evaluation. Its SkillEvaluator benchmark covers more than 300 skills for over 30 NVIDIA products. Average correctness rose from 46 to 87 on a 0-100 rubric. NVIDIA writes the skills, owns the product documentation, and builds the evaluation. The post reports no confidence intervals, and most skills use one attempt per task.
Quantitative research has silent failures
Software agents receive feedback while they work. A compiler rejects a type error, a test fails, or a service does not start. The agent can use the signal to revise its work.
Many research errors produce valid code and plausible output:
Cross-validation leakage. Training labels or feature windows overlap the test period. The backtest runs and the estimated Sharpe ratio can rise.
Pre-split standardization. A scaler fitted on the full dataset puts future information into the training data. No runtime error identifies the leak.
Current index membership. Applying today’s constituents to earlier dates excludes many firms that later left the index. The resulting backtest contains survivorship and constituent-selection bias.
The wrong cost base. Charging costs on the position instead of the trade overstates costs in a 4% to 5% rebalance. Other denominator errors can understate them. The tearsheet still renders.
A researcher has to state the relevant invariant and test whether it held.
Skills, libraries, and verifiers serve different roles
A skill gives the agent a procedure before it starts. A library provides a tested implementation while the agent works. A verifier checks the process or result. Repeated failures may require all three.
A skill instructs before execution, a library constrains implementation, and a verifier detects violations.
For example, a skill can tell the agent to fit a scaler inside each training fold. A library can expose an API that accepts fold-local data. A verifier can record which rows reached fit and reject any call that included test data.
How we designed the ML4T skills
We applied the benchmark findings directly:
Narrow scope. Each skill covers one research procedure and stays within 120 lines. Short files reduce irrelevant context.
Explicit routing. Every description states when the agent should load the skill, so the agent can select it before loading the full procedure.
Concrete failure examples. Every skill shows a plausible wrong approach and a corrected one. The examples focus on stable research invariants rather than fixed API versions or parameter values.
Library integration. Fifty skills include a Production Implementation section that shows the relevant ML4T library. CI can check the imports and function signatures against the library source.
Instruction-only skill directories. Each directory contains one
SKILL.md, with no executable hooks or companion scripts.
The collection includes procedures for leakage, lookahead bias, point-in-time data, non-stationarity, feature validation, backtest costs, portfolio risk, and live-trading controls. It also includes end-to-end research workflows and skills for building reliable research agents.
How we are evaluating the collection
Our March pilot tested five paired tasks, with one run per condition. All ten runs passed. Runs with skills used 1,253,830 tokens, compared with 1,212,535 without skills. The pilot established a cost baseline and showed that the next benchmark needs harder tasks and stronger verifiers.
The next evaluation starts from silent research failures. Each task will require the relevant procedure. Deterministic checks will inspect data splits, training inputs, or point-in-time records where possible. Research decisions that resist a deterministic rule will need a judge calibrated against human labels.
We will run repeated skill and no-skill trials on the same tasks. The report will show where each skill improves correctness, how consistently it does so, and what it costs in tokens. The results will also show when a library constraint or verifier is more effective than written guidance.
Get the skills
Inspect any public skill before installing it. One marketplace study analyzed 31,132 skills and estimated that 26.1% contained at least one vulnerability. Executable scripts increased the observed risk, but harmful instructions can also live in Markdown.
The ML4T skills are public under Apache 2.0:
git clone https://github.com/ml4t/skills.git ~/.ml4t-skills
~/.ml4t-skills/scripts/install.shInstall the collection, inspect the files, and use the skills that fit your research workflow. We will publish the evaluation results as we measure each skill’s effect on correctness, consistency, and cost.
Join us for a free 30-minute session on Sep 2, “Managing your Strategy Research Process”.
I am also teaching on September 9, How to Engineer a Multi-Agent System, covering task decomposition, deterministic harnesses, and the decisions that still require human approval.
Good reminder that the verifier matters just as much as the skill.