Snorkel AI backed its $3M open-source initiative by revealing the first cohort of grant-funded evaluation tools, including OSWorld 2.0, Frontier-Bench, and SlopCode Bench, designed to measure complex, multi-step AI agents.
On 24 July 2026, Snorkel AI highlighted the first cohort of projects supported through its Open Benchmarks Grants initiative, a $3 million grant program launched in February 2026 to fund open-source datasets, evaluation frameworks, and benchmarks for agentic AI, established with support from Hugging Face, Prime Intellect, Together AI, Factory, Harbor, and PyTorch.
The commitment consists of estimated value in Expert Data-as-a-Service (DaaS) access, compute credits, and dedicated Snorkel engineering support, rather than cash transfers. Grant recipients retain intellectual property rights to their work, but must publish outputs under permissive licenses, typically MIT or Apache 2.0 for code/tools, and CC BY 4.0 or CC0 for benchmark datasets.
Every funded project moves away from single-turn LLM tests toward multi-step agentic workflows evaluated across long time horizons. These include Frontier-Bench (an adversarial successor to Terminal-Bench 2.1 created with the Laude Institute and Harbor community), UC Berkeley RDI’s Agents’ Last Exam (evaluating workflows across 55 sub-industries via expert-validated tasks toward a 5,000-task target), and HKU XLANG Lab’s OSWorld 2.0, evaluating computer-use agents across 108 workflows in 31 self-hosted web and desktop environments.
Complementing these are specialised diagnostic tools: UC Berkeley SkyLab and UW–Madison’s Continual Learning Bench (measuring state retention across sequential tasks), UW–Madison’s SlopCode Bench (tracking code quality degradation over iterative edits), and Terminal-Bench Science (extending CLI agent testing into computational research). Beyond grant recipients, Snorkel AI highlighted Senior SWE-Bench (developed with Princeton and UW–Madison for senior-level engineering work). Grant selection is ongoing—applications remain open for researchers.
