DevRev has open-sourced Enterprise-Bench, a vendor-neutral framework that lets researchers and AI vendors independently reproduce, verify and compare enterprise AI agent performance using shared datasets, methods and public results.
DevRev has released Enterprise-Bench, an open, vendor-neutral benchmark designed to evaluate whether AI agents can operate reliably in real-world enterprise environments, making its complete framework publicly available to promote transparent and reproducible AI evaluation.
The release includes the benchmark dataset, evaluation methodology, benchmark queries, judging criteria, Harbor evaluation harness, execution traces and initial results. AI vendors, enterprise customers and researchers can independently reproduce the benchmark and submit results to a public leaderboard.
Unlike conventional AI benchmarks that focus primarily on task complexity, Enterprise-Bench evaluates organisational complexity by testing AI systems against fragmented enterprise data, siloed systems and permission boundaries that reflect production environments. The Harbor evaluation harness, developed by Laude Institute, executed and verified the benchmark, while Professor Alexandros Dimakis of UC Berkeley validated the framework.
Enterprise-Bench currently covers L1-L2 autonomy, including factual retrieval and complex multi-source enterprise queries, with L3 and L4 benchmarks planned later this year. Evaluations measure precision, efficiency and safety using an independent LLM judge, published criteria and mandatory execution traces.
DevRev said the benchmark addresses a gap in enterprise AI evaluation as organisations continue to struggle with large-scale AI deployment. According to McKinsey’s 2025 State of AI survey, nearly two-thirds of organisations have yet to scale AI across the enterprise.
In the initial benchmark, DevRev’s Computer achieved 94.3% accuracy versus 63.6% for Claude Code on identical L1-L2 tasks using the same underlying Opus 4.8 model family, while using 4.4 times fewer tokens per correct response. DevRev attributed the performance difference to its enterprise data retrieval and system architecture rather than the underlying large language model.
Ahmed Bashir, CTO at DevRev, said the benchmark measures “organizational complexity, not just task complexity,” reflecting conditions AI systems face in production.
