Atomicwork and New Measure have open-sourced ITSMBench, giving developers and enterprises a reproducible way to compare AI models and agent frameworks on real-world IT service-management tasks.
Atomicwork and New Measure have launched ITSMBench, an open-source AI benchmark for evaluating how effectively frontier models and agent frameworks perform real-world IT service-management (ITSM) tasks.
Every environment, task, and evaluation is open source, allowing researchers and enterprises to reproduce results, inspect the benchmark, and build on it. Its methodology, results, and open-source environments are also publicly available.
ITSMBench recreates enterprise service desks using 42 mocked software systems, nearly 1,800 database tables, and more than 2,000 REST endpoints. It covers 89 L2 and L3 service-desk tasks across identity and access management, devices, networking, security operations, infrastructure, and engineering escalations.
Initial results highlight differences between models and agent harnesses. Grok 4.5 leads tool discovery at 83.0%, but ranks near the bottom for task completion. Opus 5 leads execution at 63.5% once the right tools are available, despite the weakest discovery rate at 70.2%. GPT-5.6 Sol falls between the two, while GLM-5.2 costs $0.25 per trial but trails Opus 5 and GPT-5.6 Sol on execution.
Across the benchmark, 73 of 89 tasks were completed at least once, but no single model solved the full set. Agents also frequently stop after finding plausible explanations rather than verifying root causes, and can leave related systems, records, or follow-up actions incomplete.
“Frontier models are brilliant at writing code, but they are completely blind to the hidden security landmines inside enterprise workflows. ITSMBench proves that blindly trusting an AI agent right now is an invitation for an enterprise security breach,” said Vijay Rayapati, co-founder and CEO of Atomicwork.
“Enterprise service management demands more than reasoning. One model fits all is seldom the right answer. Enterprises that want an AI-run service desk need to optimise for accuracy, cost, and time to resolution at the same time,” said Arushi Gandhi, CEO at New Measure.
