Hyper-τ-Bench is an open-source benchmark that evaluates whether AI coding agents can research, design and build complete customer-service agents rather than simply operate them.
Sierra has open-sourced Hyper-τ-Bench, a long-horizon evaluation designed to measure whether AI systems can construct working agents. The benchmark extends Sierra’s earlier τ-bench, which evaluated whether models could reliably operate customer-service agents. Hyper-τ-Bench instead focuses on the more complex task of building the agent itself.
The benchmark places a developer agent inside a sandbox containing records from a simulated business, a codebase, a production API and other information needed to understand the requirements. The agent can also communicate with a simulated client to recover missing requirements. It must then investigate the available information, design an architecture, turn business operations into tools and produce a functioning customer-service agent.
Hyper-τ-Bench evaluates the resulting agent rather than simply checking whether the developer agent generated working code. The completed system is deployed against previously unseen simulated production traffic using verifiable τ-bench-style tests. The benchmark also places constraints on available models and serving costs, requiring the developer agent to make architectural and resource decisions while constructing the system.
The initial evaluation covers 53 tasks across four domains. Sierra reports that the strongest automated configuration, using Claude Opus 5 with Claude Code, passed 23.9 per cent of the evaluation simulations. An expert-authored reference system achieved 82.2 per cent, highlighting the remaining gap between automated agent construction and systems built with expert involvement.
By open-sourcing Hyper-τ-Bench, Sierra provides developers and researchers with a benchmark for studying a different stage of AI capability: not simply whether an AI agent can complete a task, but whether an AI coding system can build the agent that completes it. The benchmark is positioned alongside other evaluations of long-horizon research and engineering capabilities, while focusing specifically on the challenges involved in constructing production-style AI agents.
