Supabase Evals, an Apache-2.0 benchmark, is designed to evaluate AI coding agents on real-world engineering tasks inside containerised sandboxes.
Supabase has released Evals (supabase/evals) under the Apache-2.0 licence to evaluate AI agents, such as Claude Code, Codex, and OpenCode, against authentic engineering tasks derived from real support tickets and GitHub issues. Scenarios execute inside containerised Docker environments with actual CLI and Model Context Protocol (MCP) interfaces, combining deterministic execution checks with LLM-as-a-judge scoring.
Evaluation spans three core dimensions: Products such as Database, Auth, Storage, Edge Functions, Realtime, Cron, Queues, Vectors, Data API; Topics such as RLS, security, migrations, SQL, SDK, observability, self-hosting, declarative schemas; and Stages like build, deploy, investigate, resolve. The framework employs a dual-suite architecture comprising a Published Benchmark Suite powering public leaderboards and an Internal Regression Suite for daily failure-mode tracking.
Key findings show top-tier models, namely Opus 5, Kimi K3, achieved 100% pass rates unaided in the Build stage. Loading Supabase skills closed performance gaps for smaller models like Sonnet 5 rose from 78% to 100%, though documentation retrieval varied sharply: OpenAI Codex checked ~8 pages per scenario versus Claude Code’s ~2 pages. Agents also frequently defaulted to manual SQL migrations over declarative schemas and manual authentication checks over utility libraries like @supabase/server.
