Pipette is an open-source benchmarking platform that evaluates AI models together with their quantisation, runtime, hardware and context length to provide reproducible on-device performance results.
Liquid AI has released Pipette, an open-source platform designed to benchmark foundation models on edge devices. Developed in partnership with Artificial Analysis, which independently validated its methodology, Pipette treats on-device performance as a property of the complete deployed configuration rather than the model alone. Its benchmark unit combines the model, quantisation, runtime and device.
The initial Pipette dataset covers five on-device performance metrics across more than 1,000 model, quantisation, runtime, device and context configurations. It includes more than 30 models, multiple quantisation formats and llama.cpp builds for macOS, iOS, Windows and Android. The tested context lengths range from 256 to 8,192 tokens, with initial verified results from a MacBook Pro with M5 Max, iPhone 17 Pro and Galaxy S26 Ultra.
Pipette is intended to make on-device model comparisons more reproducible by measuring how different deployment configurations behave under controlled conditions. The methodology uses fixed token shapes, greedy decoding, a discarded warm-up and five measured repetitions, while platform-specific readiness checks are used to identify unsuitable thermal or system-load conditions before measurements are published.
The platform also separates model quality evaluation from device performance. Quality measurements use benchmarks including IFBench, GPQA Diamond and MATH-500, with the current quality evaluations performed through llama.cpp on NVIDIA H100 80GB reference systems. These results are then associated with on-device runs using the same model and quantisation, allowing quality and deployment performance to be considered together without claiming that the quality evaluation itself was performed on the target device.
Liquid AI has released Pipette’s infrastructure under the Apache 2.0 licence, including pipette-mgmt, pipette-clients and pipette-scores, along with a public results dataset, hosted dashboard and native iOS and Android benchmarking applications. The open-source approach allows developers, hardware vendors and enterprises to reproduce measurements, compare deployment configurations and evaluate models before selecting hardware, quantisation or runtime for on-device AI applications.
