Ivy AI brings speech recognition, a small language model and speech synthesis to an ESP32-S3, with firmware and development tools available under GPL-3.0.
Ivy AI, an open-source voice assistant developed by Anankush Pandit, runs speech recognition, language-model inference and speech synthesis locally on an ESP32-S3 microcontroller. The project publishes its firmware, model-processing code, training tools, host-side tests and technical documentation on GitHub under the GNU General Public Licence v3.0 (GPL-3.0-or-later). By keeping the main voice-processing pipeline on the device, it demonstrates how developers can build offline voice interfaces without depending on cloud AI services.
The implementation targets the Freenove FNK0104B development board, which combines an ESP32-S3 with 8MB of PSRAM, 16MB of flash, a touchscreen, microphone, speaker, audio codec and microSD storage. After detecting the wake phrase “Hey Ivy”, the firmware transcribes speech, processes the request using a small language model and generates a spoken response. Wi-Fi is used only for periodic time synchronisation, while the voice-assistant pipeline operates without a network connection.
The project includes a 19.7-million-parameter language model quantised to four bits, a conformer-based speech-to-text implementation and SVOX Pico text-to-speech software. Developers can also load compatible GGUF language models from the microSD card. The repository reports generation speeds of around 5.4–5.9 tokens per second with its TinyTalk model, while a smaller GGUF model reaches approximately 7.3 tokens per second. Speech recognition takes several seconds, making the system more suitable for simple conversational tasks than real-time transcription.
A key part of the open-source release is its development infrastructure. The repository contains C firmware built with ESP-IDF, model-conversion and SD-card transfer utilities, serial-console tools, host-side test harnesses and documentation covering memory allocation, model compatibility and measured performance. Developers can test parts of the inference pipeline on a computer before flashing the firmware to the microcontroller. The project also documents how it shares limited memory between speech recognition and language generation, which cannot run simultaneously within the available memory budget.
The release also identifies third-party components and their licensing terms, including Apache-2.0-licensed speech-recognition code, MIT-licensed language-model components and separately licensed model weights. Some model weights carry non-commercial dataset restrictions, so the GPL licence covering Ivy AI’s own code does not automatically make every bundled asset suitable for commercial use. By publishing the implementation and its performance limitations, Ivy AI provides embedded developers with a practical foundation for experimenting with offline speech interfaces, compact language models and memory-constrained AI on low-cost microcontrollers.
For more information, click here.
