A lightweight implementation uses a memory management technique borrowed from Google’s Gemma models to run a significantly larger language model on an ESP32-S3 microcontroller, enabling fully offline AI inference on low-cost hardware.
A developer has demonstrated a 28.9-million-parameter language model running entirely on an ESP32-S3 microcontroller, marking a significant increase in model size for this class of embedded hardware. The project achieves approximately 9.5 tokens per second while operating fully offline, showing that large language models can be deployed on inexpensive microcontrollers without cloud connectivity.
The approach employs Per-Layer Embeddings, which is a technique for managing memory allocation, first introduced by Google in the models called Gemma. Rather than loading the entire model into fast memory, 25 million parameters are stored in the ESP32-S3’s 16 MB flash memory, while the model core is kept in the device’s 512 KB SRAM. Since only the fragment of the embedding table is used at a time during the generation of tokens, the use of the slower flash memory does not significantly affect inference speed.
To reduce memory consumption, the model will be quantised down to the 4-bit format, so that the size of the model will be about 14.9 MB. The output processing and working memory will be allocated in the board’s 8 MB PSRAM.
The current model is trained using the TinyStories dataset and is able to create short fictional stories but cannot answer queries or perform complex reasoning. The existing project shows an efficient way of deploying larger neural nets on resource-constrained embedded devices. The developer has also made the code open-source so that other developers may analyse and improve the existing method.
The use of hierarchical memory and per-layer embeddings in the project proves that through architectural optimisation, it is possible to increase the potential of edge AI hardware.
