Running a 28.9M Parameter LLM on an $8 Microcontroller

An astonishing open-source project proves it is possible to run a 28.9 million parameter language model on an ESP32-S3 microcontroller that costs just . The model generates text entirely on-device at roughly 9.5 tokens per second, writing each word to a small screen without any server connectivity. This represents a hundredfold increase over the previous record for microcontroller-based language models, which topped out at 260K parameters.

The breakthrough uses Google's Per-Layer Embeddings technique from Gemma models. Instead of fitting the entire model in the chip's limited 512KB of SRAM, most parameters live in slow flash memory and only the few thousand bytes needed per token are read. The model stores 25 million parameters in a flash lookup table while the active computation core stays in fast memory, making the large model cost almost nothing to run.

The model was trained on TinyStories and produces short coherent narratives at the edge. While the 28.9M parameter model cannot answer questions or write code, the architectural innovation opens a path to truly private, offline AI on ultra-low-cost hardware. For founders building edge AI products, this demonstrates a viable route to running useful language models on inexpensive embedded devices.