How to Run a 28.9M Parameter LLM on an $8 ESP32-S3

The Triumph of Minimalist Embedded Engineering

The modern software stack is a monument to bloating. We have normalized running closed-source blobs that execute arbitrary code with root privileges, relying on massive, power-hungry cloud servers just to perform basic text processing. This dependency is not a technical necessity. It is a business model designed to lock developers into proprietary ecosystems. The real work of software engineering is not about scaling up virtual machines. It is about writing clean, readable code that respects the physical limits of the silicon.

The slvDev/esp32-ai repository stands as a direct challenge to this cloud-centric orthodoxy. This open-source project successfully runs a 28.9-million parameter language model entirely on an ESP32-S3, a microcontroller that costs roughly eight dollars. The model runs fully offline, requiring no network socket, no API keys, and no telemetry. It generates text at 9.5 tokens per second, outputting directly to a SPI-wired screen.

Before this implementation, running a language model on a microcontroller was a mere toy demonstration. Previous attempts to port small architectures, such as Karpathy's llama2.c, topped out at around 260,000 parameters on similar hardware. Squeezing a model nearly one hundred times larger onto the same silicon requires discarding modern abstraction layers. It demands a deep understanding of the compiler, the memory bus, and the underlying hardware registers.

Architectural Breakdown: Squeezing 28.9 Million Parameters into 512KB SRAM

The ESP32-S3 is a modest chip. It features a dual-core Xtensa LX7 processor, 512 KB of internal SRAM, and support for external PSRAM. A standard 28.9-million parameter model stored in 32-bit floating-point format requires over 115 megabytes of memory. This is an impossible target for a chip with less than a megabyte of fast on-chip RAM. To bypass this barrier, we must look at where the parameters actually live.

The core architectural trick is Per-Layer Embeddings, a technique borrowed from Google's Gemma model design. In a typical transformer, the embedding table is a massive weight matrix that must be kept in working memory. But this table is primarily a lookup mechanism. The developer realized that 25 million of the model's 28.9 million parameters do not undergo heavy matrix multiplication. They are simply read.

By storing this massive lookup table on the 16 MB external flash memory, the runtime only pulls about 450 bytes into SRAM per token. The active compute core is compressed using 4-bit quantization, shrinking the total footprint to 14.9 MB. The external PSRAM serves as a volatile scratch space for activation tensors. This design is detailed in the initial Tom's Hardware report on the breakthrough.

Memory Hierarchy and the Flash Bottleneck

Reading from external flash over an SPI bus is slow. In traditional systems design, this latency would ruin performance. The firmware bypasses this by executing highly optimized, sequential reads that align with the cache lines of the ESP32-S3. By treating the flash memory as a read-only block device, the kernel avoids the overhead of a standard filesystem.

The remaining compute layers must execute within the tight constraints of the PSRAM. The developer utilized the Xtensa instruction set to accelerate the 4-bit quantized matrix multiplications. This hardware-level optimization ensures that the processor does not sit idle waiting for memory transfers.

This is the essence of open-source software minimalism. Instead of throwing more hardware at the problem, we optimize the software to match the physical realities of the silicon. It is a direct rejection of the bloated web frameworks that dominate modern software development.

Reclaiming Ownership of the Silicon

If you cannot compile your code from source and run it on your own hardware, you do not own it. The tech industry wants us to believe that AI is too complex for local execution. They want to lock developers into proprietary cloud ecosystems where every token costs money. This project proves that a tiny, battery-powered node can generate coherent text without a network connection.

The training pipeline is equally elegant. The model is trained on the TinyStories dataset, a curated corpus designed to teach language models basic grammar and reasoning without requiring billions of parameters. The resulting model is highly specialized, proving that small, targeted datasets are far more efficient than brute-force web scraping.

This approach paves the way for truly decentralized systems. We can now build smart sensors, offline translators, and interactive toys that do not rely on a central server. The software is licensed under the permissive MIT license, ensuring that anyone can inspect, modify, and compile the code.

"If you cannot compile it from source, you do not own it. We must return to clean, readable code that empowers the developer, not locks them into a vendor ecosystem."

# Conceptual Python script for exporting Per-Layer Embeddings to binary format
import torch

def export_embeddings(model, filepath):
    # Extract the embedding table
    embeddings = model.get_input_embeddings().weight.data
    # Quantize to 4-bit or 8-bit depending on target
    quantized_embeddings = quantize_to_int4(embeddings)
    # Write raw bytes to binary file for direct flash flashing
    with open(filepath, 'wb') as f:
        f.write(quantized_embeddings.numpy().tobytes())
    print(f"Successfully exported embeddings to {filepath}")