A language model running on the ESP32-S3
A transformer performs inference inside the microcontroller, without the cloud: why memory is the wall, where speed comes from and how to reproduce it.
Hardware requirements:
- ESP32-S3 com PSRAM: 2 MB para o modelo de 260 mil parâmetros, 8 MB para o de 28,9 milhões
- Flash de 16 MB para o modelo maior
- Cabo USB com transmissão de dados
- ESP-IDF instalado (os projetos não usam Arduino nem PlatformIO)
Download this tutorial's materials:
- Kit de apoio (ZIP)12 KBsdkconfig.defaults comentado, tabela de partições para flash de 16 MB, checklist e o manual em PDF. Não contém o firmware: o port do llama2.c é dos autores citados, e o README leva aos repositórios originais.
- Manual (PDF)12 KBAs seis páginas do artigo em formato de leitura offline, com as tabelas de referência, solução de problemas e créditos.
Want to receive new manuals and industrial schematics? Join our channel on Telegram / WhatsApp
What this article is, and what it is not
Here a real transformer performs inference inside an ESP32-S3: no cloud, no Wi-Fi and no server on the other side. Before any command, here is the warning that prevents frustration: the model that fits on the chip is not an assistant.
The weights used in these projects come from TinyStories, short English stories with a reduced vocabulary, created to study how small a model can be while still producing coherent text. It cannot answer questions about your plant, read a manual or replace the model you would use at the supervisory layer.
What it proves is different, and that matters: the silicon already inside your panel can handle transformer mathematics.
Two published reference points are useful. In 2024, Dave Bennett ran a 260-thousand-parameter Llama on an ESP32-S3FH4R2 and measured 19.13 tokens per second. In 2026, the esp32-ai project reached 28.9 million parameters on the same chip at 9.88 tokens per second, or 94.9 milliseconds per token.
Those figures belong to the authors of the two projects and were not measured here. This article follows their repository procedures, and the number you measure on your own board is the one that matters to you.
The wall is memory, not the processor
The trap is in buying the board, and it catches almost everyone on the first attempt.
PSRAM is not optional. Even the 260-thousand-parameter model needs about 1 MB of RAM, while the ESP32-S3 has 512 KB of internal SRAM. Without PSRAM it does not run, and many generic ESP32-S3 DevKit listings do not say how much they include — or include none.
Bennett used the ESP32-S3FH4R2, with 2 MB of PSRAM in the package. esp32-ai asks for much more: 8 MB of PSRAM and 16 MB of flash because the model occupies 14.9 MB in 4-bit quantization.
The classic ESP32 and ESP32-C3 are outside this path. The C3 has one core, and the vectorized routines that provide the speed here are specific to the S3, as the code section shows.
The hierarchy that fits 28.9 million parameters
The jump from 260 thousand to 28.9 million did not come from a better chip. It came from deciding where each part of the model lives.
There are three levels. Fast, tiny SRAM holds activations and normalization weights, touched many times per token. Intermediate PSRAM holds the model core and output head, read once per position. Large, slow flash holds the table of 25 million parameters.
The trick that makes this viable is called per-layer embeddings, borrowed from the Gemma family. The embedding table is usually the largest single block in a model, and it is precisely the part moved to flash. For each token, only about 450 bytes are fetched from it. Flash is slow, but 450 bytes at a time is manageable.
Record the inversion: the largest piece went to the slowest memory and worked because what matters is not the block size but how many times it is read.
Where the time goes
In a transformer, most of the time is matrix multiplication. The original llama2.c does this with a scalar loop, which is portable and slow.
The ESP32-S3 has a faster path: the ESP-DSP library provides dot-product routines written for the LX7 core's SIMD instructions. The function used is dsps_dotprod_f32_aes3, and the aes3 suffix identifies the S3 variant. It provides the speed, which is why the C3 does not reach the same result.
This is the function's core, from main/llm.c in Dave Bennett's repository:
| 1 | void matmul(v4sf *xout, v4sf *x, v4sf *w, int n, int d) |
| 2 | { |
| 3 | *matmul_params = (MatMulTaskParams){xout, x, w, d / 2, d, n, d, TASK_1_BIT}; |
| 4 | xSemaphoreGive(semaDataReady); |
| 5 | for (int i = 0; i < d / 2; i++) |
| 6 | { |
| 7 | v4sf val = 0.0f; |
| 8 | v4sf *row = &w[i * n]; |
| 9 | dsps_dotprod_f32_aes3(row, x, &val, n); |
| 10 | xout[i] = val; |
| 11 | } |
The two cores in practice
Notice what the function does before it starts counting. It fills matmul_params and releases a semaphore, then enters the loop.
That semaphore wakes a task pinned to the other core. It processes the lower half of the rows while the calling core processes the upper half. The split is literal: d is the number of rows, and each side gets d / 2.
That is the difference between saying that the project uses both cores and showing where it happens. Without the semaphore, the second core would sit idle during the most expensive part of inference.
Measure; do not trust the impression
The feeling of speed is deceptive, especially when text appears word by word on screen. The port already includes the calculation at the end of generation:
| 1 | float tks = (pos - 1) / (double)(end - start) * 1000; |
| 2 | fprintf(stderr, "achieved tok/s: %f", tks); |
Time is measured in milliseconds, hence the multiplication by one thousand. Record your board's number before changing any setting: without a before measurement, there is no way to know whether the next change helped or hurt.
Step by step
The path below follows the original repositories. Both use ESP-IDF, not Arduino or PlatformIO. Know this before starting because the project structure is different.
1. Confirm the board before buying
Look for the PSRAM specification in the listing, not only the chip name. To reproduce the 260-thousand-parameter project, the minimum is 2 MB of PSRAM. For 28.9 million parameters, you need 8 MB of PSRAM and 16 MB of flash.
If the seller does not specify PSRAM, assume it has none. This is the most common reason the project does not boot.
2. Install ESP-IDF
Follow Espressif's official guide for your operating system and confirm the environment is active before continuing. Everything else depends on that working.
3. Build
With the repository cloned and the ESP-IDF environment loaded, build with the tool's standard command:
| 1 | idf.py build |
The first build takes time because it downloads and builds dependencies, including ESP-DSP.
4. Flash and monitor
Replace the port with the one for your board. On Linux it is usually something like ttyUSB0 or ttyACM0; on Windows it is a COM port:
| 1 | idf.py -p /dev/ttyUSB0 flash monitor |
You should see the monitor open, the board restart and text begin to appear token by token in English, in the style of a children's story. At the end of generation, a line reports tokens per second.
If text appears but is much slower than expected, the issue is usually configuration rather than code. Check that the CPU is at 240 MHz, PSRAM at 80 MHz and the instruction cache has been enlarged — the three settings the author identifies as responsible for the gain.
5. When it does not fit
An allocation failure during boot or a memory error while loading the model almost always points to the same place: PSRAM disabled in menuconfig or a board with no PSRAM.
Also check partition size. For the 14.9 MB model, a default partition table is not enough, and the firmware needs room for the embedding table in flash.
Where this meets automation
Let us be clear about the current state of the art: this is a demonstration, not a product. A TinyStories model does not belong in an electrical panel.
What it anticipates, however, is concrete. The same ESP-DSP and ESP-NN routines that accelerate this transformer support useful edge inference: motor-vibration anomaly detection, sound classification and simple vision counting. Understanding vectorized dot products and core splitting here puts you one step away from applying them there.
There is also an architectural lesson independent of any model: putting the largest block in the slowest memory can be the right decision as long as it is read only a few times. That applies to embeddings in flash and to process history on a card in your datalogger.