Can an ESP32 use public AI APIs? Yes—and where each piece works
A 28.9M-parameter language model generates text inside an ESP32-S3 at 9.5 tokens per second. The correction, and where each piece works.
Hardware requirements:
- Placa ESP32 ou ESP32-S3 com Wi-Fi
- Rede Wi-Fi com saída para internet
- Opcional: PC ou mini-servidor na mesma rede, para a parte do modelo local
Download this tutorial's materials:
- Projeto completo (ZIP)343 KBFirmware que consulta um modelo pelos dois caminhos: HTTP simples para um gateway na sua rede, ou HTTPS para a nuvem. Filtro de JSON, consulta não bloqueante, README e manual em PDF.
- Manual em PDF (7 páginas)417 KBComparativo entre nuvem e rede local, o custo de RAM do handshake TLS, filtro de JSON e checklist de dez itens.
Want to receive new manuals and industrial schematics? Join our channel on Telegram / WhatsApp
The question everyone asks
“Can you put ChatGPT inside an ESP32?”
The short answer is that an ESP32 can talk to these models, and that is very different from running one inside the chip. Understanding the difference separates a project that works on the bench from one that works in the field — and that is what this article is about.
What one of these AIs actually is
When people mention Gemini, ChatGPT or Claude, they mean a neural network. Simplified to what matters here: the model processed an enormous amount of text and stored what it learned as billions of numbers called weights.
Answering a question means doing calculations with all those numbers. It is not looking up a ready-made answer in a database — it is calculation, every time.
Keep that idea in mind because everything that follows comes from it.
Why large models do not run inside the board
The natural question is: if it is only a lot of numbers, why not write them to flash and run them there?
It depends on model size. For large models, the limit is not storage — it is compute and memory capacity. An ESP32 has a few hundred kilobytes of internal RAM. Gemini, ChatGPT and Claude need tens of gigabytes of video memory and dedicated accelerators just to answer in reasonable time. Those live in the data center, and no clever architecture changes that.
What changed since the first version of this text is where the boundary of “small” sits — and it is much farther out than expected. The correction appears at the end of the article.
This is not a defect in ESP32. These are tools for different jobs: a server does not stay powered for five months in a 45 °C panel while consuming less than one watt, and an ESP32 does not perform data-center-scale linear algebra. Each has its place.
How the ESP32 talks to a model: the API
An API is how one program talks to another. For automation engineers, the closest analogy is this: the ESP32 sends a value to a server and receives a response, like querying a remote device. The difference is that this “device” is a model on the internet, charges per request and depends on the link being up.
In practice, the board connects to Wi-Fi, makes an HTTPS request to the provider's endpoint with an access key and reads the returned text.
| 1 | #include <WiFi.h> |
| 2 | #include <HTTPClient.h> |
| 3 | const char* WIFI_SSID = "YOUR_NETWORK"; |
| 4 | const char* WIFI_PASS = "YOUR_PASSWORD"; |
| 5 | // The key is stored in the firmware. Remember this detail: |
| 6 | // it returns later in the article. |
| 7 | const char* API_URL = "https://api.example.com/v1/chat"; |
| 8 | const char* API_KEY = "your-key-here"; |
| 9 | String askModel(const String& question) { |
| 10 | HTTPClient http; |
| 11 | http.begin(API_URL); |
| 12 | http.addHeader("Content-Type", "application/json"); |
| 13 | http.addHeader("Authorization", String("Bearer ") + API_KEY); |
| 14 | String body = "{\"prompt\":\"" + question + "\"}"; |
| 15 | int code = http.POST(body); |
| 16 | String response = (code == 200) ? http.getString() : String(""); |
| 17 | http.end(); |
| 18 | return response; |
| 19 | } |
| 20 | void setup() { |
| 21 | Serial.begin(115200); |
| 22 | WiFi.begin(WIFI_SSID, WIFI_PASS); |
| 23 | // Without a timeout the board stays here forever when Wi-Fi does not |
| 24 | // come up — and this article is about depending on the link. |
| 25 | unsigned long start = millis(); |
| 26 | while (WiFi.status() != WL_CONNECTED && millis() - start < 20000) { |
| 27 | delay(500); |
| 28 | Serial.print("."); |
| 29 | } |
| 30 | Serial.println(); |
| 31 | if (WiFi.status() != WL_CONNECTED) { |
| 32 | Serial.println("No Wi-Fi: nothing to do without the link."); |
| 33 | return; |
| 34 | } |
| 35 | Serial.println(askModel("Explain MQTT in one sentence.")); |
| 36 | } |
| 37 | void loop() { |
| 38 | // Minimal example: one query at boot. |
| 39 | } |
And it works. The response appears in the serial monitor, and it is a good moment: you have just connected a low-cost microcontroller to a model running in a data center.
This is where most tutorials end. Let us go one step further.
What changes when the project leaves the bench
Nothing that follows is a board defect. These are characteristics of the chosen architecture — asking a remote service to make the decision. Each one is a criterion for deciding whether the architecture fits your case.
Link dependency. The decision happens only if the internet is up. Fine for a desk project. Not fine for equipment that must keep operating when the carrier goes down — that decision must be local.
Response time. Adding the trip, processing and return, the response takes hundreds of milliseconds to a few seconds. If the process tolerates that, great. If the machine cycle is shorter than the response, the decision arrives late.
Cost per request. It is irrelevant in a test. The number that matters is the steady state: multiply by device, cycle, shift and month. Do this calculation early, while designing.
The key in firmware. The API key is stored on the board, and firmware can be extracted from a device with physical access. On a bench that may not matter; in installed equipment it does. There is no truly secure way to store a secret in a microcontroller.
Where the data goes. Every request sends information outside your network. Irrelevant in a personal project; in a corporate environment, it is usually the first question asked — and not always a technical discussion.
Notice the pattern: none of these points is about the ESP32. They are all about where the computation happens.
The middle path: a model on your own network
Between “use the internet one” and “use nothing” there is a third path: run a smaller model inside your own network. Tools such as Ollama download the model to one of your machines and run it locally.
An important point because it is the most common confusion: Ollama does not run on the ESP32. It runs on a PC, mini-server or gateway. The ESP32 keeps doing what it does well — collecting, measuring and publishing.
Returning to the opening idea: it is not that nothing fits outside the data center. The larger models do not. A smaller model chosen for a specific task runs on an ordinary machine — without internet, a subscription or data leaving its location.
This change solves four of the previous problems: the external link no longer matters, response time drops, per-request cost disappears and data stays on your network. Four of the five points in the previous section, achieved by an architectural decision. What remains is gateway hardware — a one-time investment that belongs to you.
Where ESP32 shines in this architecture
Its role does not change, and it is precisely the role no server performs well:
1. Read a sensor continuously with reliable timing. 2. React immediately to critical events that cannot wait for a network response. 3. Publish data — MQTT, HTTP or Modbus — for whoever analyzes it. 4. Stay up for months with low power consumption in a hostile environment.
Heavy intelligence lives in the gateway on the same network, a few milliseconds away. The ESP32 delivers the data and receives the conclusion without depending on the outside world.
There is also a fourth path: run a model inside the board itself. For a long time that meant only a TensorFlow Lite Micro classifier trained for one task and unable to generate text. It remains an excellent and inexpensive option for immediate decisions. But the ceiling has moved — there is now a language model running on ESP32-S3, with a published walkthrough — and the correction below explains it.
How to choose in practice
Prototype, proof of concept or personal project: public API. Fast to build, inexpensive at test volume and educational.
Field equipment with availability or privacy requirements: local model on the gateway, ESP32 collecting.
A decision that must be immediate and always asks the same question: a small model embedded directly on the board.
In all three cases ESP32 does the same thing: it is the endpoint that sees the real world. What changes is where the computation happens — and now you can choose consciously instead of discovering the issue after installation.
Correction: a language model running inside the board
When this article was published, the prevailing view — including ours — was that a “language model on a microcontroller” did not exist: a classifier fit inside the board, while text generation belonged to a server. That was disproved in practice with hardware on the bench.
A 28.9-million-parameter stored model generates coherent text inside an ESP32-S3 N16R8, without internet, gateway or PC. End to end it reaches about 9.5 tokens per second (102.9 ms per model step).
The trick is not brute force but a memory hierarchy. Parameters are divided according to the speed of each board memory:
- a 559 KB dense core (the part run at every step) within the internal SRAM budget;
- an output head staged as int8 in PSRAM;
- a table of 25 million parameters mapped directly in flash, queried per token.
The silicon measurement is the central point: a random 512 B line in flash costs 20.3 µs, and six lines per token total about 0.12 ms, around 0.7% of memory time per token. The giant table living in flash is therefore almost free. The output head dominates at about 17.3 ms per token, a cost any model pays.
What this means — and what it does not. It is not ChatGPT inside ESP32 and never becomes that. It is a restricted-domain TinyStories model with good coherence and limited capability; “28.9M parameters” counts stored parameters enabled by memory partitioning, not a multiplier of capability. The paragraph about frontier models remains fully valid.
What changes is the map. The boundary of what runs inside the board is no longer “a classifier for one task only”. It now includes closed-domain, offline language generation with tens of milliseconds per token — no link, no per-request cost, no API key in firmware and no data leaving the network. The five concerns in “what changes when the project leaves the bench” disappear in this architecture.
The complete walkthrough is A language model running on ESP32-S3: the PSRAM trap that catches buyers, the SRAM/PSRAM/flash hierarchy and the real vectorized ESP-DSP dot product running on both cores.
So the opening question has aged. It is no longer “can you put an AI inside an ESP32?” It is “which model is worth embedding, and how much of the decision can stop depending on the internet?”.
Continue the conversation
If you work with ESP32 in industrial environments or are building something along those lines, the community exchanges experience about exactly these architecture decisions.
If what you want is not to query a model through an API but to use an AI assistant to write the firmware that makes the query, the working flow is in Using an AI assistant to program ESP32.