TinyML: Getting Started With Running a Model on a Microcontroller

Earlier today we wrote about PocketGPT — a 30-million-parameter language model, trained from scratch, answering in about six seconds on an ESP32-S3. That is the far end of this territory. This is the near end: getting any trained model running on a microcontroller, which is a well-paved path and useful long before you attempt anything that large.

What it buys you for creative work: a sensor that recognises a gesture, a microphone that spots one keyword, a camera that knows whether the gallery is empty — all with no network, no latency and no privacy conversation.

First: which repository, because the names lie

This trips up almost everyone, and it is worth two minutes before you install anything. All three of these are live and Apache-2.0, and they are not interchangeable:

Project	What it is	Status
`google-ai-edge/LiteRT`	The successor to TensorFlow Lite, for phones and edge devices — high-performance ML and GenAI with GPU/NPU delegates	3,523 stars, pushed today
`tensorflow/tflite-micro`	The microcontroller runtime: no OS, no dynamic allocation, kilobytes of RAM	3,129 stars, pushed 23 Oct 2026
`espressif/esp-tflite-micro`	`tflite-micro` packaged as an ESP-IDF component, with ESP32 optimisations	715 stars, pushed 21 Sep 2026

The trap: “TensorFlow Lite was renamed LiteRT” is true, and it is not true of the microcontroller path. tflite-micro kept its name, is actively maintained, and is what you want on an MCU. If a tutorial tells you to install LiteRT on an ESP32, it has followed the rename past the point where it applies.

If you are on an ESP32, start at espressif/esp-tflite-micro. If you are on an RP2040, nRF52 or STM32, start at tensorflow/tflite-micro directly, or use the Arduino library wrapper.

The pipeline, in five steps

train (Python, full float)
  → quantise + convert (Python) → model.tflite
  → xxd → model_data.cc   (a C array)
  → compile into firmware
  → run: interpreter + tensor arena

Everything up to step three happens on your laptop. Only the last two touch the board.

Step 1: train a deliberately small model

Train in Keras or PyTorch as normal, but design for the target from the start rather than shrinking later. Practical envelope for a small MCU:

Tens of thousands of parameters, not millions. A keyword spotter is routinely 20–50 KB.
Small, fixed input. A 32×32 greyscale image or a 40×49 MFCC spectrogram, not 224×224 RGB.
Boring layers. Conv2D, DepthwiseConv2D, fully connected, ReLU, max pooling, softmax. Every exotic op is an op the micro runtime may not implement.
Classification over regression where you have the choice; a handful of classes is easier to quantise without accuracy loss.

If you want to skip training entirely for a first run, both repositories ship a hello-world sine model and a micro-speech example. Get those on the board before training anything of your own — it isolates toolchain problems from model problems.

Step 2: quantise to int8 — not an optimisation, a requirement

This is the step people try to skip. Do not.

import tensorflow as tf
import numpy as np

def representative_dataset():
    # 100-500 real samples, same preprocessing as training.
    for sample in calibration_samples[:200]:
        yield [np.expand_dims(sample.astype(np.float32), axis=0)]

converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8

tflite_model = converter.convert()
open("model.tflite", "wb").write(tflite_model)

Three reasons it is mandatory rather than nice:

Size. Float32 → int8 is a 4× reduction, directly, in the thing that has to fit in flash.
Speed. Most MCUs have no floating-point unit worth using. Integer arithmetic is not marginally faster; it is the difference between 40 ms and 400 ms.
RAM. Activations are quantised too, which shrinks the arena you are about to size.

The representative_dataset is the part that is easy to get wrong. The converter uses it to measure the actual range of every activation so it can pick scale factors. Feed it real data with the identical preprocessing you will run on the device. Random noise technically works and produces a model that is quietly much worse. A few hundred genuine samples is plenty.

Check the damage afterwards: run your test set through both the float model and the quantised one. A point or two of accuracy is normal. Ten points means something is wrong — usually the calibration data, sometimes a layer that quantises badly.

Step 3: turn the model into a C array

The board has no filesystem to read from, so the model gets compiled in:

xxd -i model.tflite > model_data.cc

That emits unsigned char model_tflite[] = { 0x1c, 0x00, ... }; plus a length. Rename the symbol to something sane, add a matching header, and keep the generated file out of hand-editing — regenerate it every time the model changes, and add a comment recording which training run produced it. (On Windows, or if xxd is missing, both repositories ship an equivalent Python script.)

Step 4: wire up the interpreter

The minimal shape, C++:

#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/micro/micro_mutable_op_resolver.h"
#include "model_data.h"

constexpr int kArenaSize = 16 * 1024;          // tune this -- see below
alignas(16) static uint8_t tensor_arena[kArenaSize];

void setup() {
  const tflite::Model* model = tflite::GetModel(model_tflite);

  // List ONLY the ops your model uses. Each one costs flash.
  static tflite::MicroMutableOpResolver<5> resolver;
  resolver.AddConv2D();
  resolver.AddDepthwiseConv2D();
  resolver.AddMaxPool2D();
  resolver.AddFullyConnected();
  resolver.AddSoftmax();

  static tflite::MicroInterpreter interpreter(
      model, resolver, tensor_arena, kArenaSize);
  interpreter.AllocateTensors();

  TfLiteTensor* input = interpreter.input(0);
  // fill input->data.int8 with your quantised sensor data ...
  interpreter.Invoke();
  TfLiteTensor* output = interpreter.output(0);
  // read output->data.int8 ...
}

Two details that are not obvious:

Use MicroMutableOpResolver, not the all-ops resolver. The all-ops version links every supported operator into your firmware — tens of kilobytes you are not using. Listing five ops explicitly is both smaller and a useful error: if you forgot one, AllocateTensors() fails and tells you which.

Quantised in, quantised out. Your input tensor wants int8, and your output comes back as int8. To recover a real probability you need the tensor’s scale and zero point:

float score = (output->data.int8[i] - output->params.zero_point) * output->params.scale;

Forgetting this is the most common reason a model “works on the laptop and outputs garbage on the board.”

Step 5: the tensor arena — the one genuinely new concept

The arena is a single fixed block of RAM you allocate yourself, from which the interpreter carves every intermediate tensor. There is no malloc on this path, by design — a model that cannot run in the arena you gave it fails immediately rather than fragmenting the heap at 3 a.m. during an exhibition.

There is no formula. The accepted method is to guess and measure:

Start generously — say 64 KB — and get it running.
Call interpreter.arena_used_bytes() after AllocateTensors().
Set the arena to that value plus a small margin, and rebuild.

That is the whole technique, and it is worth doing properly because the arena is usually the largest single RAM allocation in the firmware. If AllocateTensors() returns an error, the arena is too small; if your board resets unpredictably under load, it may be too large and squeezing the stack.

On an ESP32 with PSRAM you have more headroom and can place the arena there, at some speed cost. On an RP2040’s 264 KB, arena sizing is the whole engineering problem.

When this is the wrong tool

You need to recognise arbitrary speech or objects. Ship audio or frames to a real machine, or use a board with an NPU.
You are doing one inference a minute and have mains power and Wi-Fi. An API call is simpler and more accurate.
You do not have labelled data. No amount of toolchain fixes that.

TinyML’s actual sweet spot is one specific recognition task, running constantly, where the network cannot be trusted or does not exist — which describes a surprising number of installations.

Where to go next

esp-tflite-micro’s examples — micro_speech is the best-documented end-to-end case and runs on commodity ESP32 boards.
CMSIS-NN on Cortex-M targets, which gives large speedups on the same int8 model for one build-flag’s worth of effort.
Pruning and clustering before quantisation, if int8 alone has not got you small enough.

First: which repository, because the names lie

The pipeline, in five steps

Step 1: train a deliberately small model

Step 2: quantise to int8 — not an optimisation, a requirement

Step 3: turn the model into a C array

Step 4: wire up the interpreter

Step 5: the tensor arena — the one genuinely new concept

When this is the wrong tool

Where to go next

Sources

Related Reading

Related Reading

Getting Started With Reflashing Off-the-Shelf ESP32 Devices

Getting Started: The ESP32 as the Brain of an Interactive Art Piece

Etherbulb Turns a Cheap Smart Bulb Into a Radio That Colours Your Room