INT8 inference on an ESP32

This repo is a small working example of running a quantized neural network on a cheap microcontroller. The demo itself is a gesture classifier, but the actual point of the repo is the engine and the pipeline behind it: a tiny C inference runtime, and the Python tooling that takes a float32 model and turns it into an INT8 binary the ESP32 can load and run.

The demo uses a three layer dense network (30 inputs, then 16, then 8, then 4 outputs). It is intentionally small so you can read every file and follow the whole path from sensor to prediction. The same engine can host larger dense only networks without changes, just by retraining and requantizing.

Some numbers from this demo

A few things worth knowing before you read the rest:

• 668 trainable parameters in the model
• float32 model is about 2.9 KB
• INT8 model is about 1.1 KB
• Quantization is symmetric per tensor, with int32 bias accumulators
• Weights shrink by roughly 4x (4 bytes per weight down to 1 byte)
• One forward pass is 640 int8 multiply-accumulates
• On an ESP32 at 240 MHz that runs in roughly 0.05 to 0.2 ms with naive C code
• The demo runs inference twice per second (once per 0.5 s sensor window)

For this specific model, quantization is not strictly necessary. The float version already fits with room to spare. The reason to do it anyway is that the same pipeline becomes essential the moment the model gets bigger, or the target chip has no FPU, or the inference has to run continuously on a battery. Those are the cases worth keeping in mind while reading the rest of this.

When quantization actually matters

INT8 vs float32 is not just a "faster" knob. It matters when at least one of the following is true.

The model is large enough to strain flash or RAM. Our 1 KB model is trivial. A CNN around 1 MB is not. On a 4 MB flash chip with firmware, OTA space, and logs, INT8 is often the only way the model fits at all.

The chip has no FPU. The original ESP32 has a single precision FPU, so float32 is not catastrophic here. But a lot of cheap chips do not: the ESP32-S2, nRF52, RP2040, many STM32 families. On those, naive float32 inference can be 5 to 10x slower than int8, which is the difference between real time and unusable.

Inference is continuous. Our classifier runs twice per second. An always on keyword spotter runs hundreds of times per second. Cycle savings compound directly into battery life and thermal budget.

Latency is critical. Wake word detection and reactive control systems care about milliseconds. Going from 40 ms to 10 ms matters there.

Battery life is the constraint. Fewer cycles per inference means more time in deep sleep between windows. For always listening devices that roughly triples battery life because the active current dominates the budget.

Where the same pattern gets used for real

The exact same pipeline, train in float32 on a laptop, quantize, ship a small binary, run with a tiny C runtime, is what people use in production for a lot of TinyML work. A few examples that map onto this engine or a slightly extended version of it:

• Wake word detection, the "Hey Siri" / "Alexa" / "Ok Google" class of problems. Small CNN over an audio spectrogram, always on, running on battery, with a 50 ms latency budget.
• Keyword spotting on hearing aids. Tiny RNN or CNN, sub 50 ms latency, and a battery the size of a pill.
• Predictive maintenance on factory machines. CNN or LSTM over vibration or current signals, continuous monitoring on battery powered sensor nodes.
• Activity recognition on a wearable. Small 1D CNN or GRU running 24/7, squeezing inference between deep sleep windows.
• Sound classification for security or smart home. Detect glass break, smoke alarms, baby cries. Always listening on a small battery.
• Person detection on an ESP32-CAM. MobileNetV1 0.25 or FOMO at low resolution. Has to fit alongside firmware in 4 MB of flash.
• Smart agriculture. Pest or disease detection from a camera on a field node. Flash constrained, often no fast connection.
• Drone obstacle avoidance. Small CNN onboard, low latency, low power, because flight time is sacred.

What is in this repo

• mpu6050_stream/mpu6050_stream.ino: a barebones ESP32 sketch that streams raw MPU6050 data as CSV over serial at about 100 Hz.
• collect.py: reads the serial stream, lets you label data with your keyboard, saves gesture_data.csv.
• train.py: trains the 30 -> 16 -> 8 -> 4 MLP from scratch using only NumPy. No torch, no sklearn, no surprises.
• quantize.py: converts the trained float weights into a quantized model_int8.bin for the ESP32.
• inference_engine.h: a tiny C inference engine that loads model_int8.bin and runs INT8 prediction. Header only, no dependencies.
• gesture_classifier/gesture_classifier.ino: a complete ESP32 demo sketch. Reads sensor, extracts features, runs inference, prints the result.
• requirements.txt: Python dependencies.

What you need

• An ESP32 dev board
• An MPU6050 IMU connected over I2C (on most ESP32 boards SDA is GPIO 21 and SCL is GPIO 22)
• A USB cable
• A computer with Python 3

Default wiring for most ESP32 boards: VCC to 3.3 V, GND to GND, SDA to GPIO 21, SCL to GPIO 22.

The pipeline

Four steps, each one producing the input for the next:

ESP32 raw sensor data
        |
        v
collect.py   ->   gesture_data.csv
        |
        v
train.py     ->   model_weights.npz
        |
        v
quantize.py  ->   model_int8.bin
        |
        v
gesture_classifier.ino runs INT8 inference on the ESP32

The model

A small MLP, dense only, three layers:

x in R^30
z1 = W1 x + b1     W1 in R^(16x30), b1 in R^16
a1 = ReLU(z1)
z2 = W2 a1 + b2    W2 in R^(8x16),  b2 in R^8
a2 = ReLU(z2)
z3 = W3 a2 + b3    W3 in R^(4x8),   b3 in R^4
p  = softmax(z3)   p in R^4

Total: 640 weights and 28 biases, so 668 parameters. The model outputs logits, then softmax turns them into a 4 class probability vector for idle, shake, flick, circle. On device we skip the softmax because argmax over logits gives the same class as argmax over probabilities, and we avoid the exp budget.

Loss is categorical cross entropy:

L = -sum_k y_k log(p_k)

The nice shortcut is that softmax combined with cross entropy gives a clean output layer gradient: dL/dz3 = p - y. We use that directly during backprop instead of deriving it through the softmax Jacobian every step.

Step 1: collect labeled data

Flash mpu6050_stream/mpu6050_stream.ino from the Arduino IDE. The sketch outputs CSV lines that look like:

ax,ay,az,gx,gy,gz
123,456,7890,-12,34,-5
...

at about 100 Hz.

Then run the collector:

pip install -r requirements.txt
python collect.py --port /dev/ttyUSB0 --out gesture_data.csv

On Windows the port will look like COM3. On macOS it might be /dev/tty.usbserial-0001.

To label, hold a key while you perform the gesture and release when you are done. The default label is idle.

• 0 = idle
• 1 = shake
• 2 = flick
• 3 = circle
• q = quit

Try to record a few hundred samples of each gesture. Do each one at different speeds and orientations, because variety helps. Do not move during the idle recordings, the model needs a real still baseline.

Every sample is written to gesture_data.csv with a timestamp and the current label.

Feature extraction

The model does not eat raw samples. It eats 30 features computed from a 0.5 second window.

For each of the 6 channels over the 50 sample window we compute 5 statistics:

mu   = (1/50) sum_t w[t]
sigma = sqrt((1/50) sum_t (w[t] - mu)^2)
mn   = min_t w[t]
mx   = max_t w[t]
rms  = sqrt((1/50) sum_t w[t]^2)

That is 6 x 5 = 30 features per window. The same extraction runs identically in train.py on the laptop and in inference_engine.h on the ESP32, so what the model sees during training matches what it sees at inference. There is no learned preprocessing, just fixed stats computed the same way on both sides.

Step 2: train the model

python train.py --data gesture_data.csv --out model_weights.npz

This will:

• read gesture_data.csv
• cut it into 50 sample windows with a 25 sample stride (50 percent overlap)
• compute the 30 features above for each window
• skip windows that cross a label boundary, so transition noise does not leak in
• standardize each of the 30 features using the training set mean and std
• train the MLP from scratch with NumPy, Adam, cross entropy
• print accuracy, confusion matrix, and per class metrics
• save the float weights and the scaler to model_weights.npz

Training takes a few seconds on a laptop. The model is intentionally small so it trains fast and quantizes cleanly.

If accuracy is low, the answer is almost always more data, not a bigger model. Try to make each gesture class more consistent across recordings.

Standardization

Before training, every feature is z-scored:

x_hat = (x - mu_d) / sigma_d

where mu_d and sigma_d are computed per feature dimension from the training set only. The scaler mean and scale get saved into model_weights.npz, and then again into model_int8.bin, because the ESP32 has to apply the exact same transform at inference time. Without this the 30 features would have wildly different scales (a few counts for gyro std vs thousands for accel max) and training would be slow and unstable.

Backprop, briefly

Output gradient, from the softmax + cross entropy shortcut:

dz3 = (1/N) (P - Y)         shape (N, 4)
dW3 = dz3^T a2              shape (4, 8)
db3 = sum_i dz3_i           shape (4,)

Then back through the second dense layer and its ReLU:

da2 = dz3 W3                shape (N, 8)
dz2 = da2 * ReLU'(z2)       shape (N, 8)
dW2 = dz2^T a1
db2 = sum_i dz2_i

And the same pattern again for layer 1, with dz2 W2 feeding da1, da1 * ReLU'(z1) giving dz1, and dz1^T x giving dW1. Nothing exotic, just the chain rule applied layer by layer. The transposes are there purely so the matrix shapes line up with PyTorch convention, where W[i] @ x is one neuron's pre activation.

Weights are updated with Adam:

m_t = b1 m_{t-1} + (1-b1) g_t
v_t = b2 v_{t-1} + (1-b2) g_t^2
theta -= lr * (m_t / (1-b1^t)) / (sqrt(v_t / (1-b2^t)) + eps)

Defaults are lr=1e-3, b1=0.9, b2=0.999, eps=1e-8. Nothing here is gesture specific, the same optimizer would train a dense MLP for any other task.

Step 3: quantize

python quantize.py --data gesture_data.csv --weights model_weights.npz --out model_int8.bin

This is the step where float32 turns into something the ESP32 can chew on cheaply.

Symmetric per tensor quantization

For each weight tensor we find a single scale:

scale_w = max(|W|) / 127
q_w    = round(W / scale_w), clip to [-128, 127]

Symmetric means the zero point is always zero, which keeps the runtime simple. Per tensor means one scale per weight matrix, not per channel. For dense only networks this is good enough and the loader code stays short. CNNs usually want per channel on the output axis, but we are not doing convs here.

Biases are kept as int32, not int8, because they get added to an int32 accumulator and shrinking them would throw out too much precision. Their scale has to match the accumulator's scale, which is the product of the input scale and the weight scale:

scale_b = scale_in * scale_w
q_b    = round(b / scale_b)   int32

Requantization between layers

After each hidden layer's MAC you have an int32 accumulator. To feed it into the next int8 layer you have to squeeze it back to int8 with a new scale. The calibration is data driven: the quantizer runs the quantized network against the training set and uses the actual ReLU output range to pick the next layer's input scale.

acc_int32 = q_W @ q_in + q_b
acc_float = acc_int32 * scale_b
a_relu    = max(0, acc_float)
q_out     = round(a_relu / scale_out), clip to [-128, 127]

After this requantize, ReLU on the int8 side is literally max(0, q), because the zero point is symmetric. That is a major reason we chose symmetric over asymmetric: ReLU becomes a single clamp instead of q = max(zp, q) with an extra zero point constant to lug around.

The output layer does not requantize. Its int32 logits go straight into argmax, no softmax, no dequantize. That is the fastest path to a class label and loses nothing compared to softmax + argmax on float.

What goes into the binary

The output file model_int8.bin is about 1 KB and contains:

• input scale and zero point
• scaler mean and scale for the 30 features
• per layer: int8 weights, int32 biases, both scales, and for the hidden layers also the output scale

The binary format is documented at the top of inference_engine.h. Small header, then each layer's weights, biases, and scales in order. It is little endian because the ESP32 is.

The script prints a quantized accuracy check at the end so you can see right away whether the INT8 version is still good. If the drop is large, you probably need more training data, or your float model has weight outliers that blow up the per tensor scale.

Step 4: run on the ESP32

The gesture_classifier.ino sketch expects model_int8.bin to already be on the ESP32 filesystem (SPIFFS). In the Arduino IDE:

1. Create a folder called data inside your sketch folder.
2. Copy model_int8.bin into that folder.
3. Install the ESP32 Sketch Data Upload tool if you haven't already.
4. Use Tools, then ESP32 Sketch Data Upload, to flash the file to SPIFFS.

Then open gesture_classifier/gesture_classifier.ino in the Arduino IDE. Make sure inference_engine.h is in the same folder as the .ino file, and upload the sketch.

Open the serial monitor at 115200 baud. You should see something like:

=== Gesture Classifier ===
MPU6050 configured: 100 Hz, +/-500 deg/s gyro, +/-4 g accel
Model loaded: 30->16->8->4
Starting classification...

Gesture: idle  (0)
Gesture: idle  (0)
Gesture: shake  (1)
Gesture: shake  (1)
Gesture: circle  (3)

Once the 50 sample buffer fills up, it classifies on every loop.

Inference on device

At inference time the engine does, in order:

Standardize and quantize the 30 features:

x_std = (x - scaler_mean) / scaler_scale
q_x  = clip(round(x_std / scale_in), -128, 127)

Layer 0, int8 MAC into int32, requantize, ReLU as max(0, q):

acc1[i]  = q_W1[i] . q_x + q_b1[i]            (int32)
q1[i]    = clip(round(acc1[i] * scale_b1 / scale_out1), 0, 127)

Same shape for layer 1. Then layer 2 stops at the int32 accumulator and argmax picks a class:

logits[i] = q_W3[i] . q2 + q_b3[i]            (int32, no requantize)
class     = argmax_i logits[i]

No float math at all on the device except the standardization step, which uses the saved scaler. If you wanted to go fully integer you would also quantize the feature pipeline, but for a 0.5 Hz classifier the standardization is cheap and not worth the extra bookkeeping. For an always on audio classifier you would push the feature pipeline into int8 too, because that becomes the real cost.

How the demo works, in plain English

The MPU6050 gives six numbers: three axes of acceleration and three axes of rotation. The ESP32 collects 50 of those readings, which is half a second of motion.

For each of the six channels it computes five statistics: mean, standard deviation, minimum, maximum, and RMS. That gives 30 numbers, which are fed into the tiny neural network. The four outputs are scores for the four gestures, and the highest score wins.

All inference on the ESP32 happens in INT8 to keep the model small and fast. The laptop does the hard part: training and quantization.

Using the engine for other models

The inference engine in inference_engine.h is not specific to gestures. It will load any dense only MLP that matches the binary format produced by quantize.py. To use it for a different task:

1. Pick an input feature vector and number of classes.
2. Train a dense only MLP in train.py (change the layer sizes and the label set).
3. Run quantize.py to produce a new model_int8.bin.
4. Flash it and call inference(&model, features) from your own sketch.

The engine handles int8 weights, int32 bias accumulators, ReLU activations, and per layer output requantization. It does not support convolutions, reshapes, or other ops, and that is deliberate. It is meant to stay minimal for dense only TinyML work, so you can actually read it end to end.

If your target task needs conv, you are better off with TFLite Micro or Edge Impulse, both of which already live in ref/ for reference.

Tips

Data quality matters more than model size

If accuracy is bad, the problem is almost always the training data, not the model. A few things worth checking:

• Do you have enough samples of each class? Aim for hundreds per gesture.
• Are your labels clean? A single transition window labeled as a gesture will confuse the model.
• Is the sensor wired correctly? recorder/recorder.ino is handy for debugging raw MPU6050 output.

Collect idle data carefully

Idle should mean the device is actually still. If you wave it around while holding 0, the model will learn the wrong thing.

Flicks have direction

A flick left and a flick right look different on the accelerometer. You either record them as the same class and accept that the model learns the general flick motion, or you split them into two classes and grow the output layer by one.

Quantization drops a little accuracy

That is normal. If the quantized accuracy is much lower than the float accuracy, the float model is probably overfit or the weights have large outliers. More diverse data usually fixes it without changing the quantization scheme.

The binary format

If you want to write your own loader, the format is documented in quantize.py and inference_engine.h. It is intentionally simple: a small header, then each layer's weights, biases, and scales in order.

License

Do whatever you want with this. It's a tiny proof of concept.