An autonomous robotic arm powered by an ESP32-S3 that learns to control its own movements in real-time. This project forsakes traditional, pre-trained AI models and instead implements a self-learning system based on predictive coding and Hebbian principles.
Date: June 30, 2025
Tags: ESP32-S3
, On-Device Learning
, Hebbian Learning
, Predictive Coding
, Robotics
, ESP-IDF
The philosophy of this project is to explore true on-device learning and self-discovery. Instead of running inference on a massive, pre-trained model, this robot starts with a randomized neural network and learns by trying to predict the sensory consequences of its own actions.
The main loop follows this cycle:
- Sense: Read the current physical state from the accelerometer.
- Act & Predict: The neural network takes the sensor state and produces two outputs:
- An action (a command for the robot's motors).
- A prediction of what the sensor state will be after the action.
- Observe: The action is performed, and the new, actual sensor state is read.
- Learn: The "prediction error" (the difference between the predicted and actual state) is calculated. The neural network then uses this error to update its own weights via a Hebbian-like rule: connections that led to accurate predictions are strengthened.
Over time, the robot builds an internal model of its own physics, learning the causal relationship between its commands and their outcomes.
- Real-Time On-Device Learning: All learning happens directly on the ESP32-S3 with no need for a PC.
- Predictive Hebbian Model: A computationally lightweight and biologically-inspired learning algorithm.
- Modular C Drivers: Clean, separated drivers for all hardware components, written for the ESP-IDF framework.
- Visual Feedback: The onboard RGB LED provides real-time feedback on the model's learning "fitness," shifting from Red (high error) to Green (low error).
- Feetech Slave Interface: The ESP32 exposes a Feetech-compatible slave interface over its native USB port. This allows it to be controlled by higher-level frameworks like LeRobot as if it were a smart servo controller.
- Microcontroller: An ESP32-S3 board with access to the native USB-OTG port (e.g., ESP32-S3-DevKitC-1 v1.1).
- Robotic Arm: SO100 / SO-101 with FeeTech Serial Bus Servos.
- Sensor: SparkFun BMA400 Accelerometer (or another BMA400 breakout).
- Bus Adapter: Waveshare Bus Servo Adapter (A). This is required to interface the ESP32's UART with the half-duplex serial bus of the FeeTech servos.
- Power Supply: An external 5V-7.4V power supply capable of driving the servos. Do not power the servos from the ESP32's 3.3V or 5V pins.
- Wiring: Jumper wires.
Properly wiring the components is critical. Follow this guide carefully.
- Connect your external power supply's
+
terminal to theVin
(orVCC
) on the Waveshare Bus Servo Adapter. - Connect your external power supply's
-
terminal to theGND
on the Waveshare Adapter AND to aGND
pin on the ESP32-S3. A common ground is essential.
- ESP32
GPIO 21
(SDA) -> BMA400SDA
- ESP32
GPIO 22
(SCL) -> BMA400SCL
- ESP32
3.3V
-> BMA400VCC
- ESP32
GND
-> BMA400GND
- IMPORTANT: The Waveshare adapter requires a non-standard "straight-through" UART connection, as specified in its documentation.
- ESP32
GPIO 17
(UART TX) -> Waveshare AdapterTXD
- ESP32
GPIO 16
(UART RX) -> Waveshare AdapterRXD
- Connect your daisy-chained FeeTech servos to the 3-pin servo bus header on the Waveshare Adapter.
- Ensure the jumper on the Waveshare adapter is set to position 'A' for UART control.
This project is built using the ESP-IDF v5.4. It relies on the following managed components, which are listed in main/idf_component.yml
:
espressif/led_strip
: The official driver for the addressable RGB LED.espressif/esp-dsp
: The ESP Digital Signal Processing library.espressif/esp_tinyusb
: The component providing the native USB driver for the slave interface.
.├── main/│ ├── bma400_driver.c│ ├── bma400_driver.h│ ├── feetech_protocol.c│ ├── feetech_protocol.h│ ├── led_indicator.c│ ├── led_indicator.h│ ├── main.c│ └── CMakeLists.txt ├── .gitignore ├── CMakeLists.txt ├── idf_component.yml └── README.md
-
Clone the Repository:
git clone <your-repo-url> cd esp32_hebbian_learning
-
Setup ESP-IDF: Ensure you have ESP-IDF v5.4 installed and the environment is activated. (e.g., run
source ~/esp/esp-idf/export.sh
on Linux/macOS or use the ESP-IDF CMD Prompt on Windows). -
Install Dependencies: The project uses the
led_strip
component from the IDF Component Manager. The first time you build, the system will automatically read theidf_component.yml
file and download it. If you need to add it manually, run:idf.py add-dependency "espressif/led_strip"
-
Set Target Chip: Tell ESP-IDF you are building for an ESP32-S3.
idf.py set-target esp32s3
-
Configure (Optional): Run
idf.py menuconfig
to set your serial port underSerial Flasher Config
. -
Build, Flash & Monitor: Run the all-in-one command to compile, upload the firmware, and view the output.
# Replace COMx with your serial port (e.g., /dev/ttyUSB0 or COM3) idf.py -p COMx flash monitor
This repository includes a .gitignore
file tailored for ESP-IDF projects. It ensures that temporary build artifacts (/build
), local configurations (sdkconfig
), and downloaded dependencies (/managed_components
) are not committed to the repository. This is best practice, as these files are generated locally on each developer's machine during the build process.
This project evolved significantly from its initial concept. The journey involved solving several common embedded development challenges:
- Initial Concept: Inspired by the
LokiMetaSmith/esp32-llm
project, the initial goal was to explore on-device training of a large model. - Algorithm Pivot: Realizing that backpropagation was too resource-intensive, the approach was pivoted to a more efficient Hebbian/Predictive Coding model.
- Toolchain Failure: The first build attempts failed due to a corrupted ESP-IDF toolchain. The compiler could not build a simple test program. Solution: A full re-installation of the ESP-IDF tools using the official online installer fixed the core environment.
- Missing Target: Early builds failed because the target chip was not specified. Solution: Running
idf.py set-target esp32s3
configured the project correctly. - Missing Header Files: The build failed multiple times with "undeclared" or "No such file or directory" errors for headers like
gpio.h
anduart.h
. Solution: Added the required componentdriver
to theREQUIRES
list inmain/CMakeLists.txt
. - Component API Change: The
led_strip
driver code failed to compile due to changes in its API. Solution: Updated the code to match the latest version of the component. - Component Dependency: The final issue was that the
led_strip
component itself was not found. Solution: Learned to use the IDF Component Manager and added the dependency withidf.py add-dependency "espressif/led_strip"
, the modern way to handle external components in ESP-IDF.