Chatbot Controller Robot Arm With Camera

The goal is to build something to sort a bucket of lego into individual kits and make rebuilding them straight forward.

1. I don't want to teach it how to do it - the chatbot must be smart
2. It can run for a week - I don't care as long as it sorts them
3. It can't require continuous input from me

Written in python, this project demonstrates how a chatbot can see and control a robot arm. The project runs completely offline. The llama3.2-vision model is not yet smart enough to understand that their is nothing between the pincers, but it can move around on command. Model evaluation is between 10 and 300 seconds with an old Nvidia TESLA m40 24GB.

Turn up volume to hear conversation

ChatbotRobotArm.mov

Technologies used

1. 6DOF Robotic Arm Kit from Amazon
2. ESP32-CAM Webcam flashed with Webcam Software
3. USB to serial UART to control arm from PC (No dependency on this hardware)
4. Ollama as the "offline" chatbot
5. GPU - I picked up a Nvidia TESLA m40 24GB for $70 on ebay
6. llama3.2-vision model as it accepts image input
7. Text-To-Speach to speak the chatbots' responses - runs offline
8. Speach Recognition to listen for commands
9. Python3.11 (No dependency on this version)
10. Windows 10 (No dependency on this Platform)

Next Steps

1. Evaluate online chatbots to speed up evaluation
2. Evaluate whether a second camera, and/or other angles helps chatbot understanding
3. Add interfaces to allow different hardware and APIs to be configured
4. Find a vision aware model that uses Ollamas newer "tools" interface

Installation

git clone https://github.com/ribbles/ChatbotRobotArm/
cd ChatbotRobotArm
pip install -r requirements

Testing

pytest

Starting

python src/server.py