General-Purpose AI Visual Assistant

An open-source multimodal assistant that sees through your camera, reasons with the Vision Language Model (VLM) of your choice, and answers with natural speech. Run it on a laptop, workstation, or Raspberry Pi while keeping full control of hardware, prompts, and providers.

---

🌟 Highlights

• Task-Driven Workflow – Define the job once (e.g., describe a room, play tic-tac-toe, inspect a circuit) and let the VLM handle perception and reasoning in a single pass.
• Docker Ready – Fully containerized with dedicated support for x86_64 and Raspberry Pi (ARM64).
• Bring-Your-Own Model – Switch between Ollama, Gemini, OpenAI, HuggingFace, or local GGUF via llama-cpp-python via config.yaml.
• Modular Architecture – Clean separation of concerns between VisionController, ConversationManager, and CoreProcessor for easy extensibility.
• Interactive + Periodic Modes – Move seamlessly between continuous scanning and interactive conversations with voice or text input.
• On-Demand OCR – Integrated PaddleOCR for quick text extraction without involving the heavy VLM.
• Video Analysis – Capture and analyze short video sequences directly through the VLM.

---

🏗️ Architecture Overview

      +-----------------+
      |  InputHandler   |
      +--------+--------+
               | (Commands)
               v
      +--------+--------+
      |   Orchestrator  | <--- Central Hub
      +--------+--------+
               |
     +---------+---------+
     |                   |
     v                   v
+---------+         +---------+       +--------------+
| Vision  |-------> |  Core   |-----> | Model Client |
| Control | (Frame) | Process |       | (API Calls)  |
+---------+         +---------+       +--------------+
                         ^
                         |
                  +--------------+
                  | Conversation |
                  |   Manager    |
                  +--------------+

1. Orchestrator: The central controller that coordinates input, vision, and processing logic.
2. VisionController: Manages camera hardware, video buffering, and preview threads.
3. ConversationManager: Centralizes chat history and keeps the system/user/assistant roles.
4. CoreProcessor: The engine coordinating vision/video capture and model inference.
5. InputHandler: Normalizes user commands from keyboard or microphone.

---

🚀 Getting Started

Prerequisites

• Python 3.10+
• Camera (Webcam or PiCamera)
• Optional: Microphone for voice commands, Speakers for TTS.

Installation

1. Clone the repo:

   git clone https://github.com/YOUR_USERNAME/YOUR_REPO_NAME.git
   cd YOUR_REPO_NAME

2. Setup environment:

   python -m venv .venv
   source .venv/bin/activate  # On Windows: .venv\Scripts\activate

3. Install dependencies:

   # Standard install
   pip install -r requirements.txt
   
   # Developer install (with testing/linting tools)
   pip install -e ".[dev]"

4. Configure API keys (only for cloud providers):

   cp .env.example .env
   # Edit .env and add your API keys

5. Configure the assistant:

   cp config.yaml.example config.yaml
   # Edit config.yaml to choose your provider, model, and preferences

Configuration (config.yaml)

The system uses a single configuration file. Key sections:

model:
  provider: huggingface        # openai | gemini | ollama | huggingface | gguf_local
  model: null                 # Optional: overrides provider default

prompt:
  system: "Descrivi l'immagine in italiano. Sii breve."
  user: "Cosa vedi?"

camera_source: webcam          # or picamera
interactive_mode: true         # true = chat, false = periodic scans
do_speak: false               # Enable Text-to-Speech

video:
  enabled: true
  duration_seconds: 3.0
  fps: 2

# Provider-specific defaults (used when model.model is null)
openai:
  model: "gpt-4o-mini"
  generation:
    max_tokens: 512
    temperature: 0.2

gemini:
  model: "gemini-2.5-flash"
  generation:
    max_output_tokens: 1024
    temperature: 0.6

ollama:
  host: "http://127.0.0.1:11434" # Localhost (default)
  # host: "http://192.168.1.100:11434" # Remote PC (e.g., if running app on Raspberry Pi)
  model: "llava"
  generation:
    num_ctx: 4096
    temperature: 0.2

💡 Tip: Distributed Inference (Raspberry Pi + Laptop)

You can run this application on a Raspberry Pi (for camera/audio) while offloading the heavy AI processing to a powerful Laptop/PC running Ollama.

1. On your Laptop, run Ollama with: OLLAMA_HOST=0.0.0.0 ollama serve
2. Find your Laptop's IP (e.g., 192.168.1.100).
3. In config.yaml on the Pi, set: host: "http://192.168.1.100:11434".

---

🐳 Docker Deployment

The application is fully containerized.

Note

Windows Users: Docker Desktop cannot access USB webcams on Windows. See DOCKER_WEBCAM_WINDOWS.md for alternatives. The recommended approach is running natively without Docker on Windows.

Desktop / Laptop (x86_64)

Use docker-compose with profiles:

# Start with webcam support (Linux only)
docker-compose --profile webcam up --build

# Start with NVIDIA GPU and webcam support
docker-compose --profile full up --build

Raspberry Pi (ARM64)

Build with the ARM-optimized Dockerfile:

docker build -t raspberry-llm -f Dockerfile.raspberry .
docker run --rm --device=/dev/video0 -v $(pwd)/config.yaml:/app/config.yaml raspberry-llm

---

🛠️ Usage & Commands

Command	Description
stop	Ends the session gracefully.
nuova immagine	Captures a fresh frame while keeping the chat context.
leggi testo	Triggers the OCR flow (PaddleOCR).
analizza video	Captures a video sequence and analyzes it.
Any text	Sends a follow-up question to the VLM.

---

💻 Developer Guide

This project follows modern Python standards using pyproject.toml.

• Testing: Run pytest to execute the test suite (configured in pyproject.toml).
• Typing: Use mypy src/ for static type checking.
• Logging: Structured logs are available via src.utils.logger.
• Packaging: The project is ready for distribution as a Python package.

---

🤝 Contributing

Contributions are welcome! Please open an issue or submit a pull request for any improvements or bug fixes.