Spatial Detector

A Python package for 3D object detection and spatial mapping using a webcam, combining YOLO object detection with monocular depth estimation.

Overview

Spatial Detector combines state-of-the-art object detection with monocular depth estimation to create a 3D representation of objects in your environment using only a standard webcam. It provides both a command-line interface for interactive use and a Python API for integration into your own applications.

The project uses YOLOv8 for object detection and MiDaS for depth estimation, with optimizations for both Apple Silicon (M1/M2) through Metal Performance Shaders and NVIDIA GPUs through CUDA.

Features

Core Capabilities

Real-time Object Detection: Identify and classify objects using YOLOv8
Monocular Depth Estimation: Calculate distances using neural network-based depth sensing
3D Spatial Mapping: Project detected objects into 3D space
Real-time Visualization: See depth maps and object positions interactively
Top-down View: Get a bird's eye perspective of your environment

Technical Features

Hardware Acceleration: Optimized for:
- Apple Silicon (M1/M2) chips using Metal Performance Shaders (MPS)
- NVIDIA GPUs using CUDA
- Fallback to CPU for compatibility
Depth Calibration: Tools for accurate real-world measurements
Interactive Controls: Toggle visualizations and adjust settings in real-time
Persistence and Tracking: Follow objects as they move through your space

Installation

Prerequisites

Python 3.7 or later
PyTorch 1.13 or later
OpenCV 4.5 or later

Option 1: Install from PyPI (Recommended)

# Install with standard pip
pip install spatial-detector

# Or use UV for faster installation
uv pip install spatial-detector

Option 2: Install from Source

# Clone the repository
git clone https://github.com/oldhero5/spatial-detector.git
cd spatial-detector

# Install with standard pip
pip install -e .

# Or use UV for faster installation
uv pip install -e .

Quick Start

After installation, run the webcam application with default settings:

spatial-detector

This will:

Access your default webcam
Load the lightweight YOLOv8n model
Initialize the MiDaS depth estimator
Show both the main detection view and a top-down spatial map

Usage

Command Line Interface

The CLI provides a flexible interface for using Spatial Detector with various options:

# Basic usage with default webcam
spatial-detector

# Specify a different webcam (by index)
spatial-detector --camera 1

# Use a different YOLO model
spatial-detector --yolo-model yolov8s.pt

# Specify resolution
spatial-detector --width 1280 --height 720

# Force a specific compute device
spatial-detector --device cuda  # For NVIDIA GPU
spatial-detector --device mps   # For Apple Silicon
spatial-detector --device cpu   # Force CPU processing

# Record video output
spatial-detector --record output.mp4

# Load calibration files
spatial-detector --calibration camera_calibration.json --depth-calibration depth_calibration.json

# Customize room dimensions for mapping
spatial-detector --room-width 10.0 --room-depth 8.0

Full list of CLI options:

Option	Description	Default
`--camera`	Camera index	0
`--yolo-model`	YOLO model path or name	yolov8n.pt
`--confidence`	Detection confidence threshold	0.25
`--device`	Computation device (mps, cuda, cpu, auto)	auto
`--calibration`	Camera calibration file	None
`--depth-calibration`	Depth calibration file	None
`--record`	Path to save video recording	None
`--width`	Camera width	640
`--height`	Camera height	480
`--room-width`	Room width in meters	5.0
`--room-depth`	Room depth in meters	5.0

Python API

For programmatic use, Spatial Detector can be integrated into your own Python applications:

from spatial_detector.detection import YOLODetector
from spatial_detector.depth import MiDaSDepthEstimator
from spatial_detector.projection import PinholeCamera
from spatial_detector.visualization import Visualizer
from spatial_detector.mapping import SpatialMap, DepthCalibrator

import cv2
import numpy as np

# Initialize components
detector = YOLODetector(model_path="yolov8n.pt", confidence=0.25)
depth_estimator = MiDaSDepthEstimator()
camera = PinholeCamera()
visualizer = Visualizer(show_depth=True, show_labels=True)
spatial_map = SpatialMap(room_dimensions=(5.0, 5.0))
depth_calibrator = DepthCalibrator()

# Open webcam
cap = cv2.VideoCapture(0)

while cap.isOpened():
    ret, frame = cap.read()
    if not ret:
        break
    
    # Detect objects
    detections = detector.detect(frame)
    
    # Estimate depth
    depth_map, depth_norm = depth_estimator.estimate_depth(frame)
    
    # Convert to metric depth
    metric_depth_map = depth_calibrator.depth_to_meters(depth_norm)
    
    # Project to 3D space
    positions_3d = []
    for detection in detections:
        center_x, center_y = detection['center']
        normalized_depth = depth_estimator.get_depth_at_point(depth_norm, center_x, center_y)
        if normalized_depth is not None:
            metric_depth = depth_calibrator.depth_to_meters(normalized_depth)
            position_3d = camera.pixel_to_3d(center_x, center_y, metric_depth, normalized_depth=False)
            positions_3d.append(position_3d)
        else:
            positions_3d.append((0, 0, 0))
    
    # Update spatial map
    spatial_map.update(detections, positions_3d)
    
    # Visualize results
    annotated_frame = visualizer.draw_detections(frame, detections, positions_3d)
    map_viz = spatial_map.get_topdown_view(width=400, height=400)
    
    # Display
    cv2.imshow('Detection', annotated_frame)
    cv2.imshow('Map', map_viz)
    
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

Controls

When using the CLI application, the following keyboard controls are available:

Key	Function
`q`	Quit application
`d`	Toggle depth visualization
`l`	Toggle object labels
`m`	Toggle map mode (topdown/3D)
`c`	Enter/exit calibration mode
`+`/`-`	Adjust calibration distance (in calibration mode)
`space`	Set calibration point (in calibration mode)
`s`	Save depth calibration

Architecture

Spatial Detector follows a modular design with these core components:

Components

YOLODetector: Handles 2D object detection using YOLOv8
- Identifies objects in the scene
- Provides bounding boxes, class labels, and confidence scores
MiDaSDepthEstimator: Provides monocular depth estimation
- Generates a depth map from a single RGB image
- Uses MiDaS deep learning model
PinholeCamera: Implements camera projection model
- Converts 2D pixel coordinates to 3D space
- Handles camera calibration
DepthCalibrator: Calibrates depth values to real-world measurements
- Converts normalized depth to metric distances
- Provides tools for distance calibration
SpatialMap: Creates and maintains a spatial representation
- Tracks objects in 3D space
- Provides top-down visualization
Visualizer: Handles visualization and UI elements
- Draws bounding boxes and labels
- Renders depth maps and other visualizations
CLI Application: Ties everything together in an interactive interface

Data Flow

Image acquisition from webcam
Parallel processing:
- Object detection (YOLOv8)
- Depth estimation (MiDaS)
Depth calibration and conversion to metrics
3D projection of detected objects
Spatial mapping and object tracking
Visualization and user interaction

Calibration

For accurate real-world measurements, Spatial Detector includes a calibration workflow:

Depth Calibration

Enter calibration mode by pressing c
Place an object at a known distance from the camera (e.g., 1 meter)
Use +/- keys to adjust the displayed distance to match the actual distance
Align the object with the center crosshair
Press space to set the calibration point
Press s to save the calibration to a file

Calibration is saved to depth_calibration.json by default or to the file specified with --depth-calibration.

Camera Calibration (Advanced)

For even more accurate 3D positioning, you can provide a camera calibration file with intrinsic parameters:

Generate a calibration file using OpenCV's camera calibration tools

Save it as a JSON file with the following format:

{
    "fx": 1000.0,
    "fy": 1000.0,
    "cx": 640.0,
    "cy": 360.0
}

Provide the file path using the --calibration option

API Reference