Aura - Agentic Unified Robotics Architecture

Control robots with your mind through high-level intent, not low-level commands.

Aura is a Brain-Computer Interface (BCI) system that enables intuitive brain-controlled robotic manipulation through EEG signal processing. Unlike traditional BCI approaches that require users to mentally control every axis of movement (exhausting and cognitively demanding), Aura's architecture focuses on high-level intent: simply select what you want and where you want it—the system handles the rest.

Built from the ground up by undergraduate CS students at De Anza College, University of Waterloo, and UC San Diego.

Overview

The Problem with Traditional BCI Robotics

Most robotics BCI projects require users to mentally control individual robotic components (X, Y, Z translation). While simple to implement, this approach leads to:

• High cognitive fatigue
• Slow operation
• Poor user experience

The Aura Approach: Intent-Based Control

Our core innovation: Instead of controlling individual axes, users make simple high-level selections:

• "I want to move object A to location B"

The robot figures out the rest—drastically reducing cognitive load and making brain-controlled manipulation practical.

How It Works

Aura represents the "brain" of robotic operation, translating human neural signals into precise robotic actions:

1. EEG Headset (OpenBCI Cyton) captures brain activity at 125Hz
2. Dual BCI Paradigms process two types of neural signals:
   • P300 Speller: Select target objects (82.8% accuracy)
   • Motor Imagery: Execute grasp/release commands (F1-score ~0.61)
3. Computer Vision (ArUco markers + planar homography) localizes objects in 3D space
4. Inverse Kinematics translates object positions to joint angles
5. 3-Axis Robotic Arm executes the physical manipulation

Key Features

• Real-time EEG Processing: 125Hz sampling from OpenBCI Cyton headset
• Dual BCI Paradigms:
  • Motor Imagery (MI) for grasp/release commands
  • P300 speller for object selection
• Multiprocessing Architecture: Separate processes for GUI, EEG acquisition, and ML inference
• Computer Vision Integration: ArUco marker tracking with planar homography
• Robotic Control: 3-axis arm with inverse kinematics

Performance Metrics

Component	Metric	Value
P300 Detection	Accuracy (LOSO)	82.8%
P300 Detection	Confidence Threshold	0.55-0.60
MI Classification	F1-Score	~0.61
MI Classification	Confidence Threshold	0.1
EEG Sampling	Rate	125 Hz
Epoch Length	MI	2.5s (50% overlap)
Epoch Length	P300	0.6s (post-stimulus)

Hardware Requirements

EEG Headset

• OpenBCI Cyton Board (8 channels, expandable to 16 with Daisy module)
• EEG Electrodes configured for:
  • Motor Imagery: C3, C4, Cz (motor cortex)
  • P300: Fz, Cz, Pz, P3, P4, P7, P8, Oz (visual processing)
• Sampling Rate: 125Hz

Robotic Arm

• 3-axis servo-controlled arm
• Arduino Uno controller
• Serial communication (configurable port, default 9600 baud)

Computer Vision

• USB camera (640x480 or higher)
• ArUco markers (IDs 0-5 for workspace + object tagging)
• Well-lit environment for marker detection

How It Works

1. BCI Headset & Signal Acquisition

The system uses app/eeg/acquire_eeg.py to interface with the OpenBCI Cyton headset via the pyOpenBCI library. The acquisition runs in a separate process to ensure real-time performance:

• Auto-detection: Automatically finds OpenBCI device on available serial ports
• Channel Configuration: Reads 7-8 channels at 125Hz
• Epoch Generation: Creates 2.5-second overlapping windows (50% overlap) for signal processing
• Queue-based Communication: Sends raw EEG data to ML processing pipelines

Key File: app/eeg/acquire_eeg.py

2. Motor Imagery (MI) Pipeline

Motor Imagery detects when the user imagines moving their hand to execute grasp or release commands—a challenging BCI problem that requires distinguishing subtle differences in brain activity.

Signal Processing (FBCSP)

We use a classic, robust BCI pipeline optimized for motor imagery:

1. Filter-Bank Common Spatial Patterns (FBCSP): Feature extractor that splits EEG into 8-30Hz sub-bands and maximizes variance between grasp/release
2. SelectKBest: Dimension reduction to select the most discriminative features
3. Linear Discriminant Analysis (LDA): Final classifier for binary grasp/release decisions

Performance: F1-score ≈ 0.61 in Leave-One-Subject-Out (LOSO) cross-validation. While this represents solid performance for a challenging classification problem, we're actively working to improve accuracy for more reliable everyday use.

Key Files:

• Model: models/mi_v3/README.md
• Runtime: app/models_runtime/mi_runtime.py
• Manager: app/ml_manager.py (MIManager class)

MI Detection Flow

Raw EEG (7 channels) → Bandpass 8-30Hz → FBCSP Feature Extraction
→ SelectKBest → LDA Classifier → Confidence Score (threshold 0.1)
→ Grasp/Release Command

3. P300 Event-Related Potential Pipeline

P300 detects the brain's natural response to visual stimuli—allowing users to select objects by simply focusing their attention. This is a simpler and more reliable classification problem than Motor Imagery.

Visual Stimulus

The GUI presents a grid of rectangles that flash in sequence at 0.05s intervals. When the user focuses on their target, their brain automatically generates a P300 response (~300ms after the target flashes)—no special training required!

Signal Processing

1. Epoch Extraction: 0.6s windows aligned to stimulus onset
2. Feature Engineering:
   • Temporal features (mean, std, peak amplitude)
   • Frequency domain (FFT power in alpha/beta bands)
3. Classification: RandomForest classifier
4. Confidence Thresholding: 0.55-0.60 for reliable detection

Performance: 82.8% test accuracy (LOSO cross-validation)—solid performance for real-time object selection!

Key Files:

• Model: models/p300_v1/README.md
• Runtime: app/models_runtime/p300_runtime.py
• Manager: app/ml_manager.py (P300Manager class)

P300 Detection Flow

Raw EEG (8 channels) → Stimulus Timestamp → 0.6s Epoch Extraction
→ Feature Engineering → RandomForest → Confidence Score (threshold 0.55-0.60)
→ Selected Object

4. Application Architecture

The system is built on a robust Python multiprocessing architecture designed to keep everything responsive and low-latency.

We run four separate processes in parallel:

1. GUI Process (Tkinter): User interface and application logic
2. OpenBCI Acquisition Process: Raw EEG data streaming at 125Hz
3. MI Processor: Heavy-duty Motor Imagery ML calculations
4. P300 Processor: P300 event-related potential ML calculations

Why multiprocessing? By separating the GUI, acquisition, and ML inference into independent processes, we ensure:

• Real-time responsiveness (no UI blocking during ML calculations)
• Parallel processing power (all cores working simultaneously)
• Modular design (each component can be developed/tested independently)

Communication: Processes exchange data via multiprocessing.Queue for thread-safe, decoupled producer/consumer patterns.

Key File: app/main.py (BCIMultiprocessingManager orchestrates all processes and queues)

5. Robot Control & Computer Vision

Now for the physical side: a low-cost, custom-designed 3-axis robotic arm built specifically for desktop robotics research.

ArUco Marker Localization

ArUco markers are specially designed QR codes that computers can easily detect and localize in camera images. Here's our clever approach:

1. Workspace Calibration: Place 4 ArUco markers at known corner positions on the workspace
2. Planar Homography: Use these corner markers to transform pixel coordinates → real-world (X, Y, Z) coordinates
3. Object Tracking: Each graspable object has its own unique ArUco marker for precise localization
4. Deposit Zones: Specialized zones are also marked for accurate drop-off targeting

The result? The system can accurately track objects and their destinations in 3D space—all from a simple overhead camera!

Inverse Kinematics

Once we know what to pick and where it needs to go, we use inverse kinematics to calculate the individual joint angles:

• IKPy Library: Solves joint angles for any target (x, y, z) position
• 3-Axis Control: Base rotation, shoulder, and elbow servos
• Serial Commands: Arduino Uno receives angle commands via USB and drives the servos

Control Flow

P300 Selection → ArUco Marker Detection → Planar Homography Transform
→ Inverse Kinematics → Servo Commands → Physical Movement

Lighting Note: Ensure good lighting without harsh shadows—shadows can interfere with marker detection!

Key Files:

• Arm Control: app/arm.py
• Marker Tracking: app/robot_arm/src/markerTracking/
• Robot Control: app/robot_arm/src/robotControl/

Getting Started

Installation

# Clone the repository
git clone <repository-url>
cd aura

# Install dependencies (Python 3.11+)
uv pip install -e .

Hardware Setup

1. OpenBCI Headset:

   • Place electrodes according to 10-20 system
   • MI channels: C3, C4, Cz, FC3, FC4, CP3, CP4
   • P300 channels: Fz, Cz, Pz, P3, P4, P7, P8, Oz
   • Connect via USB dongle

2. Robotic Arm:

   • Upload Arduino sketch from app/robot_arm/Arduino Uno/
   • Connect via USB serial
   • Configure port in app/config.py

3. Camera:

   • Position camera above workspace
   • Print ArUco markers (IDs 0-5)
   • Ensure good lighting for marker detection

Running the Application

# Start the BCI GUI
python app/main.py

Configuration

Edit app/config.py to customize:

• Serial ports for OpenBCI and Arduino
• Confidence thresholds for MI/P300
• Camera resolution and settings
• Model paths

Usage Notes

• Lighting: Ensure plentiful light on markers; harsh shadows interfere with detection
• Marker Visibility: Object markers must face the camera (both QR codes visible)
• P300 Focus: Concentrate on target rectangle during flashing sequence
• MI Calibration: May require per-user retraining for better accuracy

Project Structure

aura/
├── app/                    # Main application
│   ├── main.py             # GUI and multiprocessing manager
│   ├── ml_manager.py       # P300/MI manager classes
│   ├── eeg/                # EEG acquisition
│   ├── models_runtime/     # Runtime inference code
│   └── robot_arm/          # Arduino, CV, robot control
├── models/                 # Training code and saved models
│   ├── mi_v3/              # Motor Imagery (FBCSP+LDA)
│   ├── p300_v1/            # P300 (RandomForest)
│   └── other models
├── data/                   # Training datasets
└── tests/                  # Integration tests

Development

Testing

# Run all tests
python run_tests.py

# Run specific test suites
pytest tests/test_p300_integration.py
pytest tests/test_mi_integration.py

Model Training

See individual model READMEs:

• Motor Imagery: models/mi_v3/README.md
• P300: models/p300_v1/README.md

Limitations & Future Work

Current Limitations

• MI Model Accuracy: Current F1-score (~0.61) is solid but needs improvement for reliable everyday use
• App-Robot Interface: Communication protocol between BCI application and Arduino could be simplified
• Shadow Interference: ArUco detection can fail with partial shadows—requires good, even lighting
• P300 Latency: 0.05s stimulus intervals may be too fast for some users

Planned Improvements

1. Enhanced MI Performance:

   • Per-user calibration and transfer learning
   • Deep learning approaches (EEGNet, ShallowConvNet)
   • Additional feature engineering

2. System Reliability:

   • Simplified robot communication protocol
   • Improved lighting robustness for marker detection
   • Adjustable P300 stimulus timing

Contributors

Koichi Nakayamada

• De Anza College, AS Math for Transfer
• President, De Anza Human Computer Interaction
• Role: Aura Project Lead and Initiator, Application and pipeline architecture, ML pipeline design & integration, MI model development, EEG acquisition, Data collection, Testing, Documentation

Alka Ukrani

• University of Waterloo, BASc Biomedical Engineering (Co-op)
• External member, De Anza Human Computer Interaction
• Role: Aura Project Initiator, P300 model development, Pipeline bug fixing, ML

Ayan Syed

• De Anza College, AS Computer Science for Transfer
• Member, De Anza Human Computer Interaction
• Role: Robotic arm & computer vision hardware/software, arm control integration, testing and fine-tuning

Begench Gurbanov

• UC San Diego, BS Computer Science
• External Member, De Anza Human Computer Interaction
• Role: OpenBCI Cyton headset & Daisy module renting, Initial project discussions

---

License

This project is licensed under the MIT License.