Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
120 lines (101 loc) · 6.45 KB

ARCHITECTURE.md

File metadata and controls

120 lines (101 loc) · 6.45 KB
Raw

Architecture

System Overview

┌─────────────────────────────────────────────────────────────┐
│                     React Frontend                           │
│  ┌────────────────────────┬──────────────────────────────┐  │
│  │   BrowserView (2/3)    │   ChatPanel (1/3)            │  │
│  │   - Live screenshots   │   - Message history          │  │
│  │   - URL bar            │   - Input box                │  │
│  │   - Bounding box       │   - Model selector           │  │
│  │     overlay             │   - API key / system modals  │  │
│  └────────────────────────┴──────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘
                              │
                    WebSocket (ws://localhost:8000/ws)
                              │
┌─────────────────────────────────────────────────────────────┐
│                  FastAPI Backend                              │
│                                                              │
│  main.py (WebSocket handler)                                 │
│    ├── Receives user messages                                │
│    ├── Captures screenshot via BrowserManager                │
│    ├── Sends screenshot + message to OpenRouterClient        │
│    ├── Processes AI response through bounding box extractor  │
│    ├── Generates JS click code from coordinates              │
│    ├── Executes JS in browser via BrowserManager             │
│    └── Sends results + updated screenshot to frontend        │
│                                                              │
│  browser_manager.py                                          │
│    ├── Playwright headless Chromium lifecycle                 │
│    ├── Screenshot capture (base64 PNG)                       │
│    ├── Live screenshot streaming (2s interval)               │
│    ├── JavaScript execution via page.evaluate()              │
│    └── URL navigation                                        │
│                                                              │
│  openrouter_client.py                                        │
│    ├── OpenAI SDK → OpenRouter API (base_url override)       │
│    ├── Vision model: screenshot + user message → bbox JSON   │
│    ├── Bbox processor model: raw response → clean coords     │
│    └── Fallback chat method (no screenshot)                  │
└─────────────────────────────────────────────────────────────┘

Data Flow

Chat Message Flow

  1. User types message in ChatPanel
  2. App.js sends { type: "chat_message", content: "..." } via WebSocket
  3. Backend captures screenshot from Playwright
  4. Screenshot (base64) + user message sent to vision model via OpenRouter
  5. Vision model returns JSON with text and bounding_box (x, y, width, height)
  6. Response passed to bounding box processor model (alibaba/tongyi-deepresearch-30b-a3b:free) for coordinate cleaning
  7. If valid coordinates found: backend generates JS that calls document.elementFromPoint(centerX, centerY).click()
  8. JS executed in browser via page.evaluate()
  9. After 1s delay, new screenshot captured and sent to frontend
  10. AI text response + execution result sent to frontend

WebSocket Message Types

Client → Server:

  • chat_message — user's natural language request
  • navigate — URL navigation (from address bar)
  • screenshot_request — manual screenshot refresh
  • set_api_key — set OpenRouter API key (saved to config.json)
  • set_system_message — override default system prompt
  • set_model_name — switch AI model

Server → Client:

  • screenshot — base64 PNG screenshot (on-demand)
  • live_screenshot — base64 PNG from streaming loop
  • ai_response — AI text response + optional screenshot
  • execution_result — JS execution success/failure
  • message_received — acknowledgment of user message
  • api_key_status — API key set confirmation + model name
  • error — error message
  • success — success message

AI Pipeline

The system uses a two-model pipeline:

Model 1 (Vision): Configurable via UI, default nvidia/nemotron-nano-12b-v2-vl:free

  • Receives: screenshot (base64 image) + user message
  • Returns: JSON with text explanation and bounding_box coordinates
  • System prompt instructs it to return coordinates, NOT JavaScript

Model 2 (Bbox Processor): Hardcoded alibaba/tongyi-deepresearch-30b-a3b:free

  • Receives: raw text from Model 1
  • Returns: cleaned { x, y, width, height } JSON
  • Handles format conversion (e.g., bbox_2d [x1,y1,x2,y2] → standard format)

JavaScript Generation: Done in main.py, not by the AI

  • Calculates center of bounding box
  • Generates document.elementFromPoint() + .click() code
  • Includes fallback: expanding circle search for nearby clickable elements
  • Includes fallback: traverse parent elements for clickable ancestors

Configuration Storage

  • API key and model name saved to backend/config.json (plaintext)
  • Loaded on startup, UI changes persist across restarts
  • .env file OPENROUTER_API_KEY used as fallback if no saved config

Security Considerations

  • No authentication — single-user assumption
  • API key stored in plaintext on disk
  • CORS restricted to http://localhost:3000
  • AI-generated JavaScript executed without sandboxing
  • bypass_csp=True and --disable-web-security on the browser

Scaling Limitations

  • Single browser instance shared across all WebSocket connections
  • In-memory chat history (lost on restart)
  • Synchronous AI calls per WebSocket message (no queuing)
  • Live screenshot streaming to all connected clients