Screen-AI — Local AI PC Operator: A Phone-Approved Desktop Agent Accelerated by AMD ROCm - #327
Open
madhavabrightly wants to merge 39 commits into
Open
Screen-AI — Local AI PC Operator: A Phone-Approved Desktop Agent Accelerated by AMD ROCm#327madhavabrightly wants to merge 39 commits into
madhavabrightly wants to merge 39 commits into
Conversation
… and compound command planner Implements the four agent-spec upgrades (all additive, existing tests stay green): - Master Cognitive Planner v1.0: browser_open first-class intent, single-pass alias resolution, take-me-to/browse navigation, semantic OCR matching in screen_tools, cognitive_plan in router responses. - OS v2.0: external_planner.py (DeepSeek-V4-Flash via AMD Radeon API, env-var key only, advisory plans), take_camera_photo intent + system.capture_photo, chat mode with local rules + external model. - Task Execution Policy: per-node pipeline/models on execution graph, verify_goal node, statuses + goal in responses. - Generic Task Decomposition: CONTACT/CHANNEL entities, extract_entities() helper, entity-driven send_file plan (replaces app-specific WhatsApp flow). - Compound Command Planner: _plan_multi_intent_sequence splits chained commands into one compound_sequence execution graph; Planner sync twins (classify_intent_sync/create_plan_sync). - Docs updated: AGENTS.md, CHANG.md, MEMORY.md, README.md, ai_pc_operator/README.md, relation-map.md. Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Screen-AI is a fully local, offline-first AI agent that operates your PC from natural-language
commands sent from your phone or desktop. It perceives the screen (Windows UI Automation +
OCR + a tiny YOLOv8n UI detector), plans a safe execution graph, and performs real actions —
opening apps, browsing, managing files, logging in, and more — while keeping every risky action
behind phone approval and a reversible quarantine-first policy. No sensitive data leaves your
machine.
The project demonstrates a complete agentic pipeline (perception, planning, tool use, memory,
verification, and safety approvals) and a teacher-student distillation workflow accelerated by
AMD ROCm: OmniParser v2 labels screenshots on an AMD Instinct MI300X GPU, a YOLOv8n student is
trained on ROCm, and the result is compressed to a 3.2 MB INT8 ONNX model that runs on low-
resource laptops. Verified on ROCm 7.14 (torch 2.10.0+rocm7.0, MI300X 192 GB): the UI detector
runs at ~6.8 ms/frame (3.1x faster than CPU), and the full cloud training/export pipeline
completes on the GPU.
Track: Track 2 — Agentic AI (AMD Radeon & ROCm hackathon)
Keywords:
local AI agent, desktop automation, screen perception, phone approval, safety-first,
teacher-student distillation, YOLO, ONNX INT8, AMD ROCm, Radeon, MI300X, offline-first