OmniView

Problem Statement

In today's digital age, we consume and interact with an overwhelming amount of information across various applications on our mobile devices. However, retrieving specific details—like a fleeting conversation, a passing product recommendation, or a crucial piece of text seen in an app—is incredibly difficult. Traditional screenshot tools only capture static images without searchable context, and cloud-based AI solutions pose severe privacy risks by uploading sensitive personal screen data to external servers.

Our Solution

OmniView is an advanced, fully on-device Android screen intelligence system that acts as a continuous, semantic memory of your digital life. It solves the retrieval problem by capturing, embedding, and semantically storing your screen history locally. This enables you to interact and chat with your past context using local or optional external large language models — all with a privacy-first approach, ensuring your sensitive data remains under your complete control.

---

Architecture Diagram

graph TD
    subgraph Ingestion Module
        A[Accessibility Event] --> B[Blacklist Manager]
        C[MediaProjection] --> B
        B -->|Allowed| D[Screenshot & Text Capture]
        B -->|Blocked| Z[Stop Capture]
        D --> E{pHash Deduplication}
        E -->|No Change| Z
        E -->|Changed| F[Compress Image to WebP]
        E -->|Changed| G[Save Text to WAL]
    end

    subgraph Storage Module
        F --> H[(Local SQLite Database)]
        G --> H
    end

    subgraph Intelligence Module
        I[Device Charging & Battery > 80%] --> J[WorkManagerScheduler]
        J -->|Batch Process| H
        H --> K[Context Extraction / ML Kit OCR]
        K --> L[Vector Engine / ONNX Runtime]
        L -->|Embeddings| H
    end

    subgraph Search & RAG Module
        M[User Query] --> N[Search Engine]
        N -->|Keyword & Semantic Search| H
        N --> O[Top-K Results]
        O --> P[RAG Orchestrator]
        P --> Q{External LLM Opt-in?}
        Q -->|Yes| R[External LLM Service]
        Q -->|No| S[Local Quantized Model]
        R --> T[Enriched Response]
        S --> T
    end

    subgraph UI Module
        T --> U[Search View / Timeline Visualisation]
    end

Loading

---

Key Features

Intelligent Screen Ingestion

• Continuous Capture: Uses a foreground MediaProjection service to capture screen states efficiently.
• Sensitive Application Blacklisting: Automatically detects the active app and halts all capture and processing if a sensitive app (e.g., banking, password managers) is in the foreground. Users can manage this blacklist.
• Real-time Accessibility Extraction: Leverages Android's Accessibility Services to extract UI text natively as you navigate.
• pHash Deduplication & Optimised Storage: Computes average-hash fingerprints of screenshots to discard visually identical frames. Unique frames are compressed to WebP and text is saved to a Write-Ahead Log (WAL) before writing to the local database, ensuring high performance.

On-Device Intelligence Pipeline

• Battery-Aware Batch Processing: Heavy ML tasks (OCR and Vector Embeddings) are strictly deferred to Android WorkManager queues. The system only processes pending data when the device is charging and battery is above 80%, guaranteeing minimal impact on battery life.
• Context Extraction: Employs Google's ML Kit Text Recognition for non-native text and image OCR.
• Vector Embeddings (ONNX Runtime): Extracted context is tokenised and embedded into high-dimensional vectors locally using a Vector Engine powered by ONNX Runtime and local models.

Search, Retrieval, and RAG

• Hybrid Search Engine: Allows users to find past memories using both exact keyword matching and natural language semantic search (Cosine Similarity).
• Retrieval-Augmented Generation (RAG): Employs an orchestrator to generate enriched responses based on retrieved context.
• Optional External LLMs: While the core system and semantic search function entirely offline (using a local Quantized Model), users can explicitly opt-in to use External LLM Services for more advanced RAG-based query responses.

Timeline Visualisation & Statistics

• Chronological Timeline: The UI provides a visual, chronological timeline of all captured user activity.
• Usage Statistics: Aggregates data based on app category and time spent, providing graphical summaries and charts of your digital habits.

Privacy-First Storage

• Room Database (SQLCipher): All context, OCR data, and embeddings are stored securely on the device, protected by encryption schemas.
• Zero Cloud (Unless Opted-In): Core functionalities involve no external telemetry.
• Data Control: Features on-demand capture stoppage and the ability to instantly delete history for specific time ranges or entirely.

---

Software Architecture

The application is modularised into distinct packages:

1. ingestion: Contains ScreenshotService, ScreenAccessibilityService, BlacklistManager, and SnapshotManager. Handles raw pixel data and text capture while enforcing privacy limits.
2. intelligence: Houses the VectorEngine, ContextProcessor, and WorkManagerScheduler. Responsible for batch-processing embeddings when device charging conditions are met.
3. storage: Encapsulates IngestionRepository, HistoryManager, and local database entities (SQLite/Room).
4. search: The SearchRepository and SearchEngine for parsing queries and performing semantic/keyword retrieval.
5. ui: Jetpack Compose frontend containing the SearchViewModel, timeline visualization, charts, and settings.
6. dashboard: A React/Vite-based promotional web landing page that showcases the product features.

---

Requirements & Setup

Device Requirements

• Android 7.0 (API 24) or higher (Android 14+ recommended for full MediaProjection stability)
• Minimum 4GB RAM
• Required Permissions: FOREGROUND_SERVICE, POST_NOTIFICATIONS, PACKAGE_USAGE_STATS, and Accessibility

Running the App

1. Clone the repository and open the project in Android Studio.
2. Build and install the APK onto a physical device (emulators may struggle with local LLM inference).
3. Open the app and grant the required permissions via the Settings Drawer (Usage Access, Accessibility, Notifications).
4. Configure your Sensitive Applications Blacklist.
5. Tap Start Screenshot Service to begin building your memory.
6. Provide the LLM model (e.g., gemma-3-1b-it-q4_k_m.gguf) in the app's internal files directory via ADB (if using local RAG):

   adb push model.gguf /data/data/com.omniview.app.ui/files/