Skip to content

Add webcam/camera support and self image capture plugin for Synth #298

@XargonWan

Description

@XargonWan

Summary

Add a new non-LLM plugin to let Synthetic Heart interact with the scene camera, manage webcam state, and capture images from the virtual scene. This includes support for selfies, screenshots, photo sending, and a selfie-oriented animation pose state.

Motivation

Synth already has a digital body and animation system. Enabling real webcam/camera interactions closes the gap between virtual persona and scene-aware media actions, allowing:

  • turning the webcam on/off
  • capturing photos or screenshots
  • sending image attachments
  • triggering selfie poses before capture

Proposed Changes

  1. New webcam/camera plugin

    • Create plugins/webcam_plugin.py (or similar)
    • Implement get_supported_actions() for actions like:
      • enable_webcam
      • disable_webcam
      • take_screenshot
      • take_selfie
      • send_photo
      • set_camera_pose
  2. Session metadata persistence

    • Reuse session_meta.py for camera state storage
    • Persist webcam on/off state, current camera rect, and scene focus per interface_path
    • Ensure archive endpoints include camera metadata on save
  3. Animation support

    • Extend animation_handler.py to support selfie/camera pose states
    • Register a new logical animation state or alias for selfie pose
    • Keep state-driven animation selection compatible with the existing handler
  4. Action validation

    • Ensure the new actions are discovered by the auto-registration process in component_auto_registration.py
    • Add required field definitions for validation
  5. WebUI / interface integration

    • Wire new camera metadata into WebUI session save/load flows
    • Optionally expose camera commands in the UI once backend support exists

Acceptance Criteria

  • New webcam plugin exposes camera-related actions via get_supported_actions()
  • Core validation accepts the new action types
  • Camera/webcam state is stored and merged correctly through session_meta.py
  • Archiving preserves camera metadata with session archives
  • Selfie pose can be triggered through the animation state system
  • Existing plugin registration flow automatically discovers the new plugin
  • Add unit tests for action validation and session metadata flow

Notes

  • This PR should remain focused on backend/plugin integration and metadata plumbing.
  • Full WebUI controls for webcam toggling and photo capture can be scoped to follow-up work.
  • Selfie pose support should use logical animation states, not hardcoded file paths.

Relevant files

  • session_meta.py
  • webui.py
  • animation_handler.py
  • component_auto_registration.py
  • plugin_base.py
  • plugins (new module)

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    Status

    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions