Skip to content

REMOTE-CONTROL.md

Latest commit

 

History

History
248 lines (191 loc) · 11.2 KB

REMOTE-CONTROL.md

File metadata and controls

248 lines (191 loc) · 11.2 KB
Raw

PairUX Remote Control

How "let this participant drive my desktop" actually works, end to end.

The OS-level injection layer lives in a standalone package, @profullstack/remote-input — see its README for backends, per-OS requirements, and the safety guarantees. This document covers how PairUX drives it.

Only the desktop app can be controlled. A web host shares a browser tab and has no OS to inject into, so web-hosted sessions leave control off (apps/web/src/app/host/page.tsx) rather than showing viewers a button that could never work.

The two gates

Control is never implicit. Two independent things must be true:

  1. The session allows requests — session.settings.allowControl. Set at session creation from the desktop's Settings toggle (getDefaultAllowGuestControl() in apps/desktop/src/renderer/lib/sessionDefaults.ts). When false, the viewer never renders a "Request control" button, and the host hooks drop any control-request or input message that arrives anyway.
  2. The host approved that participant — the participant's control_state must be granted in the database. The host-side injector is enabled from that row, so approval is what actually opens the input path.

Gate 1 is a preference. Gate 2 is the security boundary: nothing is injected until a human clicks Allow control.

Flow

sequenceDiagram
    participant V as Viewer (web)
    participant T as Transport (SFU or P2P)
    participant H as Desktop host UI
    participant API as Control API
    participant OS as OS input

    V->>T: control-request
    T->>H: onControlRequest(viewerId)
    H->>H: Show Allow / Deny prompt (30s expiry)
    H->>API: PATCH control_state = granted
    API->>V: control-grant (LiveKit signal, SFU)
    H->>H: participantWithControl set -> injector enabled
    V->>T: input events (mouse / keyboard)
    T->>H: onInputReceived
    H->>OS: injectEvent -> validated, rate-limited, injected
    Note over H,OS: Ctrl+Shift+Esc revokes instantly at any point
Loading

Key files

Concern Location
Viewer input capture apps/web/src/hooks/useRemoteControl.ts
Viewer control UI apps/web/src/components/control/
Viewer transport (SFU / P2P) apps/web/src/hooks/useWebRTCSFU.ts, useWebRTC.ts
Host transport + gating apps/desktop/src/renderer/hooks/useWebRTCHostSFUAPI.ts, useWebRTCHostAPI.ts
Host prompt, grant/revoke apps/desktop/src/renderer/components/capture/CapturePreview.tsx
Host injection hook apps/desktop/src/renderer/hooks/useInputInjection.ts
Main-process facade apps/desktop/src/main/input/injector.ts
IPC + emergency hotkey apps/desktop/src/main/ipc/input.ts
OS injection + safety packages/remote-input/
Control state persistence apps/web/src/app/api/sessions/[sessionId]/participants/[participantId]/control/route.ts

Transports

A session is either sfu (LiveKit, the default) or p2p. Control works on both, over the same message shapes:

  • SFU — messages ride LiveKit data packets (publishData / RoomEvent.DataReceived). Grants are additionally pushed server-side from the control API via RoomServiceClient.sendData, so a grant lands even if the host's own data channel is momentarily unhealthy.
  • P2P — messages ride an RTCDataChannel named control.

Both host hooks expose the same surface (onControlRequest, onInputReceived, grantControl, revokeControl), and CapturePreview picks one on session.mode, so the control code above the transport is identical.

Messages

// packages/shared-types/src/signaling.ts and input.ts
type ControlMessage =
  | { type: 'control-request'; participantId: string; timestamp: number }
  | { type: 'control-grant'; participantId: string; timestamp: number }
  | { type: 'control-revoke'; participantId: string; timestamp: number };

interface InputMessage {
  type: 'input';
  timestamp: number;
  sequence: number; // ordering
  event: InputEvent; // mouse or keyboard, coordinates normalized 0-1
}

Cursor positions travel as a separate cursor message, sent lossily (~60fps) because a dropped cursor frame is irrelevant a frame later, while a dropped click is not.

Requests and approval

Guests are anonymous, so a viewer cannot write its own control_state — the control API requires an authenticated user. Requests therefore arrive over the data channel and are held in host-local state until answered:

  • The host sees an Allow control / Deny prompt naming the participant.
  • Unanswered requests expire after 30s (CONTROL_REQUEST_TIMEOUT_MS).
  • Allow resolves the viewer id back to its participant row and PATCHes control_state = granted; the injector enables off that row.
  • Deny sends control-revoke so the viewer leaves its "requested" state.
  • A viewer disconnecting clears any pending request.

Only one participant holds control at a time: granting to a second viewer revokes the first.

Two cursors

Both people keep a working cursor at the same time. Nobody hands control over, and the host's pointer is never taken away.

Every OS we support has exactly one system pointer, and none lets an ordinary process create a second (X11's XInput2 MPX aside, which exists on neither Wayland nor macOS). So the second cursor comes from not spending the real one on movement:

  • remote movement only advances a tracked position, drawn by the host as the participant's cursor (RemoteCursorsContainer, fed by onCursorUpdate)
  • the real pointer is borrowed for the instant a remote click or scroll has to land, then returned to where its owner left it
  • during a drag it stays with the remote user until they release, since restoring mid-drag would tear the drag apart

This lives in @profullstack/remote-input (virtualCursor, on by default), so every host gets it without app-side logic.

Restoring the pointer means reading where it is, which is platform-dependent:

Host Restores the local pointer
macOS Yes
Windows Yes
Linux / X11 Yes
Linux / Wayland (KDE) Yes, via the KWin helper below
Linux / Wayland (other) No — the click leaves the pointer where it landed

The KWin helper (Wayland)

Wayland refuses to tell a client where the pointer is, so on KDE the compositor is asked instead. A KWin script can only talk outward over DBus, and the bus rejects calls to a name nobody owns, so PairUX claims org.profullstack.RemoteInput, exposes SetCursorPos, and installs plus loads a script that pushes workspace.cursorPos to it. Distance-throttled, because the signal fires on every motion event.

Automatic — nothing for the user to install beyond gdbus (libglib2.0-bin), which desktops already have. Readings older than two seconds are discarded rather than used, and any failure falls back to leaving the pointer where the click landed. The script is unloaded on quit so it cannot outlive the app pushing at a dead name.

Look for one of these in the host's terminal:

[RemoteInput] KWin cursor reporting active
[RemoteInput] Cursor reporting off: KWin would not load the helper (<reason>)

GNOME's equivalent (global.get_pointer() via a Shell extension) is not implemented yet.

Coordinates

Mouse coordinates are normalized 0-1 relative to the shared surface, never pixels — the viewer does not know the host's resolution, DPI, or monitor layout. The host maps them using the captured video track's dimensions (inputScreenSize), pushed down via input:updateScreenSize whenever the capture source changes.

Stopping control

Mechanism Effect
Ctrl+Shift+Esc Global hotkey. Releases every held key/button and disables injection immediately, regardless of UI state.
Host revokes PATCHes control_state = view-only; injector disables on the next session refresh.
Viewer releases Sends control-revoke.
Viewer disconnects Host clears that viewer's control state.

Anything the remote user is holding is always released — on revoke, on disconnect, when they alt-tab or leave the video, and by a five-second watchdog if input simply stops mid-hold. A button left down would otherwise put the host's desktop into a permanent drag that swallows every click, recoverable only by rebooting.

The emergency hotkey is registered for the app's lifetime (apps/desktop/src/main/ipc/input.ts) so it works even if the renderer is wedged. Host input is never blocked while a viewer has control — the host can always take over by simply using their own mouse and keyboard.

Per-OS notes

Full requirements are in the package README. In short:

  • macOS — needs Accessibility permission; the app prompts via systemPreferences.isTrustedAccessibilityClient(true) (apps/desktop/src/main/permissions/index.ts).
  • Windows — works out of the box; only elevated target windows need the app elevated too.
  • Linux/X11 — works out of the box.
  • Linux/Wayland — needs ydotool plus a running ydotoold with /dev/uinput access. The backend attempts a systemd auto-start and, if control still is not possible, CapturePreview shows an actionable banner built from getInjectionDiagnostics() instead of failing silently.

Native modules are unpacked from the asar (asarUnpack: '**/*.node' in electron-builder.yml), which packaged builds require.

Testing

  • packages/remote-input — safety layer, injector gating, backend selection, and both Wayland backends.
  • apps/desktop/src/main/input/injector.test.ts — the full injection path against a mocked nut.js.
  • apps/desktop/src/renderer/hooks/useWebRTCHostSFUAPI.test.ts — allowControl enforcement (requests and input dropped when the session disallows control).
  • apps/desktop/src/renderer/lib/sessionDefaults.test.ts — the session-creation default that decides whether the viewer ever sees a request button.