[Feature] Ability to interact with the device screen during an Interactive (CLI) session

[thaovpnguyen]

Problem or use case

During an Interactive (CLI) session, the AI agent can send ADB shell commands via runDeviceCommand but cannot directly interact with the device UI — no tap, swipe, type, key press, or screenshot primitives exist as MCP tools. Reaching the device screen requires opening the Portal's manual session UI or writing Appium scripts, neither of which fits the conversational CLI workflow.

This blocks key use cases: guided exploration ("tap the Login button"), live remediation ("swipe past the onboarding screen"), and visual verification ("take a screenshot and describe what you see").

This needs EPBs and then EPAs:

1. EPB for adding an interactWithDevice MCP tool (tap, swipe, type, pressKey) for gesture-based interaction during a live session
2. EPB for adding a takeScreenshot MCP tool that captures the current device screen and returns an authenticated image URL
3. EPB for skill guidance describing the guided interaction loop: preview → interact → screenshot → verify

Jira: KOB-53972

Proposed solution

• Tool name: interactWithDevice

• Description: Sends UI interaction gestures (tap, swipe, type, pressKey) to the device during an active Interactive session. Reuses existing deviceControl / WebDriver interaction infrastructure.

• Input parameters: sessionId, action ("tap" | "swipe" | "type" | "pressKey"), action-specific params (coordinates, text, keyCode)

• Tool name: takeScreenshot

• Description: Captures the current device screen and returns a pre-signed image URL (plus optional inline thumbnail).

• Input parameters: sessionId

• Output: { screenshotUrl, width, height, timestamp }

Workflow steps:

1. User starts an interactive session; agent surfaces live preview URL (see companion issue)
2. User asks agent to interact: "tap the Login button at the top of the screen"
3. Agent calls interactWithDevice(sessionId, "tap", { x: 540, y: 220 })
4. Agent calls takeScreenshot(sessionId) to verify the result
5. Agent describes what changed and prompts for next action