[Bug]: java.awt.Robot / xdotool input events silently dropped on pure Wayland

[julienmerconsulting]

Spun out from #299 — focused on a single observable defect so it can be tracked cleanly.

Summary

On pure Wayland sessions (no XWayland fallback), java.awt.Robot.mousePress() / keyPress() and xdotool key/click complete without error, but no input event ever reaches the focused application. Image detection and coordinate resolution remain correct — only the synthesized action is dropped.

Steps to reproduce

1. Run a Wayland-strict GNOME (or any pure Wayland) session, with XWayland disabled or unavailable
2. Launch any OculiX script that uses Robot input, e.g. s.click(some_match) or s.type("hello")
3. Observe: the match is found, coordinates are correctly identified and printed
4. Observe: no input event reaches the target window — nothing happens on screen
5. Observe: no exception is raised, no error printed to console or logs

Expected behavior

After a successful find, the click or keystroke is delivered to the focused application, producing the documented downstream effect (button press, focus change, character insertion).

Actual behavior

The script completes silently. Coordinates are valid. No error appears in logs. Nothing happens on screen.

This matches the documented upstream limitation tracked in OpenJDK Project Wakefield (JDK-8280983): java.awt.Robot cannot emulate keyboard or mouse events under pure Wayland, and a short-term resolution is not expected.

Minimal reproducer (script)

from sikuli import *
# Any anchor image visible on screen will do
m = find("anchor.png")
if m:
    print("Match at:", m.getX(), m.getY())   # correct coordinates printed
    m.click()                                  # no visible action on pure Wayland

Operating system

Other — to be confirmed by the original reporter. cc @uwekoenig — could you confirm the exact distribution and version (Ubuntu 24.04 LTS? Fedora 40 or 41? something else), and the session type reported by echo $XDG_SESSION_TYPE from a terminal inside the affected session?

Java version

To be confirmed (java --version output from the affected machine).

OculiX version / artifact

3.0.3 stable (current release line, per the original report in #299).

Where does the bug happen?

• API (Screen / Region / Pattern / Match)
• Script runners (Jython, JRuby, Python, PowerShell, Robot)

Logs / console output

(no error — input action silently dropped, no stdout, no stderr)

Additional context

This ticket is a focused fork of #299, which covered a broader set of Wayland-related concerns (screenshot unsafe-mode, xdotool, Robot). The screenshot path has been found to work without unsafe mode, and the off-topic discussion has been parked. What remains, and what this ticket tracks, is the single defect that actually blocks Wayland-strict users: the synthesized input layer.

The two paths OculiX uses on Linux are:

Layer	Used for	Behavior on pure Wayland
java.awt.Robot (in-JVM)	mouse + keyboard synthesis	silently dropped — see JDK-8280983
xdotool (external)	window management + input fallback	silently dropped — no X server to talk to

Both share the same root cause: on pure Wayland there is no neutral system call that lets an unprivileged process synthesize input outside its own surface. The compositor owns the input pipeline, and the only sanctioned channels are:

• xdg-desktop-portal RemoteDesktop interface — requires a one-time user consent popup, then session-scoped
• libei / libeis — emerging standard, GNOME 46+ / KDE 6.x
• compositor-specific protocols such as virtual-keyboard-unstable-v1 and wlr-virtual-pointer-unstable-v1 (wlroots only)

The position taken here is that this is not a wait-for-upstream situation. JDK-8280983 has been open long enough, with no short-term resolution in sight, that it is reasonable for OculiX to ship its own answer rather than route around the problem with documentation. The plan is a small native shim (liboculix_wayland_robot.so) that bridges the JVM directly to the sanctioned Wayland input channels — libei first for modern compositors, xdg-desktop-portal RemoteDesktop as the portable fallback. Surface-side, that ships as a LinuxRobotPortal driver that mirrors the existing Robot interface so scripts do not change. Design and scope land in their own ticket once the probe code is validated against GNOME 46+, KDE 6, and a wlroots compositor. In the meantime, XWayland fallback remains the recommended workaround on distributions that still ship it.

If the shim lands, the benefit is not limited to OculiX. Every Java project that synthesizes input through java.awt.Robot — accessibility tools, headless test rigs, remote-control utilities, every Sikuli-derived stack — has been blocked on the same upstream wall for years. A clean, distributable native bridge that restores Robot semantics on Wayland would land as a meaningful piece of plumbing for the wider JVM ecosystem, not just for this tool.

The point of this ticket is to give the symptom — silent no-op of Robot/xdotool input on pure Wayland — its own home, so users hitting it land here directly instead of in the broader #299 thread, and to make the chosen direction explicit: bypass, not wait.

🦎