Windows keyboard scan codes

[ProbablePrime]

I was having a few issues with your module and some applications mostly video games. These applications won't respond to keyboard events triggered from robot.

After tracing this down and examining several other keyboard emulators(robotjs,Java's Robot Classes, My own Handler) I've pinned this down to using virtual key codes over scan codes for windows keyboard events. Which DirectInput/XInput utilizing projects seem to listen to a lot more reliably.

I've edited the keyboard code for robot to use scan codes and then tested it with the problematic applications which now see events from Robot.

Other software designed to interface with keyboard events also tend to use scan codes Examples being FreePIE and AutoHotKey

I was going to open a PR to convert to scan codes but thought some discussion might be better to start as virtual key codes might be useful for some cases. I can't find anything on the internet to suggest this.

From AutoHotKey it looks like we might be able to send both Virtual and scan codes at the same time. I can update things to do that too.

Do you have any thoughts?

[dkrutsko]

Hi, Thanks for your bug report! Somehow I feel like I've seen that before, probably in the java.awt.robot source code. But I sorta discarded it because I thought it was legacy code. Unfortunately I don't know anything about scan codes and in my tests, never came across the problem. Perhaps somebody more knowledgeable will be able to chime in.

It might be good to know which applications and games are breaking so that I can have something to test against. This will obviously come in handy for #16 as well.

As for the fix, please feel free to open a PR. With this new knowledge I will be able to look at other technologies again and apply the improvements needed to get everything working correctly. There's no rush, if you plan to look into this problem further, take as much time as you need to get a good solution in place.

And thanks again for pointing this out!

[ProbablePrime]

Oh sure, I forgot to list them.

Pretty much any AAA game title won't see inputs however the following have been tested:

• The Division
• Grand Theft Auto V
• Magicka

Oh and the Dolphin Emulator.

I'm looking for something light weight that we can use for testing.

I'll do some additional poking around and testing but if I don't find anything I'll open a PR and move the discussion there.

[dkrutsko]

Yeah I can see where GTA V would become problematic. They have three options of input, raw input (the default) being one of them. I'm not sure if scan codes would actually fix this issue for that game. Does AutoIt or java.awt.robot work with it?

[ProbablePrime]

awt.robot works for it yes. Let me fire up gtav with my changes and see if scan codes work.

@dkrutsko GTAV works with scan codes.

const keyboard = robot.Keyboard();
setTimeout(()=> {
    keyboard.click(68);
    keyboard.click(68);
    keyboard.click(68);
    keyboard.click(68);
}, 2000)

Rotated the character right a bit :D

[dkrutsko]

Okay that's good to hear. You may want to try with multiple input modes but I suspect it'll work fine.

[ProbablePrime]

I'm not entirely sure what you mean by input modes. If you can alaborate a bit i can try some stuff out.

[dkrutsko]

In the settings of GTA V there's a way to select the "input mode" or whatever it's called. I know there's a "raw input" mode which makes the input more responsive, but you can change it to other modes.

[ProbablePrime]

Sorry for the delay here, I noticed activity on the repo and remembered I needed to follow up.

@dkrutsko The library changes I made responded to all available input modes within GTAV.

Based on this I would suggest that my changes would allow Robot to interface with pretty much everything on windows.

Shall I open a PR?

[dkrutsko]

Yes please, but only if you're finished. I don't really have a set schedule for PR integrations but I'll roll everything out in one release. Everyone will have plenty of time to test everything out though so no worries.

[p120ph37]

We should not drop virtual-key support. Virtual-keys allow keyboard-layout-independant mapping. For example, if you want to type the literal letter "Z", regardless of whether the keyboard is German or English, the virtual-key mechanism allows this directly, whereas if you did it via scancodes, you'd have to first somehow detect the keyboard layout and then send different scancodes depending on the layout. This is the reason the Windows API provides both methods, and if Robot-JS does add scancode support, it should be done without removing the existing virtual-key support, since both have valid uses.