Better keyboard input passthrough on mobile

[romanZupancic]

Hi!

I'm new to this project, so apologies if these changes have already been previously litigated/there is some use-case I haven't considered that would be impacted by these changes.

I've tested this on a windows machine hosting the application, connecting through Safari on iOS.

Presently, the only way to send keyboard input to the host machine on a mobile device is to use the keyboard scene. The process for sending keystrokes is as follows:

1. Connect to the webserver through your mobile browser
2. Select the keyboard scene
3. Type an input
4. Dismiss the mobile keyboard
5. Press send
6. (if you want to enter more text) go to step 2

This scene is (to me) a little clunky to use for a few reasons:

1. Steps 2-6 are a lot if you have a lot of text you want to type incrementally
2. Keystrokes are entered directly into a text area, so you have to take your eyes off the remote screen to preview what you are typing
3. Keystrokes are buffered into a text area, so you don't get the benefits of auto-complete if you are entering text into an auto-completing field on the remote machine
4. The "send" button is covered by the mobile keyboard (at least on Safari on iOS), so the mobile keyboard has to be manually dismissed before the send button can be pressed
5. The touchpad input is no longer available on this screen.

This pull request fixes all 4 of these issues.

1. Typing and sending keystrokes are done simultaneously; touchpad is still available; keyboard never has to be dismissed manually
2. Keystrokes are now sent to the remote machine as they are pressed, nothing is being buffered on the mobile screen.
3. Since keystrokes are directly sent, you benefit from auto-complete on the remote machine
4. The UI elements at the bottom of the screen now snap to the top of the mobile keyboard; mobile keyboard no longer has to be dismissed
5. The touchpad is now available on this screen.

The new system works much better for me.

Please let me know if any of this needs clarification/you'd like some changes to be made!

[ams1]

@romanZupancic, I've also done something like this in a local version.

IMO there should be button that activates/deactivates forwarding of each keypress.

ex. when I use the ios dictation feature (which types way faster in certain scenarios and which triggers an input event, NOT key press), sometimes I need to correct what dictation did by pressing keys on the virtual keyboard - and if forwarding is always on, those keypresses used to correct a word in the input will get sent.

Try using dictation/swipe typing on that input and then using the cursor try to change a word with the virtual keyboard and you'll realize what I mean.

There are many features I would suggest for this wonderful ❗ app - and be also willing to contribute - but not sure if it's in the focus of @Unrud anymore 🤗

[romanZupancic]

@ams1
Hmmm never thought of text dictation, that's a pretty neat use-case!

I'll try and work on a toggle if I can scrap together the time.

I might also experiment with placing my hybrid keyboard/touch scene to be the main scene when you first access the web page (in place of the current mouse-only scene)... maybe hide the input field when on a desktop browser...

And then we can just use the old textarea input scene for cases like yours.

At any rate, even if changes aren't merged, we can still use our own forks!

[Unrud]

I think this has the potential to cause buggy/unexpected behavior (@ams1 mentioned one example).
To improve the situation I implemented f11cefc, 88fd6e7 and 6a81260.
In the end, this is a limitation of the web platform. The VirtualKeyboard API is only available via HTTPS (and not supported by Safari anyway).