WIP Add function for comprehensive capturing of all keyboard events

[SamWindell]

I was trying to get my plugin receiving all keyboard events on Windows and had to resort to the keyboard hook approach, as used by JUCE. It seems it's the only way to get reliable text input in Windows hosts like Ableton, Reaper, etc. It proved too difficult to do this outside of pugl, the functionality is too entwined with the native details, so I added the code in win.c.

I'm making this PR to begin a dialog about if this feature could be eventually added to pugl, and, if so, get direction on what you think the API should be.

On macOS we might need a similar backend eventually using NSEvent addLocalMonitorForEventsMatchingMask.

[drobilla]

Thanks. I'm not familiar with this issue or if/why it's needed, but at first glance the amount of API burden and code weight seems vaguely reasonable anyway.

Why is the filter thing needed? Since this seems to be controlling whether events are processed by other system functions, my knee-jerk is that this part feels "too entwined with the native details" and likely fragile

[SamWindell]

Thanks for taking a look, I can provide a little more information:

Pretty much every DAW on Windows blocks key events from reaching plugins (Reaper, Cubase, Ableton, etc.), and I believe that the only work around is for the plugin to install a keyboard hook. You would think that doing SetFocus() is enough, but the DAWs still steal some key events for their shortcuts such as spacebar to play transport, or R to enable repeat, etc. leaving it impossible to type anything meaning into the plugin.

The event filtering is necessary because otherwise the keyboard hook would consume all events, leaving nothing for the DAW. That's fine if you have a text input selected, but if for example you have a custom preset browser UI open and have a CTRL+F shortcut to use the find function, we certainly need to consume the 'F' key from the DAW's interference, but we shouldn't consume all key events - the user expects that spacebar still trigger the DAW to play/pause.

[drobilla]

I see. Never-ending fun, this stuff. I'm this close to abandoning the entire idea of embeddable UIs at this point.

leaving nothing for the DAW

So this mechanism allows the plugin to "steal" key events from the DAW, i.e., only the plugin will get it (not both)?

[SamWindell]

Oh yes it's all good fun!

So this mechanism allows the plugin to "steal" key events from the DAW, i.e., only the plugin will get it (not both)?

Yes, my understanding is that the hook is called before the DAW gets the events. If the plugin wants the event exclusively, it scrubs it (set it to an empty WM_USER message).

It's all rather nasty but I don't see any way around it! JUCE have been doing this for years:

https://github.com/juce-framework/JUCE/blob/501c07674e1ad693085a7e7c398f205c2677f5da/modules/juce_gui_basics/native/juce_WindowsHooks_windows.cpp#L54

https://github.com/juce-framework/JUCE/blob/501c07674e1ad693085a7e7c398f205c2677f5da/modules/juce_gui_basics/native/juce_Windowing_windows.cpp#L2125

[drobilla]

Ok. Well, if it needs doing, it needs doing. I might try to think of ways to avoid adding a new callback (generally ramming everything though a single even stream being the ideal of the API), but this sort of thing is best done after the implementation works everywhere. Do you have a good read on what the situation is on X11 and MacOS (and, eventually, Wayland)?

Is a test or demo program possible? I don't currently have a workflow going that involves building whole plugins and hosting them in whatever proprietary DAW, so it's much easier to confidently land things when there's a simple program in the repo that uses it. I'm not sure if a simple pugl "host" would replicate whatever the DAWs are doing that steals events though

[SamWindell]

I agree callbacks aren't usually a nice thing in an API. So I would be happy to see any other ideas for the design of that aspect.

I've been using this patch for a couple of weeks in production and its seems to have resolved all my keyboard related issues for my users on Windows.

Regarding the implementation of this feature on macOS, it's less of a ubiquitious problem. However, I have heard another dev talk about keys being stolen by the DAW (and therefore not reaching the plugin) when using the CMD on Logic Pro, Ableton and Reaper. And the workaround they use is this: https://github.com/Tremus/CPLUG/blob/b8b964b805fae6728203a05782d90bb4d39f6696/src/cplug_extensions/window_osx.m#L278-L291

On Linux, my impression is that nothing needs to be done, stealing doesn't occur. However I haven't done as much testing.

Testing isn't super simple. I honestly don't know what the DAW is doing exactly to prevent events from reaching the plugin. Maybe some research could reveal some results though.

Making a minimal test plugin to manually test in the DAWs would be possible, but not trival because we'd need it in at least both AU and VST3 formats on multiple OS. My experience is mostly with CLAP plugins. So off the top of my head it would be feasible to add a pugl GUI to this project and then wrap it to VST3 and AU using the clap-wrapper, although clap-wrapper is C++, not C.

I will probably get round to making the macOS implementation myself. This pull request is related to Floe, which could serve as a plugin to test this in too.

[drobilla]

Alright, thanks for the explanation. It's not mandatory or anything, just way easier to work on things with a simple case (plus it avoids people from different projects all fighting over various incompatible "works for me" solutions that work for them and only them, a common nightmare with plugin stuff).

I'll take a more serious look into how landable this is at some point hopefully soonish, but can't make any more specific timeline commitments at the moment.

[SamWindell]

Thanks, I appreciate you taking a look at this and for making pugl.

If you do make an alternative implementation of this feature, I can probably add it to Floe and release it to our beta testers to get some more widespread testing.

[drobilla]

This seems like trouble (across all implementations). I feel like it would be less sketchy if setting the flag was separate from grabbing focus, then the required setup done internally whenever focus is granted and released for whatever reason. This would work particularly nicely if PUGL_GREEDY_KEYS or some such was just a hint, and the filter function replaced with different semantics for the usual event handler (see other comments).

[SamWindell]

I think that could work and would be nicer.

[drobilla]

I don't see how this works / does anything useful if it's calling the filter function on a made-up event with no relation to actual input? This applies to everything in this block since it's conditional on what the filter function returns.

[SamWindell]

The important differentiation here is character events vs key down/up events. If the plugin GUI has a text input boxed focused then we need to steal all character events from the host. Alternatively, the plugin GUI might have no need for text input but instead wants to detect key press CTRL + F, for example.

This is made messy because with Windows you first need to translate key events into character events using TranslateMessage. We should only call TranslateMessage when the plugin actually wants char events, otherwise it's not giving the host the opportunity to do this and things could behave strangely for them. That's as I understand it at least.

Here's what Floe does with it's keyboard event filter:

static void RequestAllKeyboardEvents(AppWindow& window, bool wants_focus) {
    puglSetWantsAllKeyboardEvents(
        window.view,
        wants_focus,
        [](PuglView* view, PuglEvent const* event) -> bool {
            auto& window = *(AppWindow*)puglGetHandle(view);
            switch (event->type) {
                case PUGL_TEXT: return window.last_result.wants.text_input;
                case PUGL_KEY_PRESS:
                case PUGL_KEY_RELEASE:
                    if (window.last_result.wants.text_input) return true;
                    if (auto const key_code = RemapKeyCode(event->key.key);
                        key_code && window.last_result.wants.keyboard_keys.Get(ToInt(*key_code)))
                        return true;
                    return false;
                default:
            }
            return false;
        });
}

[drobilla]

Ignoring the above for a moment, this pattern here is what I was initially thinking about with my initial knee-jerk dislike of adding a new filter function. The current scheme is:

1. Call filter function to see if event is desired
2. If so, dispatch

I think it would be better to instead do:

1. Dispatch, and check the return value to see if the plugin consumed the event.

This would require a change in the rules for the event dispatch function so that e.g. PUGL_FAILURE is returned if the event isn't handled, but that's fine, because it would only affect code that opts in to "greedy" mode, and I think it's a good (and very common) principle for the event handler anyway.

I don't fully understand whether this is possible, though, because of the above funny business. What's the filter function you're using doing?

[SamWindell]

Yes that would work, but as I understand it, on Windows, we shouldn't be calling TranslateMessage unless we know we want the event.

[SamWindell]

The callback aspect could certainly still be avoided though. So long as the plugin has some way of communicating any mix of these requests:

• I want to greedily steal all CHAR events
• I want to greedily steal key events for X keys. (where X is one or more keys such as: F, for keyboard shortcut CTRL+F find, C, for CTRL+C copy, etc.)