Stub binary doesn't support `sys.executable` spawn/multiprocessing

[wtfuzz]

Describe the bug
When multiprocessing is used to fork a subprocess, the main program entrypoint is executed in the subprocess rather than the Process target.

To Reproduce
create a multiprocessing.Process() and start it using .start() The subprocess will call the parent process entry point.

Expected behavior
The target function is called as the entrypoint of the subprocess

Screenshots
If applicable, add screenshots to help explain your problem.

Environment:

• Operating System: macOS
• Python version: 3.10

Additional context
Add any other context about the problem here.

[wtfuzz]

Note this requires beeware/Python-Apple-support#132 to include the _posixshmem package in the support package required by multiprocessing

[freakboy3742]

The _posixshmem issue should have been resolved as part of beeware/Python-Apple-support#161; it is now verified as part of the support testbed.

However, the more general issue with multiprocessing remains.

The issue is that Briefcase doesn't guarantee that there will be a python executable; as a result, multiprocessing will fail in spawn mode (which is the default as of Python 3.8).

fork and forkserver mode should both work. However, spawn mode assumes that sys.executable can be invoked with the same arguments as the python interpreter, which isn't true of a macOS stub binary.

This is definitely something that should be addressed; some sort of logic in the stub app is required that verifies if the parent process is "self"; and if so, changes the interpretation of command line arguments so that spawn calls behave like a "normal" Python interpreter. This will need to be fixed on the Xcode template (see beeware/briefcase-macOS-Xcode-template#20).

[mhsmith]

forkserver mode doesn't work either, because that calls sys.executable to start the server, plus what the multiprocessing documentation calls a "resource tracker process".

For example, if you replace the __main__ module of a Briefcase app with the following:

if __name__ == '__main__':
    import multiprocessing
    import operator
    import sys

    print(f"{sys.argv=}")
    if len(sys.argv) == 1:
        multiprocessing.set_start_method("forkserver")
        with multiprocessing.Pool(2) as p:
            print(p.map(operator.neg, [1, 2, 3]))

briefcase run will log the following command lines:

sys.argv=['/Users/msmith/git/beeware/apps/helloworld/macOS/app/Hello World/Hello World.app/Contents/MacOS/Hello World']
sys.argv=['/Users/msmith/git/beeware/apps/helloworld/macOS/app/Hello World/Hello World.app/Contents/MacOS/Hello World', '-B', '-I', '-c', 'from multiprocessing.resource_tracker import main;main(7)']
sys.argv=['/Users/msmith/git/beeware/apps/helloworld/macOS/app/Hello World/Hello World.app/Contents/MacOS/Hello World', '-B', '-I', '-c', "from multiprocessing.forkserver import main; main(17, 18, ['__main__'], **{'sys_path': ['/Users/msmith/git/beeware/apps/helloworld/macOS/app/Hello World/Hello World.app/Contents/Resources/support/python38.zip', '/Users/msmith/git/beeware/apps/helloworld/macOS/app/Hello World/Hello World.app/Contents/Resources/support/python-stdlib', '/Users/msmith/git/beeware/apps/helloworld/macOS/app/Hello World/Hello World.app/Contents/Resources/support/python-stdlib/lib-dynload', '/Users/msmith/git/beeware/apps/helloworld/macOS/app/Hello World/Hello World.app/Contents/Resources/app_packages', '/Users/msmith/git/beeware/apps/helloworld/macOS/app/Hello World/Hello World.app/Contents/Resources/app']})"]

If you remove the len(sys.argv) check, the app will replicate itself exponentially and lock up the whole system.

fork mode worked fine for me on macOS. It's no longer the default on macOS because of python/cpython#77906, but it looks like that only affects apps which call Objective-C APIs in the child processes.

Unfortunately, using fork mode isn't a cross-platform solution, because it isn't available on Windows. It's also unsafe to call fork in any process that has multiple threads (Python or otherwise), for the reasons explained here.

[mhsmith]

So far we've been assuming that sys.executable needs to be the app's own executable. But what if it actually pointed at a normal python executable that we included in the app bundle? After all, the main use of sys.executable is to create Python subprocesses, and there's a lot of code out there that depends on that, both for multiprocessing and other things.

If the directory structure was standard (with the app and its requirements all under site-packages), then this subprocess wouldn't even need the PYTHONPATH environment variable.

[freakboy3742]

So far we've been assuming that sys.executable needs to be the app's own executable. But what if it actually pointed at a normal python executable that we included in the app bundle? After all, the main use of sys.executable is to create Python subprocesses.

If the directory structure was standard (with the app and its requirements all under site-packages), then this subprocess wouldn't even need the PYTHONPATH environment variable.

That's an interesting idea. If it works, it's definitely an easier solution than trying to patch the stub binary to be multiprocessing-compatible.

The macOS support package currently strips the Python executable from the distributed support package by design, to avoid dropping an execution environment onto the user's machine. If we adopt this approach, we might want to couple it with the "things to uninstall from the standard library" (see beeware/briefcase#2224).

Switching to using a "vanilla" site-packages installation location would likely also have the benefit of resolving any of the lingering site-related issues that we've got. It will require a bunch of changes to Briefcase to switch to a "single install destination" approach; and that will impact on the ability to update just the app sources (since we won't be able to purge the full app folder prior to installation). That might make this change a good thing to pair with introducing uv as an installer option, as the ability to rapidly re-install dependencies quickly will be an important factor in quality of life for developers.

There also might be some code-signing implications - site-packages is contained in the Python framework, and ordinarily, the macOS framework is signed. However, when it's embedded in an app, I think it needs to be re-signed with the app's credentials - but I need to confirm this.

[mhsmith]

This could also address other situations where code attempts to use sys.executable:

• Running pip – Support the use of pip in packaged apps briefcase#419
• @danyeaw recently encountered an issue where using a Hugging Face library caused his app to launch a second instance of itself. I'm not sure if this was related to multiprocessing, or some other use of sys.executable.