Skip to content

Conversation

@Erotemic
Copy link
Member

Fix Python 3.14.2 multiprocessing shutdown behavior by ensuring only the owner process registers the atexit reporting hook.

Problem

When running the complex example with LINE_PROFILE=1 and PROFILE_TYPE=explicit on Python 3.14.2, multiprocessing using the forkserver start method can cause non-owner processes (e.g., the forkserver process / child processes) to also register the atexit hook. At interpreter shutdown, those processes may try to print/write profiling results, resulting in:

  • duplicated “Wrote profile results …” messages,
  • empty or truncated profile_output.txt,
  • output that continues after the shell prompt appears.

Root cause

atexit.register(...) was executed in processes that should not own reporting. Even if show() includes guards, shutdown ordering and partially-torn-down globals can make those guards unreliable. The safest approach is to avoid registering the hook outside the owner process.

Fix

Move/guard atexit.register(...) so it occurs only after establishing ownership (PID/parent/process role) and only in the designated owner process. Non-owner processes never register the shutdown reporter.


The above is a GPT description of the patch, which I used heavily to find the problem. I believe the diagnosis is correct, but I don't understand what changed in 3.14.2 to cause the problem. Pushing up the patch as is to see if it fixes the dashboard.

@Erotemic
Copy link
Member Author

I think I see the issue now. It's a combination of 2 things, which is why 3.14.0 worked, but 3.14.2 broke.

First: In 3.14 they changed the default process pool start method

Changed in version 3.14: The default process start method (see Contexts and start methods) changed away from fork. If you require the fork start method for ProcessPoolExecutor you must explicitly pass mp_context=multiprocessing.get_context("fork").

However, there was an issue that prevented forkserver from setting sys.argv correctly, and when that was fixed:

gh-143706: Fix multiprocessing forkserver so that sys.argv is correctly set before main is preloaded. Previously, sys.argv was empty during main module import in forkserver child processes. This fixes a regression introduced in 3.13.8 and 3.14.1. Root caused by Aaron Wieczorek, test provided by Thomas Watson, thanks!)

Then the failure started showing up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant