Troubleshoot PyTorch with num_workers > 0

[ctrueden]

From @MichaelSNelson:

PyTorch's DataLoader supports worker processes via num_workers > 0 for prefetching/augmentation. Inside an Appose worker, anything > 0 hangs — the training loop inside the Appose worker — for batch in DataLoader: — never returns a batch when num_workers > 0. The Python worker (Appose's child) stays alive and responsive on the Appose protocol; it's the DataLoader's own child processes (PyTorch's grandchildren of the JVM) that deadlock before the first batch comes back. Nothing to do with Appose's own main-thread queue. No exception, no log, just silence after training setup completes. Dropping back to num_workers=0 (single-threaded data loading) works fine but leaves the GPU ~30–70% idle between batches during training.

We have a clean repro on current releases: our extension added a user preference exposing num_workers (defaults to 0). When set to 2, the bootstrap log confirms the patch reaches Python — DataLoader bootstrap: forcing num_workers=2, persistent_workers=True — and then training sets up through "Set 65 encoder modules to eval mode" and hangs before batch 1. Zero progress after 3+ minutes.

My mental model of why: PyTorch spawns the workers via torch.multiprocessing (spawn on Windows, fork-or-spawn elsewhere). The children inherit or re-open stdin/stdout, which Appose also uses for its JSON protocol. I can imagine a few failure modes:

1. Stdio inheritance deadlock — children holding a ref to the parent's stdin pipe, or spawn's re-import triggering input() in the worker init.
2. Some PYTHONEXECUTABLE / PATH issue (I noticed the recent commits in appose-python/main stripping conda activation vars and PYTHONEXECUTABLE, which sounded related). 3. Something Windows-specific in pixi-managed Python that confuses multiprocessing's default spawn logic.

What I'd love to know:

1. Has anyone successfully run torch.multiprocessing-based workers inside an Appose worker? Is this expected to work, or structurally a dead end given Appose's stdio-based IPC?

2. If it's not expected to work, is there a recommended pattern for CPU-bound per-batch preprocessing inside Appose — e.g., threaded prefetch, or a second Appose worker acting as a data producer over shared memory?

3. Would you be open to me filing a reduced GitHub repro if it's something that could be addressed upstream? I can strip this down to a minimal Appose-worker-calls-DataLoader-with-num_workers=2 if it's useful.
   Happy to dig in wherever's helpful. In the meantime we're pursuing an in-memory dataset cache to sidestep the I/O bottleneck without multiprocessing.

I put a minimal repro together so you can poke at it: https://github.com/MichaelSNelson/appose-dataloader-repro

[carlosuc3m]

HEllo @ctrueden @MichaelSNelson , a workaround (at least in Linux) is to use:

import multiprocessing as mp
mp.set_start_method("forkserver", force=True)

or

import multiprocessing as mp
mp.set_start_method("spawn", force=True)

at the begining of the script

[carlosuc3m]

Hello again @ctrueden and @MichaelSNelson , after some back and forth with Codex, it seems that we were able to pinpoint a little bit more the root cause of the issue and a possible fix. Bear in mind that I dont fully understand everything, so forgive me in advance if anything is incorrect:

Summary

The issue appears to be caused by forked children inheriting Appose’s sys.stdin object while the Appose receiver thread is blocked on stdin. During multiprocessing child bootstrap, Python calls multiprocessing.util._close_stdin(), which calls sys.stdin.close(). In the forked child, this blocks and the child never reaches the target.

Observed behavior

Inside an Appose Python task:

• os.fork() works.
• os.fork() + raw os.pipe() works.
• multiprocessing.Process(..., fork) creates a child, but the child never reaches the target.
• multiprocessing.Pipe/Queue(..., fork) fails to communicate.
• torch.utils.data.DataLoader(num_workers > 0) hangs before first batch.
• forkserver works.
• spawn works.
• Monkeypatching multiprocessing.util._close_stdin = lambda: None makes fork-based multiprocessing and PyTorch DataLoader workers work.

Minimal diagnostic result

Instrumenting multiprocessing.popen_fork.Popen._launch, BaseProcess._bootstrap, and multiprocessing.util._close_stdin() showed this sequence in the forked child:

child after os.fork before close fds
child before process_obj._bootstrap
child enter BaseProcess._bootstrap wrapper
child before util._close_stdin

Then it hangs. It never reaches:

child after util._close_stdin
child target start

With:

import multiprocessing.util as mp_util
mp_util._close_stdin = lambda: None

the same test reaches the target and exits:

child skipped util._close_stdin
child target start
child target end
exitcode=0

PyTorch DataLoader(num_workers=2) also works with this monkeypatch.

Likely root cause

Appose’s Python worker starts a receiver thread that reads from stdin, e.g. input() / stdin protocol handling. Later, user task code creates forked multiprocessing children.

On Linux, fork copies the process while only the calling thread survives in the child. The forked child inherits sys.stdin, including internal TextIOWrapper/buffer state and locks, but the receiver thread that may own or interact with that state does not exist in the child.

Python multiprocessing child bootstrap calls:

multiprocessing.util._close_stdin()

which does:

sys.stdin.close()
sys.stdin = open(os.devnull, ...)

In the Appose forked child, sys.stdin.close() appears to block forever. Therefore the child never reaches the multiprocessing target, and PyTorch DataLoader waits forever for batches.

Why this matters

This affects common ML workloads. PyTorch DataLoader(num_workers > 0) defaults to fork-based workers on Linux. Inside Appose, this can make training hang before the first batch.

For example, Ultralytics YOLO training through Appose hangs with dataloader workers enabled, but works with:

workers=0

or with multiprocessing start method:

forkserver

or:

spawn

Proposed Appose fix

A clean Appose-side fix would be to separate Appose protocol stdin from task/user-visible sys.stdin.

Conceptually, at Python worker startup:

  import os
  import sys

  _appose_stdin = sys.stdin
  sys.stdin = open(os.devnull, "r")

Then the Appose receiver thread should read from _appose_stdin, not from global sys.stdin.

This way:

• Appose can still receive protocol messages from Java.
• User code sees a safe /dev/null stdin.
• Forked multiprocessing children inherit /dev/null as sys.stdin.
• Python multiprocessing can safely call _close_stdin().
• Normal multiprocessing semantics are preserved better than monkeypatching _close_stdin.

A less ideal workaround would be:

  import multiprocessing.util as mp_util
  mp_util._close_stdin = lambda: None

This avoids the hang but is global and changes normal multiprocessing behavior. It also leaves Appose stdin inherited by forked children, which is not ideal.

[MichaelSNelson]

This is all mostly over my head, but since I have claude addressing this for my apposed QuPath extensions:

---

Thanks @carlosuc3m, this is a really thorough piece of detective work — your _close_stdin instrumentation was a great roadmap. We took your suggestion to test on Windows, where most of my QuPath users are. Repro and the diagnostic instrumentation live in https://github.com/MichaelSNelson/appose-dataloader-repro, with sitecustomize.py deployed into the pixi env's site-packages so spawn children get auto-instrumented (the parent's class-level monkey-patches don't propagate through Windows spawn — children re-exec python.exe and load a pristine stdlib).

A few concrete findings from Windows 11 + Python 3.14 + PyTorch 2.10 + spawn:

1. The hang reproduces, but with a different mechanism than Linux. _close_stdin does not block on Windows — it returns in 0.0000 s in every spawned worker. Workers also reach Process.run with target=_worker_loop when given enough time. So whatever's broken is not the inherited-stdin deadlock you traced on Linux.

2. The dominant Windows symptom is that child python.exe interpreter startup is blocked until the parent Appose task ends. Across three runs with timeouts of 60 s, 300 s, and 900 s, the child's sitecustomize.py fired at parent_iter() + timeout, every time. That can't be cold-start cost (15 min for a python.exe to reach site.py is implausible) — it's something the parent worker holds while alive that prevents the children from finishing their interpreter init. Plausible candidates: filesystem locks on site-packages / __pycache__ (Windows holds read locks more aggressively than Linux), antivirus serialization of python.exe spawn from a long-running parent, or Windows job-object quotas. Once the parent task is cancelled and the worker process is torn down, children proceed instantly.

3. Implication for the proposed Appose-side fix. Your private-_appose_stdin + sys.stdin = open(os.devnull) change closes Linux cleanly, and is platform-agnostic in the sense that it's safe everywhere. But based on this data it likely won't address Windows on its own — children there don't even reach _close_stdin while the parent is alive, so changing what they'd see if they did reach it doesn't unblock them. Windows looks like it needs separate work, probably around how the Appose worker holds env / handle resources during a long-running task. Happy to file a separate issue once we narrow it further if you'd like to keep this one focused on the Linux fix.

4. Separate Windows-only gotcha worth surfacing alongside whatever fix lands. On Windows + spawn, any class or function defined inside an Appose task script (the natural pattern for end users — Dataset subclass, worker_init_fn, collate_fn, lambda transforms) is unpicklable because it lives in the synthetic <string> module rather than a real importable module. The error users hit is:

   _pickle.PicklingError: Can't pickle <class 'X'>: attribute lookup X on builtins failed

and it fires before any deadlock can manifest. The repro hit it in its first incarnation. Worth a docs note even after the deadlock is fixed.

macOS Python switched the default from fork to spawn in 3.8, so the picture there should mirror Windows rather than Linux. I don't have a Mac to confirm directly; if anyone in the thread does, the repro is ready to run there.