ABI break in openmp v21.1.0 on windows?

[h-vetinari]

In conda-forge, we generally ship the latest openmp from llvm, and due to ABI stability this works as the runtime implementation even for packages built with an older clang (against an older openmp).

However, after updating to openmp v21.1, we have crashes occurring in various workflows (where prior llvm updates where unproblematic), though only on windows. Due to the widespread use of openmp (for example together with BLAS/LAPACK), I needed to mark that version as broken. I've since reproduced the crashes when using llvm v21.1.

It's unclear to me what exactly might be the issue there; we haven't changed the way we build any of the respective packages. In the numpy test suite, so far I only see traces on the python side

Details

 │ Windows fatal exception: code 0xc06d007e
 │ Thread 0x00001f68 (most recent call first):
 │   File "%SRC_DIR%_run_env\Lib\threading.py", line 363 in wait
 │   File "%SRC_DIR%_run_env\Lib\threading.py", line 659 in wait
 │   File "%SRC_DIR%_run_env\Lib\threading.py", line 1342 in run
 │   File "%SRC_DIR%_run_env\Lib\threading.py", line 1043 in _bootstrap_inner
 │   File "%SRC_DIR%_run_env\Lib\threading.py", line 1014 in _bootstrap
 │ Thread 0x00000af0 (most recent call first):
 │   File "%SRC_DIR%_run_env\Lib\site-packages\execnet\gateway_base.py", line 534 in read
 │   File "%SRC_DIR%_run_env\Lib\site-packages\execnet\gateway_base.py", line 567 in from_io
 │   File "%SRC_DIR%_run_env\Lib\site-packages\execnet\gateway_base.py", line 1160 in _thread_receiver
 │   File "%SRC_DIR%_run_env\Lib\site-packages\execnet\gateway_base.py", line 341 in run
 │   File "%SRC_DIR%_run_env\Lib\site-packages\execnet\gateway_base.py", line 411 in _perform_spawn
 │ Current thread 0x000003b0 (most recent call first):
 │   File "%SRC_DIR%_run_env\Lib\site-packages\numpy\_core\tests\test_ufunc.py", line 2877 in test_ufunc_noncontiguous
 │   File "%SRC_DIR%_run_env\Lib\site-packages\_pytest\python.py", line 157 in pytest_pyfunc_call
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_callers.py", line 121 in _multicall
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_hooks.py", line 512 in __call__
 │   File "%SRC_DIR%_run_env\Lib\site-packages\_pytest\python.py", line 1671 in runtest
 │   File "%SRC_DIR%_run_env\Lib\site-packages\_pytest\runner.py", line 178 in pytest_runtest_call
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_callers.py", line 121 in _multicall
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_hooks.py", line 512 in __call__
 │   File "%SRC_DIR%_run_env\Lib\site-packages\_pytest\runner.py", line 246 in <lambda>
 │   File "%SRC_DIR%_run_env\Lib\site-packages\_pytest\runner.py", line 344 in from_call
 │   File "%SRC_DIR%_run_env\Lib\site-packages\_pytest\runner.py", line 245 in call_and_report
 │   File "%SRC_DIR%_run_env\Lib\site-packages\_pytest\runner.py", line 136 in runtestprotocol
 │   File "%SRC_DIR%_run_env\Lib\site-packages\_pytest\runner.py", line 117 in pytest_runtest_protocol
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_callers.py", line 121 in _multicall
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_hooks.py", line 512 in __call__
 │   File "%SRC_DIR%_run_env\Lib\site-packages\xdist\remote.py", line 227 in run_one_test
 │   File "%SRC_DIR%_run_env\Lib\site-packages\xdist\remote.py", line 206 in pytest_runtestloop
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_callers.py", line 121 in _multicall
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_hooks.py", line 512 in __call__
 │   File "%SRC_DIR%_run_env\Lib\site-packages\_pytest\main.py", line 343 in _main
 │   File "%SRC_DIR%_run_env\Lib\site-packages\_pytest\main.py", line 289 in wrap_session
 │   File "%SRC_DIR%_run_env\Lib\site-packages\_pytest\main.py", line 336 in pytest_cmdline_main
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_callers.py", line 121 in _multicall
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_hooks.py", line 512 in __call__
 │   File "%SRC_DIR%_run_env\Lib\site-packages\xdist\remote.py", line 427 in <module>
 │   File "%SRC_DIR%_run_env\Lib\site-packages\execnet\gateway_base.py", line 1291 in executetask
 │   File "%SRC_DIR%_run_env\Lib\site-packages\execnet\gateway_base.py", line 341 in run
 │   File "%SRC_DIR%_run_env\Lib\site-packages\execnet\gateway_base.py", line 411 in _perform_spawn
 │   File "%SRC_DIR%_run_env\Lib\site-packages\execnet\gateway_base.py", line 389 in integrate_as_primary_thread
 │   File "%SRC_DIR%_run_env\Lib\site-packages\execnet\gateway_base.py", line 1273 in serve
 │   File "%SRC_DIR%_run_env\Lib\site-packages\execnet\gateway_base.py", line 1806 in serve
 │   File "<string>", line 8 in <module>
 │   File "<string>", line 1 in <module>
 │ ........................[gw8] node down: Not properly terminated

Presumably we'd have to try instrumenting the builds for better debug information. FYI @mgorny

[llvmbot]

@llvm/issue-subscribers-openmp

Author: None (h-vetinari)

Details

In conda-forge, we generally ship the latest openmp from llvm, and due to ABI stability this works as the runtime implementation even for packages built with an older clang (against an older openmp).

However, after updating to openmp v21.1, we have crashes occurring in various workflows (where prior llvm updates where unproblematic), though only on windows. Due to the widespread use of openmp (for example together with BLAS/LAPACK), I needed to mark that version as broken. I've since reproduced the crashes when using llvm v21.1.

It's unclear to me what exactly might be the issue there; we haven't changed the way we build any of the respective packages. In the numpy test suite, so far I only see traces on the python side

<details>

 │ Windows fatal exception: code 0xc06d007e
 │ Thread 0x00001f68 (most recent call first):
 │   File "%SRC_DIR%_run_env\Lib\threading.py", line 363 in wait
 │   File "%SRC_DIR%_run_env\Lib\threading.py", line 659 in wait
 │   File "%SRC_DIR%_run_env\Lib\threading.py", line 1342 in run
 │   File "%SRC_DIR%_run_env\Lib\threading.py", line 1043 in _bootstrap_inner
 │   File "%SRC_DIR%_run_env\Lib\threading.py", line 1014 in _bootstrap
 │ Thread 0x00000af0 (most recent call first):
 │   File "%SRC_DIR%_run_env\Lib\site-packages\execnet\gateway_base.py", line 534 in read
 │   File "%SRC_DIR%_run_env\Lib\site-packages\execnet\gateway_base.py", line 567 in from_io
 │   File "%SRC_DIR%_run_env\Lib\site-packages\execnet\gateway_base.py", line 1160 in _thread_receiver
 │   File "%SRC_DIR%_run_env\Lib\site-packages\execnet\gateway_base.py", line 341 in run
 │   File "%SRC_DIR%_run_env\Lib\site-packages\execnet\gateway_base.py", line 411 in _perform_spawn
 │ Current thread 0x000003b0 (most recent call first):
 │   File "%SRC_DIR%_run_env\Lib\site-packages\numpy\_core\tests\test_ufunc.py", line 2877 in test_ufunc_noncontiguous
 │   File "%SRC_DIR%_run_env\Lib\site-packages\_pytest\python.py", line 157 in pytest_pyfunc_call
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_callers.py", line 121 in _multicall
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_hooks.py", line 512 in __call__
 │   File "%SRC_DIR%_run_env\Lib\site-packages\_pytest\python.py", line 1671 in runtest
 │   File "%SRC_DIR%_run_env\Lib\site-packages\_pytest\runner.py", line 178 in pytest_runtest_call
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_callers.py", line 121 in _multicall
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_hooks.py", line 512 in __call__
 │   File "%SRC_DIR%_run_env\Lib\site-packages\_pytest\runner.py", line 246 in <lambda>
 │   File "%SRC_DIR%_run_env\Lib\site-packages\_pytest\runner.py", line 344 in from_call
 │   File "%SRC_DIR%_run_env\Lib\site-packages\_pytest\runner.py", line 245 in call_and_report
 │   File "%SRC_DIR%_run_env\Lib\site-packages\_pytest\runner.py", line 136 in runtestprotocol
 │   File "%SRC_DIR%_run_env\Lib\site-packages\_pytest\runner.py", line 117 in pytest_runtest_protocol
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_callers.py", line 121 in _multicall
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_hooks.py", line 512 in __call__
 │   File "%SRC_DIR%_run_env\Lib\site-packages\xdist\remote.py", line 227 in run_one_test
 │   File "%SRC_DIR%_run_env\Lib\site-packages\xdist\remote.py", line 206 in pytest_runtestloop
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_callers.py", line 121 in _multicall
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_hooks.py", line 512 in __call__
 │   File "%SRC_DIR%_run_env\Lib\site-packages\_pytest\main.py", line 343 in _main
 │   File "%SRC_DIR%_run_env\Lib\site-packages\_pytest\main.py", line 289 in wrap_session
 │   File "%SRC_DIR%_run_env\Lib\site-packages\_pytest\main.py", line 336 in pytest_cmdline_main
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_callers.py", line 121 in _multicall
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
 │   File "%SRC_DIR%_run_env\Lib\site-packages\pluggy\_hooks.py", line 512 in __call__
 │   File "%SRC_DIR%_run_env\Lib\site-packages\xdist\remote.py", line 427 in <module>
 │   File "%SRC_DIR%_run_env\Lib\site-packages\execnet\gateway_base.py", line 1291 in executetask
 │   File "%SRC_DIR%_run_env\Lib\site-packages\execnet\gateway_base.py", line 341 in run
 │   File "%SRC_DIR%_run_env\Lib\site-packages\execnet\gateway_base.py", line 411 in _perform_spawn
 │   File "%SRC_DIR%_run_env\Lib\site-packages\execnet\gateway_base.py", line 389 in integrate_as_primary_thread
 │   File "%SRC_DIR%_run_env\Lib\site-packages\execnet\gateway_base.py", line 1273 in serve
 │   File "%SRC_DIR%_run_env\Lib\site-packages\execnet\gateway_base.py", line 1806 in serve
 │   File "<string>", line 8 in <module>
 │   File "<string>", line 1 in <module>
 │ ........................[gw8] node down: Not properly terminated

</details>

Presumably we'd have to try instrumenting the builds for better debug information. FYI @mgorny

[mgorny]

I wonder if this could be related to the crashes I've started seeing with PyTorch on Linux as well. However, at least quick testing suggests I'm also seeing them after downgrading llvm-openmp — but this doesn't guarantee that something in the pipeline wasn't built with newer llvm-openmp as well.

[isuruf]

This was a packaging bug.