Skip to content

Fix Downgrade (QA): remove PATH solve from precompile workload (intermittent libpath50 segfault)#66

Merged
ChrisRackauckas merged 1 commit into
SciML:mainfrom
ChrisRackauckas-Claude:fix-downgrade-qa-pathsolver-precompile-segfault
Jun 21, 2026
Merged

Fix Downgrade (QA): remove PATH solve from precompile workload (intermittent libpath50 segfault)#66
ChrisRackauckas merged 1 commit into
SciML:mainfrom
ChrisRackauckas-Claude:fix-downgrade-qa-pathsolver-precompile-segfault

Conversation

@ChrisRackauckas-Claude

@ChrisRackauckas-Claude ChrisRackauckas-Claude commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Problem

The Downgrade (QA) job on main is red. It fails during a from-scratch precompilation of the package with a segfault in PATHSolver's native library (libpath50):

[3632] signal (11.1): Segmentation fault
in expression starting at .../src/precompilation.jl:3
Output_Printf at .../datadeps/libpath50/libpath50.so (unknown line)
ERROR: Failed to precompile ComplementaritySolve ...

Because precompilation aborts, the QA testsets (Aqua, ExplicitImports) error before they run.

The previous Downgrade fix (#64) resolved the earlier Unsatisfiable requirements resolution failures in all three groups; Core and Applications are now green. QA was the last remaining red, and its failure is a different cause: this precompile segfault.

Root cause

The @compile_workload ends by actually solving an MCP with PATHSolverAlgorithm(), which calls into the PATH native C library (libpath50.so) from inside the precompile worker process. That native call segfaults intermittently. Reproduced locally on Julia 1.10.11 with the downgrade-resolved versions (PATHSolver 1.4.0, ForwardDiff 1.0.1, Zygote 0.7.5): roughly 1 in ~20 cold-cache precompiles crashes with the identical signature (signal (11): Segmentation fault in precompilation.jl, Output_Printf at libpath50.so). Every Downgrade run does a cold precompile, so it hits this regularly.

It is not a Julia-code bug and not a resolution problem — it is the PATH native library being unsafe to invoke inside the precompile worker. (test/aqua.jl already disables persistent_tasks with the note "PATHSolver precompile workload triggers persistent task detection", and InteriorPointMethod is already excluded from the workload for stability.)

Fix

Remove the PATH solve from the precompile workload. Invoking the PATH solver is the only operation that loads/calls libpath50 during precompilation, and native C-library calls are not what precompilation caches in the first place — so excluding it loses no meaningful cached coverage while making precompilation deterministic.

Local verification (Julia 1.10.11, downgrade-resolved versions)

  • Before the fix: cold-cache QA precompile segfaults intermittently (reproduced the exact CI signature).
  • After the fix: 8/8 cold-cache QA runs pass (Aqua: 3 pass / 1 broken-as-declared; ExplicitImports: 2 pass), zero segfaults.
  • Package still precompiles cleanly at latest versions on Julia 1.10.

Please ignore until reviewed by @ChrisRackauckas


Companion CI fix: The CUDA Core / CUDA Applications reds seen on main and on this branch are a separate, infrastructure-level failure — the two GPU matrix jobs race on the shared self-hosted ~/.gitconfig inside codecov-action's safe.directory setup (tests pass; the post-step exits 255). That is fixed in #67, not here. This PR remains focused on the Downgrade (QA) precompile segfault.

The Downgrade (QA) job fails on a from-scratch precompile of the package
with a segfault in libpath50 (PATHSolver's native library), surfacing as
"Output_Printf at libpath50.so" / "signal (11): Segmentation fault" in
expression starting at src/precompilation.jl. The crash is intermittent:
locally on Julia 1.10 it reproduces roughly 1 in 20 cold precompiles, and
every Downgrade run does a cold precompile.

Invoking the PATH native solver inside the precompile worker is the only
operation that loads/calls libpath50 during precompilation, and native C
library calls are not what precompilation caches. Removing the PATH solve
from the workload makes precompilation deterministic with no loss of
cached coverage. (InteriorPointMethod is already excluded from the
workload for similar stability reasons.)

Verified locally on Julia 1.10.11 with the downgrade-resolved versions
(PATHSolver 1.4.0, ForwardDiff 1.0.1, Zygote 0.7.5): 8/8 cold-cache QA
runs (Aqua + ExplicitImports) now pass with no segfault, and the package
still precompiles cleanly at latest versions.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ChrisRackauckas ChrisRackauckas marked this pull request as ready for review June 21, 2026 11:06
@ChrisRackauckas ChrisRackauckas merged commit 6776124 into SciML:main Jun 21, 2026
20 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants