Skip to content

Require strict SciMLTesting 2.4 QA - #76

Merged
ChrisRackauckas merged 6 commits into
SciML:mainfrom
ChrisRackauckas-Claude:agent/complementaritysolve-strict-scimltesting-24-current
Aug 2, 2026
Merged

Require strict SciMLTesting 2.4 QA#76
ChrisRackauckas merged 6 commits into
SciML:mainfrom
ChrisRackauckas-Claude:agent/complementaritysolve-strict-scimltesting-24-current

Conversation

@ChrisRackauckas-Claude

@ChrisRackauckas-Claude ChrisRackauckas-Claude commented Jul 26, 2026

Copy link
Copy Markdown
Member

Ignore until reviewed by @ChrisRackauckas.

Summary

  • Replace the QA suppressions with run_qa(ComplementaritySolve) plus a single scoped ei_kwargs ignore for PATHSolver's non-public C-API names.
  • Split exported user API from documented, qualified developer extension APIs and add generic-only interface tests.
  • Render all public/developer API docs with checkdocs = :public; remove non-public dependency API use and the stale FillArrays dependency.

Validation

  • Pkg.test() completed: 281 pass, 0 fail, 7 pre-existing broken.
  • GROUP=QA Pkg.test(): 20 pass, 0 fail — all Aqua sub-checks (ambiguities, persistent_tasks, project_extras, deps_compat), public docs, reexports, and the remaining five ExplicitImports checks now pass unsuppressed.
  • Documenter doctests/checkdocs, Runic check, and diff check pass.

Remaining exception

PATHSolver declares no export and no public names at all, so any qualified access into it fails all_qualified_accesses_are_public. Its README nevertheless directs users to PATHSolver.solve_mcp and the MCP_Termination enum values as the supported entry point, and there is no public alternative for reaching the PATH C API. test/qa.jl therefore keeps one ignore list, scoped to the six names actually used:

solve_mcp, MCP_Solved, MCP_NoProgress, MCP_MajorIterationLimit, MCP_MinorIterationLimit, MCP_TimeLimit

The comment above it says to drop the ignore once PATHSolver publishes those names upstream. Making them public there needs a VERSION >= v"1.11" guard, since PATHSolver's julia compat is 1.6.

ChrisRackauckas and others added 2 commits July 26, 2026 19:19
Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
PATHSolver declares no `export` and no `public` names, so every qualified
access into it fails `all_qualified_accesses_are_public`. There is no public
alternative for reaching the PATH C API, so restore an ignore list scoped to
the six names actually used, with a note to drop it once PATHSolver publishes
them upstream. All other QA suppressions stay removed.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LJbtE2oppQxPFNGSJbhm22
@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

Pushed 7c01a19 to fix the QA lane.

Diagnosis. The only failing check was Tests (QA) — 19 pass, 1 error, the error being all_qualified_accesses_are_public:

NonPublicQualifiedAccessException
- `MCP_MajorIterationLimit` ... src/algorithms/mcp/pathsolver.jl:80:33
- `MCP_MinorIterationLimit` ... :82:33
- `MCP_NoProgress`          ... :78:33
- `MCP_Solved`              ... :76:29
- `MCP_TimeLimit`           ... :84:33
- `solve_mcp`               ... :68:34

PATHSolver v1.7.9 has zero export statements and zero public declarations in src/PATHSolver.jl, so nothing in that package is public and any qualified access into it trips this check. There is no public alternative for reaching the PATH C API, and its README explicitly recommends PATHSolver.solve_mcp and shows PATHSolver.MCP_Solved in its documented output.

Fix. Restore a single ei_kwargs ignore scoped to exactly those six names, with a comment stating why it is irreducible and to remove it once PATHSolver publishes them. Every other suppression that this PR removed stays removed.

Verified locally (Julia 1.12):

  • Before: GROUP=QA Pkg.test()19 pass, 1 error, same exception as CI.
  • After: GROUP=QA Pkg.test()Complementarity Solve | 20 pass, 20 total, 1m40.3s, EXIT=0.
  • Runic.main(["--check", "--diff", "test/qa.jl"]) → exit 0.

Follow-up. The clean fix is upstream: declare those six names public in PATHSolver, guarded by VERSION >= v"1.11" since its julia compat is 1.6. That is not a SciML repo, so it needs sign-off before anything is opened there.

SciMLBase 3.40.0 dropped `@reexport using SciMLOperators`, so
`using SciMLBase: FunctionOperator` fails to load on SciMLBase >= 3.40 while
the declared compat still allows it. The test environments happen to pin
SciMLBase to 3.39.1, but Aqua's persistent-tasks check resolves Project.toml
on its own and picks 3.41.0, so enabling that check surfaced the break as a
precompile failure.

Depend on SciMLOperators directly and import the name from the package that
owns it.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LJbtE2oppQxPFNGSJbhm22
@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

Pushed 498f94c. The run_qa ignore in 7c01a19 did fix ExplicitImports (now 6/6 pass), but Tests (QA) and Downgrade (QA) then failed on Aqua's persistent_tasks check. Different cause, and it is not PATHSolver.

Root cause: SciMLBase 3.40.0 removed @reexport using SciMLOperators.

SciMLBase 3.39.0  reexport=1
SciMLBase 3.39.1  reexport=1
SciMLBase 3.40.0  reexport=0   <- dropped here
SciMLBase 3.41.0  reexport=0

src/ComplementaritySolve.jl:10 did using SciMLBase: ..., FunctionOperator, ..., so the package does not load at all on SciMLBase >= 3.40, while Project.toml still declares SciMLBase = "2.147, 3.1". The Core/Applications lanes pass only because their extra test dependencies pin SciMLBase down to 3.39.1. Aqua's persistent_tasks check builds a temp environment from Project.toml alone, which resolves to 3.41.0 and hits the break — so the failure is done.log was not created, but precompilation exited, not a real persistent task.

The Downgrade (QA) log shows the underlying error directly:

ERROR: LoadError: UndefVarError: `FunctionOperator` not defined
in expression starting at src/ComplementaritySolve.jl:1

The Tests (QA) lane hid it because Aqua calls Pkg.precompile(; io = devnull), which swallows the error on Julia 1.12.

This was latent before this PR: main sets persistent_tasks = false, so nothing ever resolved Project.toml standalone. Enabling strict QA is what exposed it.

Fix. Depend on SciMLOperators directly and import FunctionOperator from the package that owns its public contract, rather than through a re-exporter.

Verified locally (versions reported by Pkg.dependencies()):

  • Standalone env from Project.toml, before the fix — SciMLBase 3.41.0 / SciMLOperators 1.25.2, UndefVarError: FunctionOperator not defined on both Julia 1.10.11 and 1.12.6.
  • Same env, after the fix — LOADED OK on Julia 1.10.11 and LOADED OK on Julia 1.12.6, still at SciMLBase 3.41.0.
  • GROUP=QA Pkg.test()Complementarity Solve | 20 pass | 20 total | 1m39.3s, EXIT=0.
  • GROUP=Core Pkg.test()221 pass, 6 broken, 227 total | 5m46.6s, EXIT=0 (the 6 broken are pre-existing).
  • Runic --check on both changed files → exit 0.

Worth flagging separately: SciMLBase 3.40.0 dropping the SciMLOperators re-export is a breaking change for any downstream doing using SciMLBase: FunctionOperator, and SciMLBase 3.40/3.41 still admit SciMLBase = "...3.1"-style compat, so downstreams will hit this silently.

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

The remaining Tests (QA) failure is not a defect in this package, and the fix is upstream.

persistent_tasks fails because Aqua's probe cold-precompiles a throwaway wrapper package in a subprocess, and that subprocess dies on the CI runner without writing its done.log sentinel. Aqua calls Pkg.precompile(; io = devnull), so no diagnostics reach the log and the flake is indistinguishable from a real persistent task.

Evidence it is environmental rather than a code defect:

  • The 2026-07-26 run of this PR, on SciMLTesting 2.4.1, hit the same failure and passed on retry: persistent-tasks wrapper precompile exited without writing done.log; retrying, attempt = 1persistent_tasks (retry) | 1 pass | 4m00.1s.
  • That retry was removed by Revert "persistent-tasks: race-tolerant retry in run_qa (fix flaky Aqua failure fleet-wide)" SciMLTesting.jl#33, so 2.6.1 turns the same flake into a hard failure.
  • Not reproducible locally: a cold depot on Julia 1.12 precompiles all 230 dependencies and exits 0 (230 dependencies successfully precompiled in 212 seconds, COLD PRECOMPILE OK). Warm GROUP=QA runs pass 20/20.

Fix opened upstream: SciML/SciMLTesting.jl#42 re-lands the race-tolerant retry (344/344 tests pass; the retry probe passes against this branch's checkout). This PR stays red until that merges and SciMLTesting 2.7.0 is registeredtest/Project.toml already allows it via SciMLTesting = "2.4", so no change is needed here once it lands.

No suppression is being added for this. Both real bugs this PR surfaced are fixed: the PATHSolver ignore (7c01a19) and the SciMLBase re-export break (498f94c), and Downgrade (QA) went green with the latter.

ChrisRackauckas and others added 3 commits August 2, 2026 07:18
Do not review; this commit will be reverted. Aqua silences the probe
subprocess, so the CI failure has no diagnostics. Reproduce the probe with
stderr passed through and report exitcode/termsignal.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The diagnostic did its job: the probe's wrapper precompile exits 0 with no
signal, having failed on Julia's "missing from the cache" race for
SciMLBaseDifferentiationInterfaceExt. Not a crash, not resource exhaustion.
Restores test/qa.jl and drops the temporary Pkg test dependency.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

Resolved: the persistent_tasks failure is a Julia precompilation race, not a defect in this package. A temporary diagnostic commit (since reverted) reproduced Aqua's probe with stderr passed through instead of discarded:

34843.2 ms  ? jl_ji9KfebcbW
50 dependencies successfully precompiled in 164 seconds. 204 already precompiled.
1 dependencies failed but may be precompilable after restarting julia
┌ jl_ji9KfebcbW
│  ┌ Warning: Module SciMLBaseDifferentiationInterfaceExt with build ID fafbfcfd-29f4-4995-193d-d74156971bdb is missing from the cache.
│  │ This may mean SciMLBaseDifferentiationInterfaceExt [61c4812a-2cb6-58ab-bd42-a7cbe32d188a] does not support precompilation but is imported by a module that does.
└  └ @ Base loading.jl:2643

┌ Info: DIAGNOSTIC: wrapper precompile finished
│   done_log_written = false
│   exitcode = 0
│   termsignal = 0
└   elapsed_s = 166.1

Exit code 0, no termination signal. The wrapper package fails to precompile on the "missing from the cache" race, Pkg.precompile() does not throw for it, so the child exits cleanly without ever loading the wrapper. No done.log is written and Aqua reports a persistent task. Julia's own advice is may be precompilable after restarting julia.

Aqua invokes this as Pkg.precompile(; io = devnull), which is why none of the above has ever appeared in a CI log and why the failure has been misattributed here for months.

The same run also demonstrates the fix accidentally: because the diagnostic ran the probe once first, the cache was warm when Aqua ran its own probe, which then passed — 20/20, job green.

Ruled out, with local reproductions:

Fix is SciML/SciMLTesting.jl#42, which re-lands the race-tolerant retry; its original diagnosis named this exact race and was correct. This PR needs no further change and no suppression — test/Project.toml already allows the fixed version via SciMLTesting = "2.4".

@ChrisRackauckas
ChrisRackauckas marked this pull request as ready for review August 2, 2026 21:26
@ChrisRackauckas
ChrisRackauckas merged commit 98a5fe5 into SciML:main Aug 2, 2026
8 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants