Re-land race-tolerant persistent-tasks retry in run_qa

[ChrisRackauckas-Claude]

Please ignore until reviewed by @ChrisRackauckas. Draft, opened by an agent.

Re-lands #32, which was reverted by #33 twelve minutes after merging, with no stated reason and no review comments. Re-landing was explicitly requested.

Why

Aqua's test_persistent_tasks cold-precompiles a throwaway wrapper package in a subprocess and waits for it to write a done.log sentinel. On a large dependency graph that parallel precompile can exit without writing the sentinel, and Aqua then reports the package as holding a persistent task. The failure is indistinguishable from a real one because Aqua calls Pkg.precompile(; io = devnull), so nothing reaches stderr:

┌ Error: Unexpected error: /tmp/jl_mFyfCx8Da1/done.log was not created, but precompilation exited
└ @ Aqua ~/.julia/packages/Aqua/h1qD0/src/persistent_tasks.jl:120
Persistent tasks: Test Failed at .../persistent_tasks.jl:39

run_qa disables Aqua's single-shot sub-check and runs _has_persistent_tasks_with_retry, which re-runs the wrapper only on the missing-sentinel path. Precompilation stays parallel, so the retry recompiles just the few modules the first attempt left uncached. A genuine persistent task writes done.log and then holds the process open past tmax, which is still detected — the retry fires only on a path a real task cannot reach.

Evidence this fixes a live failure

SciML/ComplementaritySolve.jl#76 is currently red on exactly this. Its 2026-07-26 run, on SciMLTesting 2.4.1 (retry present), shows the retry firing and recovering:

┌ Info: SciMLTesting: persistent-tasks wrapper precompile exited without writing done.log; retrying
│   attempt = 1
└   retries = 3
persistent_tasks (retry)   |  1   1  4m00.1s

The same repo on 2.6.1 (retry absent) fails the check on every run. I could not reproduce the underlying precompile death locally — a cold depot on Julia 1.12 precompiles all 230 dependencies and exits 0 — which is consistent with it being a resource/race condition on the 4-core GitHub runner rather than anything in the package.

Conflict resolution

The revert did not apply cleanly; _standard_aqua_kwargs was added since (#37, #38). Resolved so the retry's persistent_tasks = false merges over _standard_aqua_kwargs(aqua_kwargs) rather than over raw aqua_kwargs, preserving the piracy treat_as_own defaults:

effective_aqua_kwargs = merge(
    _standard_aqua_kwargs(aqua_kwargs),
    _aqua_broken_disable(aqua_broken),
    (; persistent_tasks = false),
)

Validation

• Pkg.test() on this branch: 344 pass, 0 fail, exit 0 (Julia 1.12.6).
• _has_persistent_tasks_with_retry(ComplementaritySolve) against the PR-76 checkout: passes, retry probe | 1 pass | 21.3s.
• Runic --check on src/SciMLTesting.jl: exit 0.
• Version bumped 2.6.2 → 2.7.0 (behavior change, non-breaking).

The truly-upstream fix still belongs in Aqua (JuliaTesting/Aqua.jl#315) — at minimum it should stop swallowing the subprocess error, since right now every failure mode looks identical.

[ChrisRackauckas-Claude]

Confirmed the root cause on a real runner. I added a temporary diagnostic to SciML/ComplementaritySolve.jl#76 that reproduces Aqua's probe with stderr passed through rather than sent to devnull:

34843.2 ms  ? jl_ji9KfebcbW
1 dependencies failed but may be precompilable after restarting julia
┌ jl_ji9KfebcbW
│  ┌ Warning: Module SciMLBaseDifferentiationInterfaceExt with build ID fafbfcfd-29f4-4995-193d-d74156971bdb is missing from the cache.
└  └ @ Base loading.jl:2643

┌ Info: DIAGNOSTIC: wrapper precompile finished
│   done_log_written = false
│   exitcode = 0
│   termsignal = 0
└   elapsed_s = 166.1

This is exactly the race #32 described. Exit code 0 and no termination signal: not a crash, not an OOM, not a real persistent task. Pkg.precompile() does not throw on the missing-from-cache failure, so the child exits without loading the wrapper and never writes done.log.

The same run demonstrates the remedy incidentally — the diagnostic warmed the cache, so Aqua's own probe immediately afterwards passed, 20/20. That is precisely what the retry in this PR automates, and it matches Julia's own guidance in the warning text (may be precompilable after restarting julia).

Alternative causes ruled out locally before reaching for the retry: no persistent task (using exits in 8s, 3/3); no native segfault (0 failures in 25 cold recompiles re-running the @compile_workload); no resource exhaustion (all 230 dependencies precompile in 214s under a 4 CPU / 16 GB cgroup cap matching a GitHub runner).