Activate a scratch copy of the group env so QA runs stop rewriting the repo's Project.toml

[ChrisRackauckas-Claude]

Closes #46.

Please ignore until reviewed by @ChrisRackauckas.

What changed and why

activate_group_env activated the group env in the repo under test — a git-tracked directory — and then Pkg.developed into it. Pkg.develop rewrites the project it acts on, so every QA run left the working tree dirty: a new [deps] entry for each parent that was not already listed, plus Pkg's normalization of [sources] and [compat]. The injected dep lands with no matching [sources] entry (the dev path lives only in the gitignored manifest), so committing that churn silently points a monorepo QA env at the registered release of the package the repo is testing.

The fix copies the env (Project.toml, Manifest.toml, LocalPreferences.toml, and the Julia* variants) into a temporary directory and activates the copy, so Pkg.develop/Pkg.instantiate never write into the repo. Relative path entries — [sources] in the project, developed entries in the manifest — are rewritten to absolute paths, since the copy no longer sits at the env's original depth ("../../../OrdinaryDiffEqCore" would otherwise resolve outside the repo). The copy is memoized per group dir per process, so re-activating the same env in one run reuses the resolved state; sandbox = false restores the old in-place behavior.

Consequence worth flagging: dirname(Base.active_project()) is now a temp dir rather than group_dir. Test bodies that locate files with @__DIR__ (all the ones I looked at) are unaffected; anything resolving paths against the active project would be. Documented in the docstring and README.

Reproduction on unmodified main

Fresh clone of SciML/OrdinaryDiffEq.jl at 9d23061 (clean tree), Julia 1.12.6, SciMLTesting main at 1796019 (v2.6.3) dev'd into a driver env, running the call OrdinaryDiffEqBDF's runtests.jl makes:

activate_group_env(qa; parent = [dirname(dirname(qa)), ode])   # sublibrary + repo root

$ git status --porcelain
 M lib/OrdinaryDiffEqBDF/test/qa/Project.toml

@@ -2,21 +2,22 @@
 AllocCheck = "9b6a8646-10ed-4001-bbdc-1d2f46dfbb1a"
 Aqua = "4c88cf16-eb10-579e-8560-4a9242c79595"
 JET = "c3a54625-cd67-489e-a8e7-0a5a0ff4e31b"
+OrdinaryDiffEq = "1dea7af3-3e70-54e6-95c3-0bf5283fa5ed"
 OrdinaryDiffEqBDF = "6ad6398a-0878-4a85-9266-38940aa047c8"
 OrdinaryDiffEqCore = "bbf590c4-e513-4bbe-9b18-05decba2e5d8"
 SciMLBase = "0bca4576-84f4-4d90-8ffe-ffa030f20462"
 SciMLTesting = "09d9d899-5365-40a9-917a-5f67fddea283"
 Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
 
-[sources.OrdinaryDiffEqCore]
-path = "../../../OrdinaryDiffEqCore"
+[sources]
+OrdinaryDiffEqCore = {path = "../../../OrdinaryDiffEqCore"}
 
 [compat]
 AllocCheck = "0.2"
 Aqua = "0.8.11"
 JET = "0.9, 0.11"
-SciMLTesting = "2.1"
-julia = "1.10"
 OrdinaryDiffEqBDF = "2"
 OrdinaryDiffEqCore = "4"
 SciMLBase = "3"
+SciMLTesting = "2.1"
+julia = "1.10"

Discriminating test

New testset activate_group_env leaves the group env's Project.toml untouched: a fake monorepo (root package, sublibrary, unregistered sibling pinned by a relative [sources] path that escapes the env directory), activated with the two-parent call, asserting the tracked Project.toml is byte-identical afterwards, no Manifest.toml is dropped into the env, and the sandbox still resolves the sibling by path.

Because reverting src/ also removes the sandbox kwarg that another testset now uses, the clean before/after below runs the new testset verbatim as a standalone script against the same package checkout, src/SciMLTesting.jl reverted to main and then restored. The full-suite numbers for both states follow.

Before the fix (git checkout origin/main -- src/SciMLTesting.jl):

ERROR: LoadError: Some tests did not pass: 3 passed, 3 failed, 0 errored, 0 broken.
activate_group_env leaves the group env's Project.toml untouched: Test Failed at discriminate.jl:37
  Expression: read(project_file, String) == before
   Evaluated: "[deps]\nRootPkg = \"00000000-0000-0000-0000-0000000000d3\"\nSib = \"00000000-0000-0000-0000-0000000000d1\"\nSub = \"00000000-0000-0000-0000-0000000000d2\"\n\n[sources]\nSib = {path = \"../../../Sib\"}\n" == "[deps]\nSub = \"00000000-0000-0000-0000-0000000000d2\"\nSib = \"00000000-0000-0000-0000-0000000000d1\"\n\n[sources.Sib]\npath = \"../../../Sib\"\n"
activate_group_env leaves the group env's Project.toml untouched: Test Failed at discriminate.jl:38
  Expression: !(isfile(joinpath(group_dir, "Manifest.toml")))
activate_group_env leaves the group env's Project.toml untouched: Test Failed at discriminate.jl:41
  Expression: ((active["sources"])["Sib"])["path"] == abspath(sib)
   Evaluated: "../../../Sib" == "/home/crackauc/claude_tmp/jl_RQeIbc/lib/Sib"
Test Summary:                                                    | Pass  Fail  Total   Time
activate_group_env leaves the group env's Project.toml untouched |    3     3      6  11.1s

After the fix:

Test Summary:                                                    | Pass  Total   Time
activate_group_env leaves the group env's Project.toml untouched |    6      6  10.1s

Test suite

julia --project=. -e 'using Pkg; Pkg.test()', Julia 1.12.6, with src/ reverted to main (the three activation testsets fail; the error is the not-yet-existing sandbox kwarg):

Test Summary:                                                                | Pass  Fail  Error  Total   Time
SciMLTesting                                                                 |  361     7      1    369  59.1s
  activate_group_env                                                         |    2     2      1      5  11.4s
  activate_group_env leaves the group env's Project.toml untouched           |    3     3             6   3.3s
  run_tests folder mode: QA folder with own Project.toml is activated        |    4     2             6   1.7s

With the fix, Julia 1.12.6:

Test Summary: | Pass  Total   Time
SciMLTesting  |  370    370  55.0s
     Testing SciMLTesting tests passed

With the fix, Julia 1.11.9:

Test Summary: | Pass  Total     Time
SciMLTesting  |  370    370  1m02.9s
     Testing SciMLTesting tests passed

With the fix, Julia 1.10.11 (LTS — the develop_sources! [sources] backport path; fewer tests because the 1.11+-gated ones do not run):

Test Summary: | Pass  Total   Time
SciMLTesting  |  354    354  50.7s
     Testing SciMLTesting tests passed

Downstream verification

Throwaway clone of SciML/OrdinaryDiffEq.jl at 9d23061. Where the lane needed the patched SciMLTesting, the pin was committed first, so git status --porcelain measures only harness churn.

A full GROUP=QA Pkg.test() lane, Julia 1.12.6, lib/OrdinaryDiffEqBDF, patched SciMLTesting v2.7.0 pinned by [sources]. The lane passes and the tree stays clean:

Test Summary:    | Pass  Broken  Total     Time
Allocation Tests |    2      18     20  4m27.7s
Test Summary: | Pass  Broken  Total     Time
JET Tests     |   18      23     41  1m16.7s
Test Summary: | Pass  Total     Time
Aqua          |   20     20  2m18.7s
     Testing OrdinaryDiffEqBDF tests passed

$ git status --porcelain      # empty

The same lane against the unpatched baseline (registered SciMLTesting v2.6.2, same commit, [sources] pin reverted) dirties the tree with exactly the diff from #46:

$ git status --porcelain
 M lib/OrdinaryDiffEqBDF/test/qa/Project.toml

(That baseline run also fails public API has docstrings on :SciMLBase — unrelated to this PR; it is what #45 already fixed on main, which is why the patched run passes it.)

Full disclosure on flakiness: the first patched run of this lane failed one check, Aqua's Persistent tasks, which took 4m14s while ~200 other Julia processes were on this box; Aqua builds its own wrapper env (Pkg.generate + Pkg.develop) independent of the active project and fails when precompilation exceeds tmax = 10s. The re-run quoted above passes it. The baseline run passed it in 16.3s under lighter load.

The sandbox that absorbed the churn during that run, for illustration — this is what used to be written into the repo:

[deps]
AllocCheck = "..."
Aqua = "..."
JET = "..."
OrdinaryDiffEq = "1dea7af3-3e70-54e6-95c3-0bf5283fa5ed"     # <- the injected dep
...
[sources]
OrdinaryDiffEqCore = {path = "/…/ode-st46/lib/OrdinaryDiffEqCore"}   # <- rewritten absolute

A second full lane on the LTS reproduces the bug and shows the class is wider than one dep. GROUP=QA Pkg.test(), Julia 1.10.11, lib/OrdinaryDiffEqSIMDRK, registered SciMLTesting v2.6.2 (Testing OrdinaryDiffEqSIMDRK tests passed, tree dirty afterwards):

--- a/lib/OrdinaryDiffEqSIMDRK/test/qa/Project.toml
+++ b/lib/OrdinaryDiffEqSIMDRK/test/qa/Project.toml
 [deps]
 AllocCheck = "9b6a8646-10ed-4001-bbdc-1d2f46dfbb1a"
+DiffEqBase = "2b5f629d-d688-5b77-993f-72d75c75574e"
 JET = "c3a54625-cd67-489e-a8e7-0a5a0ff4e31b"
+OrdinaryDiffEq = "1dea7af3-3e70-54e6-95c3-0bf5283fa5ed"
 OrdinaryDiffEqCore = "bbf590c4-e513-4bbe-9b18-05decba2e5d8"
 ...
-[sources.OrdinaryDiffEqCore]
-path = "../../../OrdinaryDiffEqCore"
-
 [compat]
 AllocCheck = "0.2"
 JET = "0.9, 0.11"
-julia = "1.10"
 OrdinaryDiffEqCore = "4"
 OrdinaryDiffEqSIMDRK = "2"
 SciMLBase = "3"
+julia = "1.10"
+
+[sources.OrdinaryDiffEqCore]
+path = "../../../OrdinaryDiffEqCore"

On the LTS the [sources] backport walks the graph, so DiffEqBase is injected as well. Repeating that same SIMDRK activation on Julia 1.10.11 with the patched SciMLTesting v2.7.0 ([sources] is ignored on 1.10, so the pin is a Pkg.develop in a driver env) leaves the tree clean:

┌ Info: versions
│   pkgversion(SciMLTesting) = v"2.7.0"
└   VERSION = v"1.10.11"
  Activating project at `~/claude_tmp/jl_sciml_group_env_FAVwmA`
  [09d9d899] + SciMLTesting v2.7.0 `~/…/wt-scimltesting46`

$ git status --porcelain      # empty

Sandboxing does not change Pkg's validation. A group env whose [sources] names a package missing from [deps]/[extras] is rejected identically either way (found while wiring the pin above):

sandbox=false : Sources for `Other` not listed in `deps` or `extras` section at ".../test/qa/Project.toml".
sandbox=true  : Sources for `Other` not listed in `deps` or `extras` section at ".../jl_sciml_group_env_mvf2PG/Project.toml".

Alternatives, and why they lose

Both of the narrower options in the issue were checked rather than assumed:

• Snapshot the project and restore it in a finally. It does not just leave a window — it breaks the lane. activate_group_env returns before the group's test files run, so restoring the pre-develop [deps] makes the developed parent unloadable. Verified directly: develop a package into an env, restore the original Project.toml, then using P → ArgumentError: Package P not found in current path.
• Only develop parents that are not already resolvable. Pkg does not normalize a project it did not change (verified: developing a parent already present in [deps] leaves the file byte-identical), so this would help the no-op case — but the case that actually mutates is the one where the parent genuinely must be added. In OrdinaryDiffEq's sublibrary QA envs the umbrella OrdinaryDiffEq is not in [deps], and developing it is the whole point of the lane, so this alternative cannot fix the reported bug.

Also checked

Every group env in OrdinaryDiffEq (qa, root AD/Downstream/ODEInterfaceRegression) that goes through activate_group_env gets the fix, as does DiffEqDevTools' QA env; run_tests' env group specs route through the same function.

Hand-rolled helpers still living in the repos bypass SciMLTesting entirely and are unaffected by this PR: OrdinaryDiffEqDifferentiation's activate_sparse_env/activate_modelingtoolkit_env, DiffEqBase's activate_static_env, and the various activate_gpu_envs all do their own Pkg.activate + Pkg.develop. The sparse/modelingtoolkit/gpu envs already list their parent in [deps], so they leave the file alone; lib/DiffEqBase/test/static/Project.toml does not list DiffEqBase, so activate_static_env has the same defect — the fix is to migrate those helpers to activate_group_env, which is a separate OrdinaryDiffEq PR.

Version

2.6.3 → 2.7.0. sandbox is a new public keyword argument and the default behavior of a public function changes (the activated project is a copy), so this is a minor bump, not a patch. 2.6.3 is not yet registered (General has 2.6.2).

Not verified

• Nothing on the CI matrix: local runs were Linux-only (Julia 1.10.11 / 1.11.9 / 1.12.6), but all 12 PR checks are green — Tests on {1, lts, pre} × {ubuntu, macos, windows}, Runic, typos, Downgrade.
• GPU group envs (no GPU on this machine) — they use raw Pkg.activate in the repos anyway and are untouched by this PR.
• Every other SciML repo's group envs; only OrdinaryDiffEq was exercised downstream.
• Committed test-env Manifest.tomls: I rewrite relative manifest path entries when copying, which the OrdinaryDiffEq lane does exercise for a generated manifest, but no repo I checked commits one.

Reviewer, push back on

• Always sandboxing, including for develop = false, instantiate = false (a pure activation writes nothing, so the copy is not strictly needed there). I chose the simpler invariant "the working tree is never written" over a conditional one.
• Memoizing the copy per process rather than making a fresh one per call, and never writing the resolved manifest back — a group env is re-resolved once per process now.