Skip to content

fix(config): enable JAX for the JAX-native searches scripts#45

Merged
Jammy2211 merged 2 commits into
mainfrom
feature/multistart-adam-release-jax
Jul 15, 2026
Merged

fix(config): enable JAX for the JAX-native searches scripts#45
Jammy2211 merged 2 commits into
mainfrom
feature/multistart-adam-release-jax

Conversation

@Jammy2211

@Jammy2211 Jammy2211 commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Fixes #44.

What broke

nightly-release has been red five nights running (2026-07-11 → 2026-07-15). Stage 3 dispatches PyAutoHeart workspace-validation.yml at mode=release, and exactly one shard fails — run_scripts (3.12, autofit_test, searches):

scripts/searches/MultiStartAdam.py ...   FAIL (1.7s)
ValueError: MultiStartAdam is a JAX-native gradient search and requires a JAX-traceable
Analysis (e.g. `AnalysisImaging(..., use_jax=True)`). The supplied analysis is not running
on the JAX backend.

Every other shard in that run is green. This is not the shifting perf-flake tail from the release-tail epic — it is deterministic, and it is fallout from the multi-start gradient search promotion (PyAutoFit #1370, this repo's #43).

Root cause

MultiStartAdam._fit hard-guards on the backend:

if not getattr(analysis, "_use_jax", False):
    raise ValueError(...)

config/build/env_vars_release.yaml — the mode=release profile, not env_vars.yaml which is the per-PR smoke gate — pins PYAUTO_TEST_MODE: "0" (so _fit really runs, no bypass) and PYAUTO_DISABLE_JAX: "1" (so use_jax=True is silently flipped to False). The guard fires. The profile re-enables JAX for exactly two patterns — jax_assertions/ and searches/BlackJAXNUTS — and searches/MultiStartAdam was never added.

autofit_workspace/scripts/searches/mle.py uses the same search and passes only because that workspace's release profile pins PYAUTO_DISABLE_JAX: "0". The autolens/autogalaxy guides construct the search but never .fit(), so they never reach the guard.

Changes

Config only — no script logic, no library source, no public API.

config/build/env_vars_release.yaml — three set: { PYAUTO_DISABLE_JAX: "0" } overrides, following the searches/BlackJAXNUTS precedent (this profile's header documents that overrides only ever set:, never unset:, a var defaults pins):

  • searches/MultiStartAdam — the blocking fix.
  • searches/Dynesty_jax, searches/Nautilus_jax — these declare use_jax=True but were also missing from the overrides, so they validated the NumPy path while presenting as JAX tests. Unlike MultiStartAdam they carry no backend guard, so they passed vacuously and nothing failed loudly — the same failure mode the profile's own jax_assertions/ note describes.

config/build/env_vars.yaml (the smoke env profile) — one unset: [PYAUTO_TEST_MODE, PYAUTO_DISABLE_JAX] for searches/MultiStartAdam. Its PYAUTO_TEST_MODE=2 bypasses the sampler before _fit, so the script passed without ever executing its JAX path. Running it costs ~6s. Scoped to MultiStartAdam; the nested samplers stay bypassed under this profile on cost grounds.

What this profile does and does not gate (an earlier revision of this PR body got this wrong — see the correction comment). env_vars.yaml feeds PyAutoHeart's workspace-validation.yml at mode=smoke, which runs every script weekly (Mondays 03:00 UTC). It does not feed the per-PR gate — that is .github/workflows/smoke_tests.yml running the curated smoke_tests.txt list, which contains neither MultiStartAdam nor BlackJAXNUTS. So this override buys a weekly net on the JAX path, not a per-PR one. Genuine per-PR coverage would mean promoting the script into smoke_tests.txt — a separate decision, against the standing "keep the curated subset small" preference, deliberately not taken here.

Verification

All run on this branch with the release-profile env (PYAUTO_TEST_MODE=0 PYAUTO_DISABLE_JAX=0 JAX_ENABLE_X64=True):

Script Result
MultiStartAdam.py exit 0 — recovered centre=50.156, normalization=25.196, sigma=9.858 (truth 50/25/10)
Dynesty_jax.py exit 0 (~10s)
Nautilus_jax.py exit 0 (~25s)
MultiStartAdam.py (smoke-resolved env, both vars unset) exit 0 (~6s)

No bug was hiding behind the two vacuous passes, so all three release overrides are safe to ship.

Resolver check. _pattern_matches substring-matches against the full path including extension, so an under-specified pattern silently over-matches (searches/Nautilus would also catch Nautilus_jax.py). Drove autobuild.env_config.build_env_for_script over the real configs for every scripts/searches/*.py:

===== env_vars.yaml (smoke) =====        ===== env_vars_release.yaml =====
script              TEST_MODE  JAX       script              TEST_MODE  JAX
BlackJAXNUTS.py       <unset>  <unset>   BlackJAXNUTS.py             0    0
MultiStartAdam.py     <unset>  <unset>   MultiStartAdam.py           0    0
Dynesty_jax.py              2        1   Dynesty_jax.py              0    0
Nautilus_jax.py             2        1   Nautilus_jax.py             0    0
Nautilus.py                 2        1   Nautilus.py                 0    1
DynestyStatic.py            2        1   DynestyStatic.py            0    1
DynestyDynamic.py           2        1   DynestyDynamic.py           0    1
Emcee.py / LBFGS.py         2        1   Emcee.py / LBFGS.py         0    1
Zeus.py                     2        1   Zeus.py                     0    1

Nautilus.py and DynestyStatic/Dynamic.py correctly stay at DISABLE_JAX=1 — no over-match.

Ship-gate legs

  1. Tests — n/a: no library test dir in this repo; the scripts are the tests, and every affected one was executed directly (above).
  2. Smoke — the changed surface is the searches shard; env resolution verified against the real configs and the three JAX scripts run end-to-end. The PR's own smoke (3.12/3.13) checks pass but do not exercise MultiStartAdam (it is not in smoke_tests.txt), so they are not evidence for this change; the direct runs above are.
  3. Review — 23 lines of config + rationale comments.
  4. Heart — YELLOW score 55 at ship time. Its first reason, "workspace validation not passing", is what this PR fixes.

Human signed off on the ship in-session (autonomy: supervised, the bug cap). Merge stays human.

🤖 Generated with Claude Code

The mode=release profile (env_vars_release.yaml) pins PYAUTO_DISABLE_JAX=1
alongside PYAUTO_TEST_MODE=0, so searches scripts really run but on the NumPy
backend. MultiStartAdam is JAX-native and hard-raises when the analysis is not
JAX-traceable, so it has failed the autofit_test/searches shard — and with it
nightly-release Stage 3 — every night since it landed in #43.

Add the missing `set: { PYAUTO_DISABLE_JAX: "0" }` overrides, following the
searches/BlackJAXNUTS precedent. Dynesty_jax and Nautilus_jax get the same
treatment: they declare use_jax=True but were also absent from the overrides,
so they were validating the NumPy path while presenting as JAX tests — passing
vacuously, since unlike MultiStartAdam they carry no backend guard.

Also stop the smoke profile bypassing MultiStartAdam. Its PYAUTO_TEST_MODE=2
skips the sampler before _fit runs, so the script passed without executing its
JAX path at all — which is why the per-PR gate could not catch this and it
surfaced only in mode=release. Running it costs ~6s.

Verified with JAX enabled: MultiStartAdam exit 0 (recovers 50.156/25.196/9.858
vs truth 50/25/10), Dynesty_jax exit 0, Nautilus_jax exit 0, and MultiStartAdam
under the smoke-resolved env exit 0. Env resolution driven through
autobuild.env_config.build_env_for_script over the real configs for every
scripts/searches/*.py: Nautilus.py and DynestyStatic/Dynamic.py correctly stay
at DISABLE_JAX=1, confirming the substring patterns do not over-match.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Jammy2211 Jammy2211 added the pending-release PR queued for the next release build label Jul 15, 2026
@Jammy2211

Copy link
Copy Markdown
Collaborator Author

⚠️ Correction — the smoke-profile rationale in the PR body and commit message is wrong

I claimed the env_vars.yaml change makes the per-PR gate catch this class of bug. That is not true, and I should correct it before merge rather than let it stand in main's history.

What I got wrong. There are two distinct things both loosely called "smoke":

  1. The per-PR gate.github/workflows/smoke_tests.yml, which runs the curated smoke_tests.txt subset (11 scripts: Emcee, DynestyStatic, Nautilus, features/*, database/*, …). Neither MultiStartAdam nor BlackJAXNUTS is in that list. This gate never runs either script, so the env_vars.yaml override does not affect it at all.
  2. The smoke env profileconfig/build/env_vars.yaml, consumed by PyAutoHeart's workspace-validation.yml at mode=smoke, which runs every script (weekly, Mondays 03:00 UTC, plus manual dispatch).

The env_vars.yaml change lands on (2), not (1).

What the change is actually worth. Still real, just on a different cadence than I said. Before it, the weekly full-validation run executed MultiStartAdam.py with PYAUTO_TEST_MODE=2, bypassing the sampler before _fit — a vacuous pass. After it, that run really executes the JAX path, so a backend regression is caught within a week on the weekly validation, instead of only at release time. That still satisfies the intent ("it should not stay bypassed if running it is what ensures the bug is caught") — it just is not per-PR.

It also means the searches/BlackJAXNUTS precedent I copied is itself about the full-script-set runs, not the per-PR gate. My reading of it was right; my description of what it gates was not.

Nothing about the diff changes. All four overrides remain correct and verified, and the fix for the blocking failure (the release profile) is unaffected — that was always the load-bearing part.

The open judgement I am NOT deciding unilaterally: getting genuine per-PR coverage would mean adding searches/MultiStartAdam.py to smoke_tests.txt. That is promoting an integration script into the deliberately-small curated subset, which cuts against the standing "smoke tests are a small curated subset — don't mass-promote" preference. Worth a separate decision on its own merits; say the word and I will file it.

CI status on this PR: smoke (3.12) and smoke (3.13) both SUCCESS — though per the above, those runs did not exercise MultiStartAdam and are not evidence for this change.

The comment added in 615b229 claimed running MultiStartAdam under this profile
lets "the per-PR gate" see it. That is wrong, and the wrong claim would have
landed in main.

Two different things are called smoke here:

  1. The per-PR gate is .github/workflows/smoke_tests.yml, which runs the
     curated smoke_tests.txt subset. Neither MultiStartAdam nor BlackJAXNUTS
     is in that list, so this profile never touches it.
  2. env_vars.yaml is the `smoke` env profile consumed by PyAutoHeart's
     workspace-validation.yml at mode=smoke, which runs every script weekly.

The override lands on (2). The change is still worth keeping — the weekly sweep
now really executes the JAX path instead of bypassing it, so a backend
regression is caught within a week rather than at release — but on a weekly
cadence, not per-PR. Correct the comment to say so, and record that the
BlackJAXNUTS precedent above is likewise about the full-script-set runs.

No behaviour change: comment only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Jammy2211 Jammy2211 merged commit ad95dd8 into main Jul 15, 2026
4 checks passed
@Jammy2211 Jammy2211 deleted the feature/multistart-adam-release-jax branch July 15, 2026 07:55
Jammy2211 added a commit that referenced this pull request Jul 15, 2026
PYAUTO_DISABLE_JAX is read in exactly one place that matters to this workspace:
af.Analysis.__init__, where it forces use_jax=False. Every analysis here is
af.ex.Analysis or a local subclass, and af.Analysis defaults to use_jax=False.

So the "1" default was a no-op for the ~35 scripts that never opt in - they run
numpy either way - and silently sabotaged the ~7 that explicitly pass
use_jax=True, which are exactly the scripts written to exercise JAX. It bought
no speed and cost correctness.

Its only real effect was to require a hand-maintained "remember to add your
script" override list. That list is the trap: a JAX-native script that forgets
it does not fail loudly, it silently validates numpy and passes. Dynesty_jax
and Nautilus_jax did that for as long as they have existed; MultiStartAdam only
failed loudly (nightly-release red five nights, #44) because it happens to
carry its own backend guard.

Pin "0" and delete all five overrides, which become no-ops. This matches the
sibling autofit_workspace release profile, which has always pinned "0" for the
same release-fidelity reason - this repo was the inconsistent one. PR #45 fixed
the symptom by adding the three missing entries; this removes the reason the
list exists.

Verified rather than asserted: resolved every scripts/**/*.py through
autobuild.env_config.build_env_for_script under both profiles (31 change 1->0,
11 unchanged), then confirmed behaviour empirically -
features/latent_nan_robustness.py (opts in; JAX half runs for the first time)
exit 0, and Nautilus.py / Emcee.py / latent.py / model_composition.py all exit
0 identically under DISABLE_JAX=1 and =0.

Closes #46

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pending-release PR queued for the next release build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(config): release profile disables JAX, breaking MultiStartAdam validation shard

1 participant