feat(search): add batch_size to the multi-start gradient searches#1374
Merged
Conversation
The multi-start searches vmap every start's value_and_grad at once. For a
memory-heavy likelihood that OOMs: measured on an A100 80GB with a pixelized
(kernel-CDF) lens likelihood at n_starts=16 in float64, the vmapped jvp fusion
is f64[16,15361,2,31,512] = 58.13 GiB -> RESOURCE_EXHAUSTED, with no knob to
reduce it (only fp32, a science compromise, or fewer starts, which defeats
multi-start).
batch_size=None (default) keeps the single vmap, so there is no regression.
When set, the tiling is handed to jax.lax.map(..., batch_size=), which vmaps
within a chunk and scans across chunks, handling a ragged final chunk without a
second compile. lax.map is used ONLY when batching is requested — with
batch_size=None it degrades to a sequential scan, which would throw away the
default path's parallelism.
batch_size is numerically inert: verified through _fit that best_fom and
best_params are identical for batch_size in {None,1,4,14,100} (incl. ragged
14%4==2 and batch>n_starts). Unlike af.Nautilus's n_batch — Nautilus's own
algorithmic knob that autofit forwards — this is purely implementation-level
tiling; n_starts remains the algorithm.
Resolves #1373.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The multi-start gradient searches (#1369/#1370) vmap every start's
value_and_gradat once. For memory-heavy likelihoods that OOMs, with no knob to reduce it.Measured on an A100 80GB, pixelized (kernel-CDF mesh) lens likelihood,
n_starts=16, float64:The only workarounds were float32 (a science compromise) or fewer starts (defeats multi-start).
Fix
batch_size: Optional[int] = NoneonAbstractMultiStartGradient(inherited byMultiStartAdam/MultiStartADABelief/MultiStartLion).None(default) → unchanged: onejax.vmapover all starts. No regression.jax.lax.map(..., batch_size=), JAX's own chunked map: vmaps within a chunk, scans across chunks, and handles a ragged final chunk without a second compile.lax.mapis used only when batching is requested — withbatch_size=Noneit degrades to a sequential scan, which would destroy the default path's parallelism.Only the memory-dominant gradient evaluation is tiled; the optax update still runs over all starts.
Not the same thing as Nautilus's
n_batchaf.Nautilus'sn_batchis Nautilus's own algorithmic knob (how many points it proposes per iteration) that autofit forwards. Heren_startsis the algorithm;batch_sizeis purely implementation-level tiling and is numerically inert.(For the record: optax offers nothing for this —
MultiSteps/apply_everyare gradient accumulation over steps for a single parameter set fed many data batches. Our starts are independent parameter sets; accumulating across them would change the algorithm.)Test Plan
test_autofit/full suite: 1480 passed, 1 skipped (no regression)_fitpath:best_fom/best_paramsidentical forbatch_size∈ {None, 1, 4, 14, 100} — incl. ragged (14 % 4 == 2) and batch > n_startsNone, carried by all three rules)Unblocks the pixelized multi-start experiment (autolens_workspace_developer#100), where the vram batch table can now be consumed by the gradient searches as it already is by Nautilus.
Generated by the PyAutoLabs agent workflow.