Skip to content

feat(search): add batch_size to the multi-start gradient searches#1374

Merged
Jammy2211 merged 1 commit into
mainfrom
feature/multi-start-batch-size
Jul 15, 2026
Merged

feat(search): add batch_size to the multi-start gradient searches#1374
Jammy2211 merged 1 commit into
mainfrom
feature/multi-start-batch-size

Conversation

@Jammy2211

Copy link
Copy Markdown
Collaborator

Summary

The multi-start gradient searches (#1369/#1370) vmap every start's value_and_grad at once. For memory-heavy likelihoods that OOMs, with no knob to reduce it.

Measured on an A100 80GB, pixelized (kernel-CDF mesh) lens likelihood, n_starts=16, float64:

f64[16, 15361, 2, 31, 512] = 58.13 GiB  ->  RESOURCE_EXHAUSTED

The only workarounds were float32 (a science compromise) or fewer starts (defeats multi-start).

Fix

batch_size: Optional[int] = None on AbstractMultiStartGradient (inherited by MultiStartAdam / MultiStartADABelief / MultiStartLion).

  • None (default) → unchanged: one jax.vmap over all starts. No regression.
  • Set → tiling handed to jax.lax.map(..., batch_size=), JAX's own chunked map: vmaps within a chunk, scans across chunks, and handles a ragged final chunk without a second compile.
  • lax.map is used only when batching is requested — with batch_size=None it degrades to a sequential scan, which would destroy the default path's parallelism.

Only the memory-dominant gradient evaluation is tiled; the optax update still runs over all starts.

Not the same thing as Nautilus's n_batch

af.Nautilus's n_batch is Nautilus's own algorithmic knob (how many points it proposes per iteration) that autofit forwards. Here n_starts is the algorithm; batch_size is purely implementation-level tiling and is numerically inert.

(For the record: optax offers nothing for this — MultiSteps/apply_every are gradient accumulation over steps for a single parameter set fed many data batches. Our starts are independent parameter sets; accumulating across them would change the algorithm.)

Test Plan

  • test_autofit/ full suite: 1480 passed, 1 skipped (no regression)
  • Numerically inert through the real _fit path: best_fom/best_params identical for batch_size ∈ {None, 1, 4, 14, 100} — incl. ragged (14 % 4 == 2) and batch > n_starts
  • numpy-only plumbing tests (defaults None, carried by all three rules)
  • JAX equivalence assertion in autofit_workspace_test (follow-up, library-first)

Unblocks the pixelized multi-start experiment (autolens_workspace_developer#100), where the vram batch table can now be consumed by the gradient searches as it already is by Nautilus.

Generated by the PyAutoLabs agent workflow.

The multi-start searches vmap every start's value_and_grad at once. For a
memory-heavy likelihood that OOMs: measured on an A100 80GB with a pixelized
(kernel-CDF) lens likelihood at n_starts=16 in float64, the vmapped jvp fusion
is f64[16,15361,2,31,512] = 58.13 GiB -> RESOURCE_EXHAUSTED, with no knob to
reduce it (only fp32, a science compromise, or fewer starts, which defeats
multi-start).

batch_size=None (default) keeps the single vmap, so there is no regression.
When set, the tiling is handed to jax.lax.map(..., batch_size=), which vmaps
within a chunk and scans across chunks, handling a ragged final chunk without a
second compile. lax.map is used ONLY when batching is requested — with
batch_size=None it degrades to a sequential scan, which would throw away the
default path's parallelism.

batch_size is numerically inert: verified through _fit that best_fom and
best_params are identical for batch_size in {None,1,4,14,100} (incl. ragged
14%4==2 and batch>n_starts). Unlike af.Nautilus's n_batch — Nautilus's own
algorithmic knob that autofit forwards — this is purely implementation-level
tiling; n_starts remains the algorithm.

Resolves #1373.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Jammy2211 Jammy2211 added the pending-release PR queued for the next release build label Jul 15, 2026
@Jammy2211 Jammy2211 merged commit 7262f83 into main Jul 15, 2026
5 checks passed
@Jammy2211 Jammy2211 deleted the feature/multi-start-batch-size branch July 15, 2026 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pending-release PR queued for the next release build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant