prompt: pixelized-gradient-experiment - batch_size shipped, gradient settled, search question open, Nautilus baseline running

Author: Jammy2211

active.md

@@ -3,12 +3,16 @@
 
 ## pixelized-gradient-experiment
 - issue: https://github.com/PyAutoLabs/autolens_workspace_developer/issues/100
-- status: in-progress, PAUSED for bedtime 2026-07-14 — feasibility ANSWERED (pix gradients DO work; certified test imaging_pixelization.py passed exit 0). RESUME = build the sampler experiment on the CORRECT config.
-- worktree: ~/Code/PyAutoLabs-wt/pixelized-gradient-experiment (autolens_workspace_developer on feature/pixelized-gradient-experiment; probes + pix_gradient_findings.md committed+pushed, NOT PR'd)
+- status: in-progress — A100 pipeline WORKING. Gradient question SETTLED (yes). Search question OPEN. Nautilus baseline job 330379 running on RAL (submitted 2026-07-15, 4h limit).
+- worktree: ~/Code/PyAutoLabs-wt/pixelized-gradient-experiment (autolens_workspace_developer on feature/pixelized-gradient-experiment, pushed, NOT PR'd)
 - autonomy: supervised (research)
-- finding: can af.MultiStartAdam/ADABelief/Lion work for a pixelized source? YES — pix likelihoods are gradient-differentiable. My first probe's "no" was a methodology error (human caught it; certified in autolens_workspace_test/scripts/jax_grad/imaging_pixelization.py, re-run passed). Correct config = kernel-CDF mesh RectangularKernelAdaptDensity(bandwidth=0.1) [os_pix=1] OR adaptive mesh at over_sample_size_pixelization=4; truth-centred GaussianPriors; small FD step sweep near truth.
-- resume: STEP 1 DONE-ish — searches_minimal/probe_grad_pix.py REWRITTEN to the certified recipe (commit 227c4c0): kernel-CDF mesh, explicit over_sample_size_pixelization, SLaM-pix-1 model (MGE linear light geometry FIXED at truth, free Isothermal+shear, free reg), truth-centred GaussianPriors + small FD step sweep (1e-8..1e-6) near truth. Truth from jax_profiling/simulators/imaging.py: einstein_radius=1.6, centre=(0,0), q=0.9/45deg, shear=(0.05,0.05). BUT NOT YET RUN TO A VERDICT — local CPU is too slow (kernel-CDF pixelized JAX x64 ~10min+ compile then ~40 FD evals); runs were terminated. NEXT: (1) run the probe on GPU/A100 (or shrink mask/mesh) → expect OK; (2) build the sampler experiment on that objective and run af.MultiStartAdam/ADABelief/Lion + af.Nautilus baseline; (3) A100 on RAL. REAL question = can multi-start gradient descent from BROAD starts recover the mass basin vs Nautilus (gradient correctness is settled=yes, certified test passes). See [[project_pixelized_gradient_sampler_infeasible]] memory.
-- note: broad multi-starts are the crux — the certified gradients are validated NEAR TRUTH; the samplers deliberately start broad, where the pixelized landscape may be pathological. That tension is the experiment's actual finding to measure.
+- SETTLED: pix likelihoods ARE gradient-differentiable. A100 FD probe (kernel-CDF RectangularKernelAdaptDensity(bandwidth=0.1), os_pix=1, x64): every mass/shear param FD-matched ~1e-6 (einstein_radius rel=8.8e-7), logL +25537 at a truth-centred point. My earlier "no" was a methodology error (human caught it). NEVER use adaptive meshes at os_pix=1 (certified staircase = dead mass gradient); kernel-CDF is live at os_pix=1, adaptive needs os_pix=4.
+- SHIPPED: PyAutoFit#1374 batch_size (merged 7262f832) — unbatched pix multi-start OOMs 80GB A100 (58.13 GiB jvp fusion); batch_size=4 completed a full f64 fit. Do NOT use fp32 (science compromise) or apply_sparse_operator (human: separate question, may slow the likelihood).
+- OPEN (the real question): adam ran ONCE — wall 2090s, max logL -39888, einstein_radius 1.4169 (truth 1.6), per-start basin 0/16. "IN TRUTH BASIN: True" is an ARTIFACT of a slack 0.3 tol — do not trust it. logL -39888 vs the probe's +25537 at truth = the optimizer did NOT find the basin. Suspects: n_steps=300 too few from broad starts; lr=1e-2 mis-scaled; my per-start diagnostic indexing may be wrong.
+- resume: (1) read Nautilus 330379 result (/mnt/ral/jnightin/pixgrad_logs/samp_pixgrad_nautilus_330379.log) — if Nautilus ALSO misses the basin, the model/priors are at fault, not the gradient optimizer; (2) then debug adam (steps/lr/diagnostics); (3) then adabelief + lion. Only adam has run — no ADABelief/Lion/Nautilus results yet.
+- RAL: shared /mnt/ral/jnightin/PyAuto/PyAutoFit left CLEAN on main (human's release version-bumps in README/docs/paper preserved). My isolated PyAutoFit worktree /mnt/ral/jnightin/pixgrad_pyautofit (batch_size branch, now merged to main) is PYTHONPATH-prepended by the sbatch and still used by job 330379 — safe to delete once it finishes, then the shared mirror on main suffices.
+- TRAPS: foreground `timeout ssh` does NOT kill the remote side (it severed a git op, leaving a stale index.lock + half-applied checkout in the SHARED mirror) — use nohup+setsid+sentinel. Nautilus on a JAX row MUST set force_x1_cpu=True + use_jax_vmap=True (else fork corrupts JAX state) and n_batch<=16 without the sparse operator (default 100 needs ~100GB).
+- intakes filed: draft/feature/autolens_profiling/jax_compile_time_profiling.md (Fable, tomorrow — compile is ~all the wall time; autotune RULED OUT: 2100s vs 2090s identical), draft/refactor/autofit/split_fitness_batch_size_lh_vs_latent.md
 - repos:
   - autolens_workspace_developer: feature/pixelized-gradient-experiment