maint: PreOptimizationTimes breakdown + README dashboard (polish 4)

[Jammy2211]

Overview

Phase 4 of 4 of the autolens_profiling polish series (parent: polish.md, phased 2026-07-08). Completes the PreOptimizationTimes baseline campaign with two deliverables: per-step likelihood_breakdown decompositions for one representative config per dataset kind, and the README dashboard designed in phase 1, populated with all key PreOptimizationTimes results. This locks in the final pre-optimization reference numbers before speed-up work begins.

Blocked by: phase 3 (profiling-preopt-campaign, #56) — needs its PreOptimizationTimes runtime baseline tagged and its autolens_profiling worktree claim released. Queued in PyAutoMind/planned.md until then.

Plan

• Wait for phase 3 to ship (baseline tagged, campaign branch merged, claim released).
• Run likelihood_breakdown per-step decompositions for HST (imaging), Alma_high (interferometer), and datacube on laptop CPU.
• Dispatch the same breakdown runs on HPC CPU and A100 (+ mixed precision where supported), reusing phase 3's infeasibility map to pre-scope cells.
• Save results per the phase-1 conventions, tagged PreOptimizationTimes (replacing the stale v2026.5.29.4 breakdown rows).
• Populate the README dashboard designed in phase 1 with all key PreOptimizationTimes results (runtime + breakdown), linking down into the per-package .md result pages.
• Verify dashboard values against the baseline JSONs; ship via ship_workspace.

Detailed implementation plan

Affected Repositories

• autolens_profiling (primary)

Branch Survey

Repository	Current Branch	Dirty?
./autolens_profiling	main	untracked results only (searches/, simulators/)

Claim conflict at survey time: autolens_profiling claimed by profiling-preopt-campaign (feature/profiling-preopt-campaign, ~/Code/PyAutoLabs-wt/profiling-preopt-campaign) — hard block until #56 ships.

Suggested branch: feature/preopt-breakdown-dashboard
Worktree root: ~/Code/PyAutoLabs-wt/preopt-breakdown-dashboard/ (created by /start_workspace once unblocked)
Classification: Workspace

Implementation Steps

1. Confirm phase 3 shipped: PreOptimizationTimes runtime baseline exists (build_baseline.py --name PreOptimizationTimes output), campaign branch merged, claim released.
2. Run breakdown sweeps in likelihood_breakdown/{imaging,interferometer,datacube}/ — one representative config each: HST (imaging), Alma_high (interferometer), datacube. Laptop CPU first.
3. Dispatch HPC CPU + A100 legs via hpc/ scripts when RAL is available; A100 mixed precision only where source supports it. Pre-scope with phase 3's infeasibility map (documented on maint: PreOptimizationTimes campaign — local-CPU leg (polish phase 3) #56: mge alma+ inherently blocked; ao mesh models laptop-infeasible; jvla simulation OOM).
4. Save per-step JSON + .md under results/breakdown/ per the phase-1 conventions; tag the snapshot PreOptimizationTimes.
5. Extend build_readme.py (phase 3 tooling) to fill the README <!-- BEGIN auto-table:headline --> block from the PreOptimizationTimes baseline — the runtime table (currently "No data yet") plus refreshed breakdown table — with links to results/{runtime,breakdown}/**/*.md.
6. Validation: rebuild README from baseline JSONs, check link integrity, and cross-check breakdown step-sums against phase-3 runtime totals for the same cells.

Key Files

• likelihood_breakdown/{imaging,interferometer,datacube}/ — per-step decomposition runners
• results/breakdown/ — output layout per phase-1 conventions
• results/runtime/ — phase-3 runtime results the dashboard also surfaces
• README.md — auto-table:headline dashboard block
• build_readme.py / build_baseline.py — phase-3 aggregation + snapshot tooling (on the campaign branch until merged)
• hpc/ — RAL dispatch scripts for HPC CPU / A100 legs

Out of Scope

• Searches; point_source; laptop GPU (user follow-up prompt); the future PyAutoBrain profiling agent (separate feature/pyautobrain/ prompt).

Original Prompt

Click to expand starting prompt

polish phase 4 — likelihood breakdown and README dashboard

Type: maintenance
Target: autolens_profiling
Difficulty: medium
Autonomy: supervised
Priority: normal
Status: formalised

Phase 4 of 4 of polish.md (see parent for full intent). Depends on phase 3
(PreOptimizationTimes runtime results in).

Two deliverables:

1. likelihood_breakdown — per-step decomposition for one representative
   config per dataset kind: HST for imaging, Alma_high for interferometer, and
   datacube. Same platforms as phase 3 (laptop CPU, HPC CPU, HPC A100 + mixed
   precision where supported); results saved per the phase-1 conventions and
   tagged PreOptimizationTimes.

2. README dashboard — populate the high-level dashboard designed in phase 1
   on the GitHub README, showing all key PreOptimizationTimes results at a
   glance, with links down into the per-package .md result pages.

Out of scope: searches; point_source; laptop GPU (user follow-up); the future
PyAutoBrain profiling agent (separate feature/pyautobrain/ prompt when its
time comes).

[Jammy2211]

Launched --auto 2026-07-10 (human-directed). Effective autonomy: supervised (header binds over the maintenance cap). Claim-override recorded: proceeds alongside the phase-3 campaign's autolens_profiling claim — phase 4 works likelihood_breakdown/ + README, phase 3 works likelihood_runtime/; merge order coordinated at ship.

Execution order: deliverable 1 (breakdown runs, laptop CPU first) proceeds now; HPC CPU/A100 legs dispatch when RAL is available; deliverable 2 (README dashboard) checkpoints if the phase-3 PreOptimizationTimes runtime baseline still isn't tagged when I get there. Ship parks at sign-off per the supervised contract.

[Jammy2211]

Campaign started (laptop-CPU leg) + one batched question (supervised checkpoint-and-continue — work proceeds on the non-blocked parts).

In flight now (sequential, laptop CPU fp64): imaging mge/pixelization/delaunay @ hst (dense + sparse where applicable), interferometer delaunay @ alma_high, datacube delaunay + inversion_setup_decompose @ alma_high. Filenames follow the canonical single-config pattern. One rider commit: interferometer/delaunay.py now honors --instrument (same pattern as the datacube scripts; default unchanged) — needed to run alma_high without editing the knob.

Q1 — platform matrix for breakdown. The phase-4 prompt says breakdown runs on the same platforms as phase 3 (laptop CPU, HPC CPU, A100 + mp). But the phase-1 locked design in likelihood_breakdown/README.md explicitly rejects multi-config breakdown: per-step JIT compile is ~an hour per cell per config, cross-hardware comparison is the runtime package's job, and the package's single multi-config knob is --gpu opt-in. Options:

• (a) honor the phase-1 design (my recommendation): breakdown = laptop CPU fp64 only (running now); the dashboard's cross-platform story comes from the runtime baseline; note the rationale on the README.
• (b) follow the prompt literally: dispatch HPC CPU + A100 (+mp) breakdown legs when RAL is up, with --config-name labels (~an hour of compile per cell per config).

Note (not blocking): datacube runs at alma_high per the parent prompt ("Alma_high for interferometer and datacube"), superseding the old sma row — say if you want sma kept alongside.

Next while runs proceed: results-note scaffolding; dashboard work checkpoints on the phase-3 PreOptimizationTimes baseline (build tooling lives on the campaign branch).

[Jammy2211]

Q1 answered (human, in-session): GPU-first breakdown doctrine. Breakdown should always aim for a full GPU decomposition; CPU fp64 is the fallback when no GPU is available, not the design center. Applied:

• likelihood_breakdown/README.md platform-policy section rewritten (commit 3bb613a) — canonical decomposition per cell = A100 fp64 + mp where supported, CPU fp64 = provisional fallback. (Also removes a stale --gpu flag reference; backend selection is via JAX env vars in the submits.)
• 15 hpc/batch_gpu/submit_breakdown_* scripts committed — the 8 PreOptimizationTimes cells at hpc_a100_fp64 + hpc_a100_mp where the source supports it (all but imaging/mge). 4h walltime each (compile-bound).
• HPC-CPU breakdown leg dropped — CPU-only decomposition is the low-value case; the cross-platform CPU story stays in likelihood_runtime.

RAL status: login node is back up, but sinfo shows both GPU nodes down — the A100 leg is dispatch-ready but parked until nodes return. When they do: sync the branch to /mnt/ral/jnightin/autolens_profiling (HPCPullPyAuto), sbatch the 15 submits, commit result JSONs back.

Laptop-CPU leg in flight (now the fallback tier): mge @ hst done (rc=0, 150s); pixelization running; 6 cells behind it.

[Jammy2211]

Laptop-CPU leg: imaging complete (5/5 clean), alma_high cells on retry.

Cell (hst)	Path	Step-sum total	Top step
imaging/mge	dense	0.95 s	Blurred mapping matrix (59%)
imaging/pixelization	dense	28.1 s	Curvature matrix F (49%)
imaging/pixelization	sparse	26.0 s	Curvature matrix F (45%)
imaging/delaunay	dense	28.3 s	Curvature matrix F (50%)
imaging/delaunay	sparse	33.4 s	Curvature matrix F (51%)

Headline finding — real slowdown since May. All cells sit 2.5–5.3× above the May (v2026.5.29.4) rows, with byte-identical cell configurations (same mask/15361 px, 1500 vertices, 39×39 mesh). Cross-checking against phase 3's quiet-machine runtime numbers isolates it: full-pipeline single-JIT hst pixelization is now 13.7 s — above May's 7.8 s step-sum upper bound — so there is a genuine ≥ ~1.8× library-level slowdown on mesh cells since late May, independent of any contention in my runs. Curvature matrix (F) dominates everywhere (~50%). This is exactly what a PreOptimizationTimes baseline is for; the results note will carry the per-step deltas, and the optimization phase owns the bisect.

Caveat recorded: these step-sums were measured under ambient session load (laptop at 12/15 GB from parallel work) — treat absolute laptop-CPU values as fallback-tier per the GPU-first doctrine; the A100 leg is the canonical decomposition.

alma_high failures classified + fixed: all three shared one root cause — the fresh worktree lacked dataset/interferometer/alma_high and auto-simulate was OOM-killed under the ambient memory load. Seeded the 234 MB simulated dataset from the phase-3 worktree (identical simulation inputs); retry in flight.

[Jammy2211]

CORRECTION — the "real ≥1.8× library slowdown" claim two comments up is retracted. The per-step deltas were suspiciously uniform (every step 2–5×, including trivial ones), which pointed at environment, not code. Found it:

~/venv/PyAuto/bin/activate:74 exports XLA_FLAGS="--xla_disable_hlo_passes=constant_folding" — a global leftover from the pure_callback const-fold debugging. Control run: imaging/mge with the flag = 0.946 s; without = 0.292 s (3.2×). Against the May row (0.178 s) the residual is ~1.6×, attributable to thread pinning/ambient load — no library-regression claim survives.

The repo's documented convention is vmap for honesty (latent/README.md, datacube shared_preloads.py) — nothing in the repo prescribes the XLA flag, and production users don't run with it, so flagged timings overstate real cost.

Actions taken:

• _profile_cli.device_info_dict now records xla_flags / omp_num_threads / cpu_count in every result JSON (7edf06a) — drift like this becomes attributable at a glance.
• The 5 imaging cells are rerunning flag-free (background); the committed v2026.7.6.649 JSONs/READMEs will be replaced with corrected numbers.
• A with/without control pair on likelihood_runtime/imaging/mge.py is running to quantify the impact on the phase-3 runtime matrix — cross-posting to maint: PreOptimizationTimes campaign — local-CPU leg (polish phase 3) #56, since the entire local-CPU PreOptimizationTimes runtime leg (2026-07-08/09, post-flag) is likely inflated the same way, while the A100 rows (HPC activate.sh, no flag) are not — skewing local-vs-A100 ratios.

Batched question (non-blocking): the venv-activate export is your environment file, so I haven't touched it — should it be removed in favour of the repo's vmap convention (my recommendation), or kept as deliberate policy? If kept, it needs documenting in the methodology READMEs and the May rows re-flagged as non-comparable.

[Jammy2211]

Reconciliation — the earlier correction over-corrected. Reading PyAutoConf/autoconf/jax_wrapper.py settles it: the library itself sets XLA_FLAGS=--xla_disable_hlo_passes=constant_folding at import (unconditionally, when absent) and explicitly recommends users export it in their terminal — it exists for JAX compile-time sanity. The venv-activate export is the library's own recommendation, not a leak — withdraw my suggestion to remove it.

Clean measurements (back-to-back control pair on likelihood_runtime/imaging/mge.py, equal load):

• Flag effect on single-JIT: 1.54× (0.141 s no-flag → 0.218 s flag) — disabling constant folding trades steady-state cost for compile time, as designed.
• vmap per-call: insensitive (0.110 s both ways) — consistent with the repo's vmap-is-honest doctrine; batch tracers leave nothing to fold.

Revised attribution for the original 2.5–5.3× vs May: mostly CPU contention (the first campaign ran with the laptop at 12–13/15 GB from parallel sessions; the flag-free rerun on a quieting machine came back 0.87–1.9× of May). My earlier "3.2× flag effect" control was confounded — it ran minutes after I killed the NUFFT process, so it measured contention relief.

Two side findings:

1. Provenance subtlety: autoconf re-sets the env var in-process even when the shell didn't — so xla_flags in the result JSON can show the flag while XLA never saw it (backend init order decides). The recorded field is the env at write time; treat shell-level provenance as canonical. (The A100 submits don't export it either — they rely on the same ambiguous in-process set; worth making explicit there.)
2. check_pinned on main crashes on vmap arrays (float(got) on batch results) — already fixed on the phase-3 campaign branch (e2578ae); the runtime controls' rc=1 is that, post-measurement.

Final baseline run in flight: the 5 imaging cells re-running under the canonical policy env (flag explicitly exported, uniform with May/phase-3/venv convention) on a now-quiet machine (load 0.6, 8 GB free), with provenance recorded. Those numbers become the PreOptimizationTimes laptop-CPU fallback rows.

[Jammy2211]

Laptop-CPU baseline FINAL + A100 tier DISPATCHED.

Final fallback-tier rows (canonical policy env — explicit XLA_FLAGS — quiet machine, load 0.6):

Cell (hst)	Path	Step-sum	vs May	Top step
imaging/mge	dense	179.5 ms	1.01×	Blurred mapping matrix (59%)
imaging/pixelization	dense	8.65 s	1.11×	Curvature F (48%)
imaging/pixelization	sparse	10.17 s	1.14×	Curvature F (46%)
imaging/delaunay	dense	10.07 s	0.89×	Curvature F (42%)
imaging/delaunay	sparse	8.81 s	1.11×	Curvature F (48%)

Verdict: no library drift since May (0.89–1.14× = scatter). F-matrix is the optimization target on every mesh cell; MGE is convolution-bound. Baseline + methodology (contention signature, flag provenance, GPU-only classification of alma_high) in results/notes/preopt_breakdown_baseline.md; READMEs regenerated. Branch feature/preopt-breakdown-dashboard pushed (029dcc4).

RAL GPU nodes returned — all 15 A100 breakdown jobs dispatched: 330058–330072 (datacube trio already RUNNING). Repo synced to /mnt/ral/jnightin/autolens_profiling (code only; datasets were already present). Note: HPC libraries are main @ 2026.7.9.1 vs local 2026.7.6.649 — per-JSON version fields keep the tiers attributable.

Remaining: ingest A100 results when jobs finish (≤4 h walltime each) → commit → the README dashboard leg (gated on phase 3 tagging the runtime baseline) → ship sign-off (parks for human per the supervised contract).

Side note for the humans: hpc/batch_gpu/output/output.330046.out (the NNLS A100 addendum flagged as pending in the ledger) exists on RAL — worth ingesting while you're there.

[Jammy2211]

A100 tier progress: imaging COMPLETE (9/9), datacube landing, interferometer alma_high blocked by a real library gap.

Imaging (jobs 330062–330070), committed d4f63a8 — headline validates the GPU-first policy: the bottleneck moves on GPU.

Cell (hst)	A100 fp64	vs laptop CPU	Top step on A100 (CPU comparison)
mge dense	7.8 ms	23×	Blurred mapping matrix — same as CPU
pixelization dense/sparse	~58 ms	150–176×	Regularized reconstruction (NNLS) ~65% — CPU said F-matrix
delaunay dense/sparse	~97 ms	90–103×	Inversion setup ~42% — CPU said F-matrix

Optimization driven from the CPU decomposition would have targeted the wrong step on the cells that matter. Mixed precision is flat (≤2%) at hst scale. build_readme.py now renders config-tagged artifacts with a Platform column (--check idempotence passes).

alma_high fix + failure taxonomy: the first 6 alma_high jobs died on missing nufftax in the RAL venv (third-party → pip-installed 0.6.1 per the venv convention; note the submit template prints "Finished." even on failure — no set -e — worth a phase-3-tooling fix). On resubmit:

• datacube: WORKING — inversion_setup_decompose fp64 done (inversion_hpc_a100_fp64.json), 3 jobs still running.
• interferometer delaunay: OOM on the A100 — the inversion-matrix extraction NUFFTs all 1500 mapping-matrix columns onto the 1600² fine grid at once: 1600×1600×1500×16 B = exactly the failed 61.44 GB allocation. The configured 1M-visibility chunking chunks the gather, not the columns. Phase 3 had already marked this cell GPU-only for runtime (never yet measured anywhere); breakdown needs column-chunked mapping-matrix NUFFT — a PyAutoArray/nufftax-boundary feature, out of phase-4 scope. Proposing a follow-up prompt maintenance/autolens_profiling/ → PyAutoArray for it; until then the cell is classified gpu_unusable_breakdown (extraction path; the fused runtime path may still fit — phase 3's A100 runtime legs will tell).

[Jammy2211]

Measured A100 four-way split of the delaunay inversion-setup block (job 330079, --split-setup prefix-JIT decomposition; eager regression PASSED; prefix-sum 39.96 ms vs 41.14 ms combined = faithful, no fusion redistribution):

Piece	A100	share of block	share of full likelihood (96.6 ms)
Border relocation	0.90 ms	2%	~1%
Triangulation + interpolation	26.62 ms	65%	~27%
Mapping matrix build	6.34 ms	15%	~7%
Blurred mapping matrix (PSF)	6.10 ms	15%	~6%

Conclusion for the optimization phase: the single largest target in the whole A100 delaunay likelihood is the triangulation + interpolation step (~27%), ahead of the NNLS solve (~34% delaunay / ~65% pixelization — see the solver ledger for why NNLS has no algorithmic lever left). Within triangulation+interpolation, the qhull host callback is a few ms at most (1500 points + transfer), so the work item is the JAX-side visibility-walk point location + barycentric interpolation — gather-heavy, divergence-heavy GPU code. Candidate levers: better initial-simplex seeding to shorten walks, fixed-iteration batch-friendly walk restructure, or caching the triangulation across nearby evaluations.

CPU contrast (same split, laptop): the block is convolution-dominated there (PSF-convolved mapping matrix 3.37 s of 3.88 s) — a third demonstration that CPU decompositions point at the wrong GPU target.

--split-setup is opt-in and committed; the canonical JSON now carries setup_split.