PreOptimizationTimes breakdown + dashboard — A100 tier, baseline column, four-way split#63
Merged
Merged
Conversation
#59) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Q1 decision (human, 2026-07-10): breakdown always aims for a full GPU decomposition; CPU fp64 is the fallback when no GPU is available, not the design center. 15 submit_breakdown_* scripts cover the 8 PreOptimizationTimes cells at hpc_a100_fp64 + hpc_a100_mp where the source supports it. The HPC-CPU breakdown leg is dropped (CPU-only decomposition adds little; cross-platform CPU story lives in likelihood_runtime). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ing 5/5 (#59) Fresh per-step decompositions at v2026.7.6.649 for the hst imaging cells (mge dense; pixelization + delaunay dense and sparse). READMEs regenerated; the render also picks up the previously-committed-but-unrendered cluster rows (README drift on main). Running-section --gpu flag reference replaced with the real env-var invocation + submit-script pointer. alma_high interferometer/datacube cells: classified GPU-only tier on laptop CPU — the NUFFT precision operator exceeded the 2h timeout twice under ambient load (matches the phase-3 infeasibility map); they land via the A100 submits when RAL GPU nodes return. Headline: all cells sit 2.5-5.3x above the May v2026.5.29.4 rows with byte-identical cell configurations; cross-check against phase-3 quiet-machine runtime confirms a real >=~1.8x library-level slowdown on mesh cells since late May. Details in the results note. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A stray env XLA_FLAGS or thread pin rescales every timing by integer factors; recording them makes cross-run drift attributable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The autoconf in-process set is order-dependent; the explicit shell export guarantees the canonical env on the A100 tier, uniform with local runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s + results note (#59) Imaging cells re-measured under the canonical policy env (XLA_FLAGS explicitly exported per autoconf policy) on a quiet machine: 0.89-1.14x of the May rows — no library drift; F-matrix dominates every mesh cell (~42-48%), MGE is convolution-bound. READMEs regenerated. results/notes/preopt_breakdown_baseline.md records the baseline, the GPU-only classification of the alma_high cells (NUFFT >2h on CPU), and the measurement methodology: explicit shell-level XLA_FLAGS (in-process set is init-order-dependent), quiet machine mandatory (uniform per-step inflation is the contention signature — the first pass read 2.5-5.3x slow under load and was nearly misread as a regression), flag effect 1.54x single-JIT / vmap-insensitive. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…#59) Ingest the 9 A100 imaging results (jobs 330062-330070): 23-176x over laptop CPU, and the bottleneck MOVES on GPU — CPU's F-matrix dominance (42-48%) gives way to the NNLS regularized reconstruction on pixelization (~65%) and inversion setup on delaunay. This is the empirical case for the GPU-first breakdown policy. Mixed precision flat (<=2%) at hst scale. build_readme.py learns config-tagged breakdown artifacts (<script>_<config>[_sparse].json; instrument/version read from the payload) and the breakdown table gains a Platform column. --check idempotence passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-setup block (#59) Nested prefix-JITs (params -> step-5/6/7/8 outputs) attribute the combined 41.9ms A100 block to border relocation / triangulation+interpolation / mapping matrix / PSF convolution before optimization targets are chosen. Opt-in flag; canonical steps list unchanged; split lands in the JSON as setup_split. Validated locally (eager regression assert passes). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
4 results (delaunay + inversion_setup_decompose x fp64/mp, 34 channels): per-channel inversion ~182ms and almost fully channel-invariant (variant ~0-2ms) — the cube amortizes setup across channels; mp flat. The datacube path avoids the dense column-NUFFT that OOMs the interferometer cell. Datacube JSONs use their own schema (no total_step_by_step) so the README table shows no step-sum for them — schema unification left for phase-3 tooling alignment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lock (#59) Job 330079 (--split-setup): border relocation 0.90ms / triangulation + interpolation 26.62ms / mapping matrix 6.34ms / PSF-convolved mapping matrix 6.10ms. Prefix-sum 39.96ms vs combined 41.14ms — the decomposition is faithful (no fusion redistribution). The Delaunay-specific triangulation+interpolation is the single largest optimization target of the whole A100 likelihood (~27%); the qhull host callback is a few ms of it at most, so the JAX-side point location / barycentric interpolation is the work item. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…own-dashboard # Conflicts: # README.md # likelihood_breakdown/README.md # simulators/README.md
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the phase-4 (
likelihood_breakdown+ dashboard) leg of thepolish.mdPreOptimizationTimes campaign (#59). The phase-3 runtime baseline (#56/#62) is now on main, so this rebases/merges onto it and wires the baseline column into the dashboard.What landed
--split-setup): triangulation + interpolation ≈ 26.6 ms ≈ ~27% of the full likelihood → the top optimization target. NNLS split confirmed 65% pix / 34% delaunay.build_readme.pynow renders config-tagged A100 artifacts with a Platform column, and grows aPreOptimizationTimesbaseline column in the runtime table automatically fromresults/baselines/. All READMEs regenerated;--checkidempotence passes; ruff clean.results/notes/preopt_breakdown_baseline.md(contention signature, flag provenance, per-step deltas). Verdict: no library drift since May (0.89–1.14× = scatter); F-matrix dominates mesh cells, MGE is convolution-bound.Known gap (follow-up filed)
interferometer delaunay @ alma_highisgpu_unusable_breakdown: the inversion-matrix extraction NUFFTs all 1500 mapping-matrix columns onto the 1600² fine grid at once = 61.44 GB (fp64), OOMing the A100; the existing 1M-visibility chunking chunks the gather, not the columns. Filed as a PyAutoArray feature prompt (feature/autoarray/nufft_mapping_matrix_column_chunking.md) for column-chunked mapping-matrix NUFFT. The fused runtime path is separately GPU-only and unaffected by this.Checks
build_readme.py --checkidempotence: pass (all READMEs unchanged on re-run).🤖 Generated with Claude Code