PreOptimizationTimes breakdown + dashboard — A100 tier, baseline column, four-way split

README.md

@@ -45,15 +45,28 @@ Results are framed by **astronomy instrument** (HST, Euclid, JWST, …) rather t
 
 **Likelihood breakdown** — latest per-step decompositions:
 
-| Cell | Instrument | Inversion path | Step-sum total | PyAutoLens version |
-|------|------------|----------------|----------------|--------------------|
-| `cluster/image_plane` | — | dense (mapping) | 4.99 s | v2026.7.6.649 |
-| `cluster/source_plane` | — | dense (mapping) | 3.1 ms | v2026.7.6.649 |
-| `imaging/delaunay` | hst | dense (mapping) | 11.29 s | v2026.5.29.4 |
-| `imaging/delaunay` | hst | sparse (w-tilde) | 7.96 s | v2026.5.29.4 |
-| `imaging/mge` | hst | dense (mapping) | 178.2 ms | v2026.5.29.4 |
-| `imaging/pixelization` | hst | dense (mapping) | 7.79 s | v2026.5.29.4 |
-| `imaging/pixelization` | hst | sparse (w-tilde) | 8.94 s | v2026.5.29.4 |
+| Cell | Instrument | Platform | Inversion path | Step-sum total | PyAutoLens version |
+|------|------------|----------|----------------|----------------|--------------------|
+| `cluster/image_plane` | — | local_cpu_fp64 | dense (mapping) | 4.99 s | v2026.7.6.649 |
+| `cluster/source_plane` | — | local_cpu_fp64 | dense (mapping) | 3.1 ms | v2026.7.6.649 |
+| `datacube/delaunay` | alma_high | hpc_a100_fp64 | dense (mapping) | — | v2026.7.6.649 |
+| `datacube/delaunay` | alma_high | hpc_a100_mp | dense (mapping) | — | v2026.7.6.649 |
+| `datacube/inversion` | alma_high | hpc_a100_fp64 | dense (mapping) | — | v2026.7.6.649 |
+| `datacube/inversion` | alma_high | hpc_a100_mp | dense (mapping) | — | v2026.7.6.649 |
+| `imaging/delaunay` | hst | local_cpu_fp64 | dense (mapping) | 10.07 s | v2026.7.6.649 |
+| `imaging/delaunay` | hst | local_cpu_fp64 | sparse (w-tilde) | 8.81 s | v2026.7.6.649 |
+| `imaging/delaunay` | hst | hpc_a100_fp64 | dense (mapping) | 96.6 ms | v2026.7.6.649 |
+| `imaging/delaunay` | hst | hpc_a100_fp64 | sparse (w-tilde) | 98.0 ms | v2026.7.6.649 |
+| `imaging/delaunay` | hst | hpc_a100_mp | dense (mapping) | 96.8 ms | v2026.7.6.649 |
+| `imaging/delaunay` | hst | hpc_a100_mp | sparse (w-tilde) | 95.5 ms | v2026.7.6.649 |
+| `imaging/mge` | hst | local_cpu_fp64 | dense (mapping) | 179.5 ms | v2026.7.6.649 |
+| `imaging/mge` | hst | hpc_a100_fp64 | dense (mapping) | 7.8 ms | v2026.7.6.649 |
+| `imaging/pixelization` | hst | local_cpu_fp64 | dense (mapping) | 8.65 s | v2026.7.6.649 |
+| `imaging/pixelization` | hst | local_cpu_fp64 | sparse (w-tilde) | 10.17 s | v2026.7.6.649 |
+| `imaging/pixelization` | hst | hpc_a100_fp64 | dense (mapping) | 57.6 ms | v2026.7.6.649 |
+| `imaging/pixelization` | hst | hpc_a100_fp64 | sparse (w-tilde) | 57.9 ms | v2026.7.6.649 |
+| `imaging/pixelization` | hst | hpc_a100_mp | dense (mapping) | 56.4 ms | v2026.7.6.649 |
+| `imaging/pixelization` | hst | hpc_a100_mp | sparse (w-tilde) | 55.5 ms | v2026.7.6.649 |
 <!-- END auto-table:headline -->
 
 The tables above are auto-generated by `scripts/build_readme.py` from the artifacts under [`results/`](./results/README.md) — never edit them by hand; run `python scripts/build_readme.py` after a profiling run and commit the result (CI checks idempotence via `--check`). Narrative context — per-cell "where to optimize next" recommendations and the mp-vs-fp64 verdicts — lives in [`likelihood_runtime/OPTIMIZATION_NOTES.md`](./likelihood_runtime/OPTIMIZATION_NOTES.md).
@@ -108,7 +121,7 @@ This repo is being built in phases (bootstrap history now archived in `PyAutoMin
 | 4 | Top-level + per-section README dashboard with instrument framing | ✓ shipped |
 | 5 | GitHub Actions for lint + profile re-runs + README refresh | ✓ shipped (`lint.yml` per-PR; `profile.yml` manual/on-release) |
 | 6 | Design lock-in + results/dashboard groundwork ([#52](https://github.com/PyAutoLabs/autolens_profiling/issues/52)) | in progress |
-| 7 | **PreOptimizationTimes** baseline campaign (vram-first, then runtime + breakdown) | queued |
+| 7 | **PreOptimizationTimes** baseline campaign (vram-first, then runtime + breakdown) | ✓ shipped (runtime [#56](https://github.com/PyAutoLabs/autolens_profiling/issues/56); breakdown + dashboard [#59](https://github.com/PyAutoLabs/autolens_profiling/issues/59); laptop-GPU legs extend in a later re-run) |
 
 ### Future enhancements (Phase 4 follow-ups)
 

_profile_cli.py

@@ -146,6 +146,13 @@ def device_info_dict() -> dict:
     info = {
         "backend": jax.default_backend(),
         "device": str(jax.devices()[0]),
+        # Environment provenance: a stray XLA_FLAGS (e.g. disabling
+        # constant_folding) or thread pinning silently rescales every timing
+        # in a result by integer factors — record them so drift between runs
+        # is attributable (found the hard way: autolens_profiling#59).
+        "xla_flags": os.environ.get("XLA_FLAGS") or None,
+        "omp_num_threads": os.environ.get("OMP_NUM_THREADS") or None,
+        "cpu_count": os.cpu_count(),
     }
     if info["backend"] == "gpu":
         try:

hpc/batch_gpu/submit_breakdown_datacube_delaunay_a100_alma_high_fp64

@@ -0,0 +1,48 @@
+#!/bin/bash -l
+#
+# A100 likelihood-breakdown profile: datacube/delaunay x alma_high x fp64.
+# DENSE inversion path (mapping matrices).
+# Per-step JIT decomposition (PreOptimizationTimes campaign, autolens_profiling#59).
+# Compile-heavy: every pipeline step is JIT-compiled in isolation, so wall
+# time is dominated by XLA compiles, not the steady-state timing loop.
+
+#SBATCH -J breakdown_datacube_delaunay_alma_high_fp64
+#SBATCH --partition=gpu
+#SBATCH --gres=gpu:1
+#SBATCH --ntasks=1
+#SBATCH --cpus-per-task=4
+#SBATCH --mem=64gb
+#SBATCH --time=4:00:00
+#SBATCH -o output/output.%A.out
+#SBATCH -e error/error.%A.err
+#SBATCH --mail-type=END,FAIL
+#SBATCH --mail-user=james.w.nightingale@durham.ac.uk
+
+export AP_ROOT=/mnt/ral/jnightin/autolens_profiling
+source $AP_ROOT/activate.sh
+
+export JAX_PLATFORM_NAME=cuda
+export JAX_PLATFORMS=cuda,cpu
+export XLA_PYTHON_CLIENT_PREALLOCATE=false
+export JAX_ENABLE_X64=True
+export XLA_FLAGS="--xla_disable_hlo_passes=constant_folding"
+export NUMBA_CACHE_DIR=/tmp/numba_cache
+export MPLCONFIGDIR=/tmp/matplotlib
+
+nvidia-smi
+
+echo "=========================================="
+date
+echo "Cell:           likelihood_breakdown/datacube/delaunay"
+echo "Instrument:     alma_high"
+echo "Precision:      fp64"
+echo "Inversion path: dense (mapping)"
+
+cd $AP_ROOT
+
+python3 likelihood_breakdown/datacube/delaunay.py \
+    --config-name hpc_a100_fp64 \
+    --instrument alma_high
+
+echo "Finished."
+date

hpc/batch_gpu/submit_breakdown_datacube_delaunay_a100_alma_high_mp

@@ -0,0 +1,49 @@
+#!/bin/bash -l
+#
+# A100 likelihood-breakdown profile: datacube/delaunay x alma_high x mp.
+# DENSE inversion path (mapping matrices).
+# Per-step JIT decomposition (PreOptimizationTimes campaign, autolens_profiling#59).
+# Compile-heavy: every pipeline step is JIT-compiled in isolation, so wall
+# time is dominated by XLA compiles, not the steady-state timing loop.
+
+#SBATCH -J breakdown_datacube_delaunay_alma_high_mp
+#SBATCH --partition=gpu
+#SBATCH --gres=gpu:1
+#SBATCH --ntasks=1
+#SBATCH --cpus-per-task=4
+#SBATCH --mem=64gb
+#SBATCH --time=4:00:00
+#SBATCH -o output/output.%A.out
+#SBATCH -e error/error.%A.err
+#SBATCH --mail-type=END,FAIL
+#SBATCH --mail-user=james.w.nightingale@durham.ac.uk
+
+export AP_ROOT=/mnt/ral/jnightin/autolens_profiling
+source $AP_ROOT/activate.sh
+
+export JAX_PLATFORM_NAME=cuda
+export JAX_PLATFORMS=cuda,cpu
+export XLA_PYTHON_CLIENT_PREALLOCATE=false
+export JAX_ENABLE_X64=True
+export XLA_FLAGS="--xla_disable_hlo_passes=constant_folding"
+export NUMBA_CACHE_DIR=/tmp/numba_cache
+export MPLCONFIGDIR=/tmp/matplotlib
+
+nvidia-smi
+
+echo "=========================================="
+date
+echo "Cell:           likelihood_breakdown/datacube/delaunay"
+echo "Instrument:     alma_high"
+echo "Precision:      mp"
+echo "Inversion path: dense (mapping)"
+
+cd $AP_ROOT
+
+python3 likelihood_breakdown/datacube/delaunay.py \
+    --config-name hpc_a100_mp \
+    --instrument alma_high \
+    --use-mixed-precision
+
+echo "Finished."
+date

hpc/batch_gpu/submit_breakdown_datacube_inversion_setup_decompose_a100_alma_high_fp64

@@ -0,0 +1,48 @@
+#!/bin/bash -l
+#
+# A100 likelihood-breakdown profile: datacube/inversion_setup_decompose x alma_high x fp64.
+# DENSE inversion path (mapping matrices).
+# Per-step JIT decomposition (PreOptimizationTimes campaign, autolens_profiling#59).
+# Compile-heavy: every pipeline step is JIT-compiled in isolation, so wall
+# time is dominated by XLA compiles, not the steady-state timing loop.
+
+#SBATCH -J breakdown_datacube_inversion_setup_decompose_alma_high_fp64
+#SBATCH --partition=gpu
+#SBATCH --gres=gpu:1
+#SBATCH --ntasks=1
+#SBATCH --cpus-per-task=4
+#SBATCH --mem=64gb
+#SBATCH --time=4:00:00
+#SBATCH -o output/output.%A.out
+#SBATCH -e error/error.%A.err
+#SBATCH --mail-type=END,FAIL
+#SBATCH --mail-user=james.w.nightingale@durham.ac.uk
+
+export AP_ROOT=/mnt/ral/jnightin/autolens_profiling
+source $AP_ROOT/activate.sh
+
+export JAX_PLATFORM_NAME=cuda
+export JAX_PLATFORMS=cuda,cpu
+export XLA_PYTHON_CLIENT_PREALLOCATE=false
+export JAX_ENABLE_X64=True
+export XLA_FLAGS="--xla_disable_hlo_passes=constant_folding"
+export NUMBA_CACHE_DIR=/tmp/numba_cache
+export MPLCONFIGDIR=/tmp/matplotlib
+
+nvidia-smi
+
+echo "=========================================="
+date
+echo "Cell:           likelihood_breakdown/datacube/inversion_setup_decompose"
+echo "Instrument:     alma_high"
+echo "Precision:      fp64"
+echo "Inversion path: dense (mapping)"
+
+cd $AP_ROOT
+
+python3 likelihood_breakdown/datacube/inversion_setup_decompose.py \
+    --config-name hpc_a100_fp64 \
+    --instrument alma_high
+
+echo "Finished."
+date

hpc/batch_gpu/submit_breakdown_datacube_inversion_setup_decompose_a100_alma_high_mp

@@ -0,0 +1,49 @@
+#!/bin/bash -l
+#
+# A100 likelihood-breakdown profile: datacube/inversion_setup_decompose x alma_high x mp.
+# DENSE inversion path (mapping matrices).
+# Per-step JIT decomposition (PreOptimizationTimes campaign, autolens_profiling#59).
+# Compile-heavy: every pipeline step is JIT-compiled in isolation, so wall
+# time is dominated by XLA compiles, not the steady-state timing loop.
+
+#SBATCH -J breakdown_datacube_inversion_setup_decompose_alma_high_mp
+#SBATCH --partition=gpu
+#SBATCH --gres=gpu:1
+#SBATCH --ntasks=1
+#SBATCH --cpus-per-task=4
+#SBATCH --mem=64gb
+#SBATCH --time=4:00:00
+#SBATCH -o output/output.%A.out
+#SBATCH -e error/error.%A.err
+#SBATCH --mail-type=END,FAIL
+#SBATCH --mail-user=james.w.nightingale@durham.ac.uk
+
+export AP_ROOT=/mnt/ral/jnightin/autolens_profiling
+source $AP_ROOT/activate.sh
+
+export JAX_PLATFORM_NAME=cuda
+export JAX_PLATFORMS=cuda,cpu
+export XLA_PYTHON_CLIENT_PREALLOCATE=false
+export JAX_ENABLE_X64=True
+export XLA_FLAGS="--xla_disable_hlo_passes=constant_folding"
+export NUMBA_CACHE_DIR=/tmp/numba_cache
+export MPLCONFIGDIR=/tmp/matplotlib
+
+nvidia-smi
+
+echo "=========================================="
+date
+echo "Cell:           likelihood_breakdown/datacube/inversion_setup_decompose"
+echo "Instrument:     alma_high"
+echo "Precision:      mp"
+echo "Inversion path: dense (mapping)"
+
+cd $AP_ROOT
+
+python3 likelihood_breakdown/datacube/inversion_setup_decompose.py \
+    --config-name hpc_a100_mp \
+    --instrument alma_high \
+    --use-mixed-precision
+
+echo "Finished."
+date

hpc/batch_gpu/submit_breakdown_imaging_delaunay_a100_hst_fp64

@@ -0,0 +1,47 @@
+#!/bin/bash -l
+#
+# A100 likelihood-breakdown profile: imaging/delaunay x hst x fp64.
+# DENSE inversion path (mapping matrices).
+# Per-step JIT decomposition (PreOptimizationTimes campaign, autolens_profiling#59).
+# Compile-heavy: every pipeline step is JIT-compiled in isolation, so wall
+# time is dominated by XLA compiles, not the steady-state timing loop.
+
+#SBATCH -J breakdown_imaging_delaunay_hst_fp64
+#SBATCH --partition=gpu
+#SBATCH --gres=gpu:1
+#SBATCH --ntasks=1
+#SBATCH --cpus-per-task=4
+#SBATCH --mem=64gb
+#SBATCH --time=4:00:00
+#SBATCH -o output/output.%A.out
+#SBATCH -e error/error.%A.err
+#SBATCH --mail-type=END,FAIL
+#SBATCH --mail-user=james.w.nightingale@durham.ac.uk
+
+export AP_ROOT=/mnt/ral/jnightin/autolens_profiling
+source $AP_ROOT/activate.sh
+
+export JAX_PLATFORM_NAME=cuda
+export JAX_PLATFORMS=cuda,cpu
+export XLA_PYTHON_CLIENT_PREALLOCATE=false
+export JAX_ENABLE_X64=True
+export XLA_FLAGS="--xla_disable_hlo_passes=constant_folding"
+export NUMBA_CACHE_DIR=/tmp/numba_cache
+export MPLCONFIGDIR=/tmp/matplotlib
+
+nvidia-smi
+
+echo "=========================================="
+date
+echo "Cell:           likelihood_breakdown/imaging/delaunay"
+echo "Instrument:     hst"
+echo "Precision:      fp64"
+echo "Inversion path: dense (mapping)"
+
+cd $AP_ROOT
+
+python3 likelihood_breakdown/imaging/delaunay.py \
+    --config-name hpc_a100_fp64
+
+echo "Finished."
+date

hpc/batch_gpu/submit_breakdown_imaging_delaunay_a100_hst_fp64_sparse

@@ -0,0 +1,48 @@
+#!/bin/bash -l
+#
+# A100 likelihood-breakdown profile: imaging/delaunay x hst x fp64.
+# SPARSE inversion path (w-tilde operator).
+# Per-step JIT decomposition (PreOptimizationTimes campaign, autolens_profiling#59).
+# Compile-heavy: every pipeline step is JIT-compiled in isolation, so wall
+# time is dominated by XLA compiles, not the steady-state timing loop.
+
+#SBATCH -J breakdown_imaging_delaunay_hst_fp64_sparse
+#SBATCH --partition=gpu
+#SBATCH --gres=gpu:1
+#SBATCH --ntasks=1
+#SBATCH --cpus-per-task=4
+#SBATCH --mem=64gb
+#SBATCH --time=4:00:00
+#SBATCH -o output/output.%A.out
+#SBATCH -e error/error.%A.err
+#SBATCH --mail-type=END,FAIL
+#SBATCH --mail-user=james.w.nightingale@durham.ac.uk
+
+export AP_ROOT=/mnt/ral/jnightin/autolens_profiling
+source $AP_ROOT/activate.sh
+
+export JAX_PLATFORM_NAME=cuda
+export JAX_PLATFORMS=cuda,cpu
+export XLA_PYTHON_CLIENT_PREALLOCATE=false
+export JAX_ENABLE_X64=True
+export XLA_FLAGS="--xla_disable_hlo_passes=constant_folding"
+export NUMBA_CACHE_DIR=/tmp/numba_cache
+export MPLCONFIGDIR=/tmp/matplotlib
+
+nvidia-smi
+
+echo "=========================================="
+date
+echo "Cell:           likelihood_breakdown/imaging/delaunay"
+echo "Instrument:     hst"
+echo "Precision:      fp64"
+echo "Inversion path: sparse (w-tilde)"
+
+cd $AP_ROOT
+
+python3 likelihood_breakdown/imaging/delaunay.py \
+    --config-name hpc_a100_fp64 \
+    --sparse
+
+echo "Finished."
+date