|
1 | | -# I want to use the autolens_assistant to perform Expectation Propagation |
| 1 | +# EP hierarchical power-law slope recovery — autolens_assistant science project |
2 | 2 |
|
3 | | -Type: docs |
4 | | -Target: PyAutoLens |
| 3 | +Type: research |
| 4 | +Target: graphical_ep |
5 | 5 | Repos: |
6 | | -- PyAutoLens |
7 | 6 | - autolens_assistant |
| 7 | +- PyAutoFit |
| 8 | +- PyAutoLens |
8 | 9 | - autolens_workspace |
9 | 10 | Difficulty: too-large |
10 | 11 | Autonomy: supervised |
11 | 12 | Priority: high |
12 | 13 | Status: formalised |
13 | 14 |
|
| 15 | +## Original request (verbatim) |
| 16 | + |
14 | 17 | I want to use the autolens_assistant to perform Expectation Propagation (EP) analysis of a Cosmology science case. |
15 | 18 |
|
16 | 19 | The science case is inferring the Hubble constant from time delay lensed quasars, and a example package already |
@@ -40,4 +43,77 @@ The goal are: |
40 | 43 |
|
41 | 44 | 4) To test all the graphical EP diagnostics and result analysis code, especially tyhe recent EP updates we did last week. |
42 | 45 |
|
43 | | -<!-- formalised by the Intake (Conception) Agent on 2026-07-16 from file:/tmp/claude-1000/-home-jammy-Code-PyAutoLabs/32708468-5918-4dbc-a763-583805364341/scratchpad/intake_ep_cosmology.md --> |
| 46 | +## Intake corrections & grounding (2026-07-16) |
| 47 | + |
| 48 | +**Corrected source paths** (verified on disk; the request has typos): |
| 49 | + |
| 50 | +- Legacy EP scripts: `/mnt/c/Users/Jammy/Science/concr/scripts/cosmology/` |
| 51 | + — `ep.py`, `graphical.py`, `one_by_one.py`, `simple_plot.py`. These are the |
| 52 | + "main scripts" to port: EP fit, joint graphical fit, and one-by-one fits. |
| 53 | +- Legacy simulator: `/mnt/c/Users/Jammy/Science/concr/simulators/cosmology.py`; |
| 54 | + an existing simulated dataset lives at `concr/dataset/cosmology__time_delay`. |
| 55 | +- Workspace examples: `autolens_workspace/scripts/guides/modeling/advanced/` |
| 56 | + — `expectation_propagation.py`, `graphical.py`, `hierarchical.py`. |
| 57 | +- Expect API drift in the `concr` package (old PyAutoLens); brush up against |
| 58 | + the installed stack, grounding via the PyAuto API gate / `dir()` rather than |
| 59 | + guessing. |
| 60 | + |
| 61 | +**Descoped goal (this task).** No H0 / time delays yet. Simulate N power-law + |
| 62 | +shear lenses with slopes drawn from a hierarchical (parent) distribution; |
| 63 | +example scripts recover the per-lens slopes and — the primary metric — the |
| 64 | +mean and scatter of the parent slope distribution. Keep the project layout |
| 65 | +H0-compatible so time-delay cosmography can be re-scoped in later. |
| 66 | + |
| 67 | +**Project home.** New science project stamped from |
| 68 | +`autolens_assistant/skills/start-new-project.md` (project skeleton: `scripts/`, |
| 69 | +`dataset/`, `results/`, `hpc/`, `wiki/project/profile.md`). Private science |
| 70 | +project, same pattern as PJ011646. |
| 71 | + |
| 72 | +**Goal 1 sampler — the `(SPECIFY)` decision, to settle at start_dev.** |
| 73 | +Candidates now first-class in PyAutoFit: multi-start JAX gradient optimizers |
| 74 | +(`MultiStartAdam` / ADABelief / Lion — promoted across lib + workspaces) for |
| 75 | +MAP-style recovery, and gradient-based posterior sampling (blackjax NUTS/HMC) |
| 76 | +for errors. The non-EP baseline must produce parent mean+scatter *with |
| 77 | +uncertainties*, so a sampling (not just optimization) path is needed for the |
| 78 | +comparison in goal 2. |
| 79 | + |
| 80 | +**Goal 4 context.** The "recent EP updates last week" are the diagnostics / |
| 81 | +result-analysis work shipped under `ep_framework_review.md` (this folder, |
| 82 | +execution-complete 2026-07-08, wrap-up on PyAutoFit#1330). This project is the |
| 83 | +end-to-end exercise of that shipped tooling on a realistic lensing case. |
| 84 | + |
| 85 | +**HPC (goal 3).** Run path is RAL via the assistant HPC link: |
| 86 | + |
| 87 | +- Cluster access: SSH alias `euclid_jump` (ProxyJump through `jump_finan`); |
| 88 | + projects under `/mnt/ral/jnightin/<project>`; PyAuto venv at |
| 89 | + `/mnt/ral/jnightin/PyAuto` kept in sync with local `main`s (`HPCPullPyAuto`), |
| 90 | + activated via `activate.sh` (never pip-install PyAuto* into the venv). |
| 91 | +- Project machinery: copy the assistant's `hpc/` (batch_cpu/, batch_gpu/, |
| 92 | + `sync`, `template.py`) into the new project per the start-new-project HPC |
| 93 | + step; docs at `autolens_assistant/wiki/core/operations/hpc_infrastructure.md`. |
| 94 | + Pipeline scripts must preserve the `template.py` HPC interface |
| 95 | + (`parse_fit_args`, `--sample`/`--dataset`/`--use_cpu`/`--number_of_cores`). |
| 96 | +- Drive runs with `hpc/sync push-submit gpu <script>` (SLURM array, one |
| 97 | + dataset per task, JAX auto-uses the GPU), then `hpc/sync jobs`/`tail`/`pull`. |
| 98 | + The legacy `concr/hpc/` uses the same layout — port its `sync.conf` pattern, |
| 99 | + not its stale scripts. |
| 100 | +- RAL `/mnt/ral` is NFS-slow: detach long remote work (nohup+setsid, poll a |
| 101 | + log), never foreground-`timeout` an ssh. |
| 102 | +- N-lens array scaling is the design driver: one lens per SLURM array task for |
| 103 | + the one-by-one/EP factor fits; the joint graphical fit needs a single |
| 104 | + larger-memory task. |
| 105 | + |
| 106 | +**Relationship to existing backlog** (this folder): |
| 107 | + |
| 108 | +- `ep_scoping.md` — EP per-fit overhead / scale-up (performance): directly |
| 109 | + relevant once N grows; do not duplicate its profiling work here. |
| 110 | +- `graphical_scoping.md` — joint-graph scale-up (performance): the joint-fit |
| 111 | + baseline in goal 1/2 will hit exactly the dimensionality limits it maps. |
| 112 | +- `ep_framework_review.md` — execution-complete; its diagnostics are what |
| 113 | + goal 4 validates. |
| 114 | + |
| 115 | +**Sizing.** too-large is right: expect phased delivery at start_dev |
| 116 | +(project stamp + simulator port → one-by-one/JAX-gradient baseline → EP fit + |
| 117 | +diagnostics → HPC scale-up), one PR per phase. |
| 118 | + |
| 119 | +<!-- formalised by the Intake (Conception) Agent on 2026-07-16 from file:/tmp/claude-1000/-home-jammy-Code-PyAutoLabs/32708468-5918-4dbc-a763-583805364341/scratchpad/intake_ep_cosmology.md; header + grounding hand-fixed at review (docs/autolens → research/graphical_ep) --> |
0 commit comments