From a4f04f87d530d67ed64a085dd10cc121aef0d5bf Mon Sep 17 00:00:00 2001 From: Jammy2211 Date: Thu, 16 Jul 2026 09:36:09 +0100 Subject: [PATCH 1/2] =?UTF-8?q?feat:=20euclid=20mode=20=E2=80=94=20five=20?= =?UTF-8?q?euclid=5F*=20skills=20pairing=20the=20Euclid=20pipeline=20(#73?= =?UTF-8?q?=20phase=201)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds the euclid_.md domain-mode skill family pairing euclid_strong_lens_modeling_pipeline to the assistant: - euclid_setup_pipeline — install, dataset layout, black-box start_here.py - euclid_prepare_data — segmentation validation, binary masks, extra-galaxy GUI tools, hand-assembled dataset folders - euclid_model_lens — the staged pipelines (initial MGE+SIE, Sersic photometry, lens-only subtraction, multi-waveband, full SLaM) and when to use each - euclid_workflow_products — workflow/ csv/fits/png aggregator products - euclid_hpc_runs — hpc/sync config + SLURM batch-array submission Registers the family in skills/README.md (new naming convention line + index section), adds .claude/skills symlinks, a euclid-mode routing note in AGENTS.md (skills layer + fourth sub-wiki), and the pipeline repo in sources.yaml so Project:path citations resolve. The wiki/euclid/ pages the skills reference land in the phase-2 commit. Co-Authored-By: Claude Fable 5 --- .claude/skills/euclid_hpc_runs.md | 1 + .claude/skills/euclid_model_lens.md | 1 + .claude/skills/euclid_prepare_data.md | 1 + .claude/skills/euclid_setup_pipeline.md | 1 + .claude/skills/euclid_workflow_products.md | 1 + AGENTS.md | 16 ++- skills/README.md | 22 ++++ skills/euclid_hpc_runs.md | 83 ++++++++++++++ skills/euclid_model_lens.md | 122 +++++++++++++++++++++ skills/euclid_prepare_data.md | 103 +++++++++++++++++ skills/euclid_setup_pipeline.md | 118 ++++++++++++++++++++ skills/euclid_workflow_products.md | 70 ++++++++++++ sources.yaml | 5 + 13 files changed, 539 insertions(+), 5 deletions(-) create mode 120000 .claude/skills/euclid_hpc_runs.md create mode 120000 .claude/skills/euclid_model_lens.md create mode 120000 .claude/skills/euclid_prepare_data.md create mode 120000 .claude/skills/euclid_setup_pipeline.md create mode 120000 .claude/skills/euclid_workflow_products.md create mode 100644 skills/euclid_hpc_runs.md create mode 100644 skills/euclid_model_lens.md create mode 100644 skills/euclid_prepare_data.md create mode 100644 skills/euclid_setup_pipeline.md create mode 100644 skills/euclid_workflow_products.md diff --git a/.claude/skills/euclid_hpc_runs.md b/.claude/skills/euclid_hpc_runs.md new file mode 120000 index 0000000..427abbc --- /dev/null +++ b/.claude/skills/euclid_hpc_runs.md @@ -0,0 +1 @@ +../../skills/euclid_hpc_runs.md \ No newline at end of file diff --git a/.claude/skills/euclid_model_lens.md b/.claude/skills/euclid_model_lens.md new file mode 120000 index 0000000..64bb812 --- /dev/null +++ b/.claude/skills/euclid_model_lens.md @@ -0,0 +1 @@ +../../skills/euclid_model_lens.md \ No newline at end of file diff --git a/.claude/skills/euclid_prepare_data.md b/.claude/skills/euclid_prepare_data.md new file mode 120000 index 0000000..5b857ea --- /dev/null +++ b/.claude/skills/euclid_prepare_data.md @@ -0,0 +1 @@ +../../skills/euclid_prepare_data.md \ No newline at end of file diff --git a/.claude/skills/euclid_setup_pipeline.md b/.claude/skills/euclid_setup_pipeline.md new file mode 120000 index 0000000..4e0edd1 --- /dev/null +++ b/.claude/skills/euclid_setup_pipeline.md @@ -0,0 +1 @@ +../../skills/euclid_setup_pipeline.md \ No newline at end of file diff --git a/.claude/skills/euclid_workflow_products.md b/.claude/skills/euclid_workflow_products.md new file mode 120000 index 0000000..eba6a18 --- /dev/null +++ b/.claude/skills/euclid_workflow_products.md @@ -0,0 +1 @@ +../../skills/euclid_workflow_products.md \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md index 2f57535..b3b3c51 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -65,19 +65,25 @@ Map every request onto one or more layers: 2. **Skills** (`skills/*.md`, symlinked into `.claude/skills/`) — *procedural*: how to do a task. Lensing skills are `al_.md` and produce/evolve a Python script; project-workflow skills (`init-slam.md`, `start-new-project.md`) drive repo-level - operations. Skills starting with `_` (`_style.md`, `_bootstrap_skill.md`) are - meta-skills — don't surface them when answering science questions. + operations; **euclid mode** skills are `euclid_.md` — they pair the + collaboration's `euclid_strong_lens_modeling_pipeline` repo to the assistant, and any + request about modeling *Euclid* data routes through them (entry: + `skills/euclid_setup_pipeline.md`). Skills starting with `_` (`_style.md`, + `_bootstrap_skill.md`) are meta-skills — don't surface them when answering science + questions. 3. **Wiki** (`wiki/**/*.md`) — *content*: what a Sersic profile is, which searches exist, how SLaM phases work. > **Rule of thumb.** *How do I do X?* → a skill. *What / which / why X?* → the wiki. *Build > something end-to-end?* → compose skills, citing wiki pages as you go. -The wiki has three sub-wikis: **`wiki/core/`** (curated PyAuto\* reference, read-only — +The wiki has four sub-wikis: **`wiki/core/`** (curated PyAuto\* reference, read-only — refreshed by `al_update_wiki`), **`wiki/literature/`** (strong-lensing science reference, own schema in [`wiki/literature/AGENTS.md`](./wiki/literature/AGENTS.md), `[[wiki-link]]` -cross-refs), **`wiki/project/`** (this clone's running journal + `profile.md`). "The wiki" -means `wiki/core/` unless `literature/` or `project/` is named. +cross-refs), **`wiki/euclid/`** (Euclid mission + Euclid strong-lensing literature, +same schema as `literature/`, paired with the `euclid_*` skills), **`wiki/project/`** +(this clone's running journal + `profile.md`). "The wiki" means `wiki/core/` unless +`literature/`, `euclid/` or `project/` is named. --- diff --git a/skills/README.md b/skills/README.md index d66ee8e..fe5228f 100644 --- a/skills/README.md +++ b/skills/README.md @@ -12,6 +12,8 @@ configured) via symlinks; the canonical files live here. - File names use the `al_` convention for lensing-API skills, e.g. `al_run_search.md`. - Project-workflow skills (repo-level operations, template manipulation) use a plain kebab-case name, e.g. `init-slam.md`, `start-new-project.md`. +- Domain-mode skills that pair an external pipeline repo to the assistant use that + domain as prefix — `euclid_.md` for the Euclid pipeline (euclid mode). - Meta-skills (writing guide, bootstrap protocol) start with `_`. - Every lensing-API skill is **python-first**: the deliverable is a runnable `.py` script + the understanding to evolve it. Project-workflow skills may instead drive `rsync`, @@ -66,6 +68,26 @@ them. pipeline script(s) copied from `autolens_workspace` and tailored to the chosen data type. +### Euclid mode (pipeline-paired) + +Skills pairing the collaboration's +[`euclid_strong_lens_modeling_pipeline`](https://github.com/PyAutoLabs/euclid_strong_lens_modeling_pipeline) +to the assistant; the paired literature context is the dedicated +[`wiki/euclid/`](../wiki/euclid/index.md) sub-wiki. + +- [`euclid_setup_pipeline.md`](./euclid_setup_pipeline.md) — clone/install the pipeline, + the `dataset///` layout, and the black-box `start_here.py` run. +- [`euclid_prepare_data.md`](./euclid_prepare_data.md) — segmentation validation, binary + mask tuning, extra-galaxy masks/centres via the GUI tools, hand-assembling a dataset + folder. +- [`euclid_model_lens.md`](./euclid_model_lens.md) — choose and run the staged + pipelines: initial MGE+SIE, Sersic photometry, lens-only subtraction, multi-waveband, + full SLaM. +- [`euclid_workflow_products.md`](./euclid_workflow_products.md) — aggregate many fits + into .csv catalogues, .fits stacks, and one-line .png summaries via `workflow/`. +- [`euclid_hpc_runs.md`](./euclid_hpc_runs.md) — `hpc/sync` configuration and the + SLURM batch-array templates for sample-scale runs. + ### Data preparation - [`al_prepare_imaging_data.md`](./al_prepare_imaging_data.md) — load and preprocess diff --git a/skills/euclid_hpc_runs.md b/skills/euclid_hpc_runs.md new file mode 100644 index 0000000..1f0d754 --- /dev/null +++ b/skills/euclid_hpc_runs.md @@ -0,0 +1,83 @@ +--- +name: euclid_hpc_runs +description: Run the Euclid pipeline on a supercomputer — configure hpc/sync (push code/data, pull results), adapt the batch_cpu / batch_gpu SLURM array templates (one dataset per array task), submit, monitor, and pull results back. Use when a Euclid lens sample outgrows the local machine; NOT for single interactive fits (euclid_setup_pipeline) or generic HPC concepts (wiki/core/operations/hpc.md). +--- + +# Running Euclid samples on HPC + +One lens is a laptop job; a sample is not. The pipeline ships an HPC workflow built on +two pieces: a **sync CLI** that mirrors the project to the cluster and pulls results +back, and **SLURM array templates** that run one dataset per array task — the natural +unit, since every pipeline stage takes `--sample`/`--dataset` arguments and lenses are +independent. GPU nodes are the fast path (JAX accelerates the likelihood ~50×; one GPU +per task is enough); CPU templates exist for clusters without them. Generic +HPC/JAX/SLURM concepts live in +[`wiki/core/operations/hpc.md`](../wiki/core/operations/hpc.md) — this skill is the +Euclid-pipeline application of them. + +## Ask + +- *"Which cluster, and is sync configured?"* — `hpc/sync.conf` is per-machine and + gitignored; first-time setup is the first branch. +- *"GPU or CPU partition?"* — picks `batch_gpu/` vs `batch_cpu/` templates. +- *"Which stage over which lenses?"* — each template pairs with a pipeline stage + (`submit_start_here`, `submit_full_model`); the dataset list defines the array. + +## Branch — configure sync (once per machine) + +```bash +cp hpc/sync.conf.example hpc/sync.conf +# edit: HPC_HOST (ssh alias or user@host), HPC_BASE (remote project root), +# PROJECT_NAME (defaults to the folder name) +``` + +`$HPC_BASE/$PROJECT_NAME` becomes `$PROJECT_PATH` inside the batch scripts, so +activation and paths stay consistent between local and remote. Then: + +```bash +hpc/sync push # code + dataset/ up to the cluster +hpc/sync pull # results (output/, dataset products) back down +hpc/sync sync # push then pull (default) +hpc/sync status # dry run — show what would transfer +``` + +Citation: `euclid_strong_lens_modeling_pipeline:hpc/sync`. + +## Branch — adapt a batch template and submit + +The templates (`hpc/batch_gpu/submit_start_here`, `hpc/batch_gpu/submit_full_model`, +CPU twins under `hpc/batch_cpu/`) are SLURM array jobs: a `datasets=( … )` list, the +`--array=0-N` range to match, `sample=` for the dataset folder, and per-task resources +(`--partition=gpu --gres=gpu:1`, 32 GB, 2 h for the initial fit — raise the time limit +for `full_model`, which chains five searches). Edit the list, the array range, and the +mail address, then on the cluster: + +```bash +cd $PROJECT_PATH && sbatch hpc/batch_gpu/submit_start_here +``` + +Logs land in `output/output.%A_%a.out` / `error/error.%A_%a.err` per array task — +`tail -f` the newest to watch a fit, and check the error file first when a task dies. +After completion, `hpc/sync pull` locally brings `output/` home, and +[`euclid_workflow_products`](./euclid_workflow_products.md) turns it into catalogues +and summaries. + +Two habits that save cluster time: smoke the exact submission path once with +`PYAUTO_TEST_MODE=1` on a single-lens array before a big run, and remember completed +fits resume/skip — resubmitting an array over partly-finished lenses only runs what's +missing. + +## Combine + +- [`euclid_model_lens`](./euclid_model_lens.md) — what each stage does and which to + batch. +- [`euclid_workflow_products`](./euclid_workflow_products.md) — sample-scale inspection + once results are pulled. +- [`wiki/core/operations/hpc_infrastructure.md`](../wiki/core/operations/hpc_infrastructure.md) + — the assistant's generic HPC template/sync machinery, of which the pipeline's + `hpc/` folder is the Euclid instance. + +## Further reading + +- **General reference** — [euclid_strong_lens_modeling_pipeline README](https://github.com/PyAutoLabs/euclid_strong_lens_modeling_pipeline#readme): + the repo the batch templates run from. diff --git a/skills/euclid_model_lens.md b/skills/euclid_model_lens.md new file mode 100644 index 0000000..5d9df6e --- /dev/null +++ b/skills/euclid_model_lens.md @@ -0,0 +1,122 @@ +--- +name: euclid_model_lens +description: Model a Euclid strong lens through the pipeline's staged scripts — the initial MGE + SIE fit (start_here.py), Sersic lens/source photometry for SED fitting (sersic_lens_model.py), lens-only MGE subtraction (mge_lens_only.py), multi-waveband NIR/EXT fits with the VIS model fixed (lens_model_waveband.py), and the full SLaM pixelized-source + PowerLaw model (full_model.py). Use to choose and run the right Euclid pipeline stage for a science goal; NOT for setting up the repo (euclid_setup_pipeline) or hand-composed non-Euclid models (al_build_imaging_model / al_run_slam_pipeline). +--- + +# Modeling a Euclid lens: the staged pipelines + +The pipeline models every Euclid lens through the same staged progression. Each stage +answers a different science need, and each is seeded by the previous stage's result so +convergence stays robust — the same chained-search logic as PyAutoLens's SLaM pipelines +([`wiki/core/concepts/slam_pipeline.md`](../wiki/core/concepts/slam_pipeline.md)), +specialised to Euclid data. All stages run with one CLI: + +```bash +python