feat: euclid mode — pipeline-paired skills + dedicated euclid wiki (#73)

.claude/skills/euclid_hpc_runs.md

@@ -0,0 +1 @@
+../../skills/euclid_hpc_runs.md

.claude/skills/euclid_model_lens.md

@@ -0,0 +1 @@
+../../skills/euclid_model_lens.md

.claude/skills/euclid_prepare_data.md

@@ -0,0 +1 @@
+../../skills/euclid_prepare_data.md

.claude/skills/euclid_setup_pipeline.md

@@ -0,0 +1 @@
+../../skills/euclid_setup_pipeline.md

.claude/skills/euclid_workflow_products.md

@@ -0,0 +1 @@
+../../skills/euclid_workflow_products.md

AGENTS.md

@@ -65,19 +65,25 @@ Map every request onto one or more layers:
 2. **Skills** (`skills/*.md`, symlinked into `.claude/skills/`) — *procedural*: how to do a
    task. Lensing skills are `al_<task>.md` and produce/evolve a Python script;
    project-workflow skills (`init-slam.md`, `start-new-project.md`) drive repo-level
-   operations. Skills starting with `_` (`_style.md`, `_bootstrap_skill.md`) are
-   meta-skills — don't surface them when answering science questions.
+   operations; **euclid mode** skills are `euclid_<task>.md` — they pair the
+   collaboration's `euclid_strong_lens_modeling_pipeline` repo to the assistant, and any
+   request about modeling *Euclid* data routes through them (entry:
+   `skills/euclid_setup_pipeline.md`). Skills starting with `_` (`_style.md`,
+   `_bootstrap_skill.md`) are meta-skills — don't surface them when answering science
+   questions.
 3. **Wiki** (`wiki/**/*.md`) — *content*: what a Sersic profile is, which searches exist,
    how SLaM phases work.
 
 > **Rule of thumb.** *How do I do X?* → a skill. *What / which / why X?* → the wiki. *Build
 > something end-to-end?* → compose skills, citing wiki pages as you go.
 
-The wiki has three sub-wikis: **`wiki/core/`** (curated PyAuto\* reference, read-only —
+The wiki has four sub-wikis: **`wiki/core/`** (curated PyAuto\* reference, read-only —
 refreshed by `al_update_wiki`), **`wiki/literature/`** (strong-lensing science reference,
 own schema in [`wiki/literature/AGENTS.md`](./wiki/literature/AGENTS.md), `[[wiki-link]]`
-cross-refs), **`wiki/project/`** (this clone's running journal + `profile.md`). "The wiki"
-means `wiki/core/` unless `literature/` or `project/` is named.
+cross-refs), **`wiki/euclid/`** (Euclid mission + Euclid strong-lensing literature,
+same schema as `literature/`, paired with the `euclid_*` skills), **`wiki/project/`**
+(this clone's running journal + `profile.md`). "The wiki" means `wiki/core/` unless
+`literature/`, `euclid/` or `project/` is named.
 
 ---
 

skills/README.md

@@ -12,6 +12,8 @@ configured) via symlinks; the canonical files live here.
 - File names use the `al_<task>` convention for lensing-API skills, e.g. `al_run_search.md`.
 - Project-workflow skills (repo-level operations, template manipulation) use a plain
   kebab-case name, e.g. `init-slam.md`, `start-new-project.md`.
+- Domain-mode skills that pair an external pipeline repo to the assistant use that
+  domain as prefix — `euclid_<task>.md` for the Euclid pipeline (euclid mode).
 - Meta-skills (writing guide, bootstrap protocol) start with `_`.
 - Every lensing-API skill is **python-first**: the deliverable is a runnable `.py` script
   + the understanding to evolve it. Project-workflow skills may instead drive `rsync`,
@@ -66,6 +68,26 @@ them.
   pipeline script(s) copied from `autolens_workspace` and tailored to the chosen data
   type.
 
+### Euclid mode (pipeline-paired)
+
+Skills pairing the collaboration's
+[`euclid_strong_lens_modeling_pipeline`](https://github.com/PyAutoLabs/euclid_strong_lens_modeling_pipeline)
+to the assistant; the paired literature context is the dedicated
+[`wiki/euclid/`](../wiki/euclid/index.md) sub-wiki.
+
+- [`euclid_setup_pipeline.md`](./euclid_setup_pipeline.md) — clone/install the pipeline,
+  the `dataset/<sample>/<dataset>/` layout, and the black-box `start_here.py` run.
+- [`euclid_prepare_data.md`](./euclid_prepare_data.md) — segmentation validation, binary
+  mask tuning, extra-galaxy masks/centres via the GUI tools, hand-assembling a dataset
+  folder.
+- [`euclid_model_lens.md`](./euclid_model_lens.md) — choose and run the staged
+  pipelines: initial MGE+SIE, Sersic photometry, lens-only subtraction, multi-waveband,
+  full SLaM.
+- [`euclid_workflow_products.md`](./euclid_workflow_products.md) — aggregate many fits
+  into .csv catalogues, .fits stacks, and one-line .png summaries via `workflow/`.
+- [`euclid_hpc_runs.md`](./euclid_hpc_runs.md) — `hpc/sync` configuration and the
+  SLURM batch-array templates for sample-scale runs.
+
 ### Data preparation
 
 - [`al_prepare_imaging_data.md`](./al_prepare_imaging_data.md) — load and preprocess

skills/euclid_hpc_runs.md

@@ -0,0 +1,83 @@
+---
+name: euclid_hpc_runs
+description: Run the Euclid pipeline on a supercomputer — configure hpc/sync (push code/data, pull results), adapt the batch_cpu / batch_gpu SLURM array templates (one dataset per array task), submit, monitor, and pull results back. Use when a Euclid lens sample outgrows the local machine; NOT for single interactive fits (euclid_setup_pipeline) or generic HPC concepts (wiki/core/operations/hpc.md).
+---
+
+# Running Euclid samples on HPC
+
+One lens is a laptop job; a sample is not. The pipeline ships an HPC workflow built on
+two pieces: a **sync CLI** that mirrors the project to the cluster and pulls results
+back, and **SLURM array templates** that run one dataset per array task — the natural
+unit, since every pipeline stage takes `--sample`/`--dataset` arguments and lenses are
+independent. GPU nodes are the fast path (JAX accelerates the likelihood ~50×; one GPU
+per task is enough); CPU templates exist for clusters without them. Generic
+HPC/JAX/SLURM concepts live in
+[`wiki/core/operations/hpc.md`](../wiki/core/operations/hpc.md) — this skill is the
+Euclid-pipeline application of them.
+
+## Ask
+
+- *"Which cluster, and is sync configured?"* — `hpc/sync.conf` is per-machine and
+  gitignored; first-time setup is the first branch.
+- *"GPU or CPU partition?"* — picks `batch_gpu/` vs `batch_cpu/` templates.
+- *"Which stage over which lenses?"* — each template pairs with a pipeline stage
+  (`submit_start_here`, `submit_full_model`); the dataset list defines the array.
+
+## Branch — configure sync (once per machine)
+
+```bash
+cp hpc/sync.conf.example hpc/sync.conf
+# edit: HPC_HOST (ssh alias or user@host), HPC_BASE (remote project root),
+#       PROJECT_NAME (defaults to the folder name)
+```
+
+`$HPC_BASE/$PROJECT_NAME` becomes `$PROJECT_PATH` inside the batch scripts, so
+activation and paths stay consistent between local and remote. Then:
+
+```bash
+hpc/sync push      # code + dataset/ up to the cluster
+hpc/sync pull      # results (output/, dataset products) back down
+hpc/sync sync      # push then pull (default)
+hpc/sync status    # dry run — show what would transfer
+```
+
+Citation: `euclid_strong_lens_modeling_pipeline:hpc/sync`.
+
+## Branch — adapt a batch template and submit
+
+The templates (`hpc/batch_gpu/submit_start_here`, `hpc/batch_gpu/submit_full_model`,
+CPU twins under `hpc/batch_cpu/`) are SLURM array jobs: a `datasets=( … )` list, the
+`--array=0-N` range to match, `sample=` for the dataset folder, and per-task resources
+(`--partition=gpu --gres=gpu:1`, 32 GB, 2 h for the initial fit — raise the time limit
+for `full_model`, which chains five searches). Edit the list, the array range, and the
+mail address, then on the cluster:
+
+```bash
+cd $PROJECT_PATH && sbatch hpc/batch_gpu/submit_start_here
+```
+
+Logs land in `output/output.%A_%a.out` / `error/error.%A_%a.err` per array task —
+`tail -f` the newest to watch a fit, and check the error file first when a task dies.
+After completion, `hpc/sync pull` locally brings `output/` home, and
+[`euclid_workflow_products`](./euclid_workflow_products.md) turns it into catalogues
+and summaries.
+
+Two habits that save cluster time: smoke the exact submission path once with
+`PYAUTO_TEST_MODE=1` on a single-lens array before a big run, and remember completed
+fits resume/skip — resubmitting an array over partly-finished lenses only runs what's
+missing.
+
+## Combine
+
+- [`euclid_model_lens`](./euclid_model_lens.md) — what each stage does and which to
+  batch.
+- [`euclid_workflow_products`](./euclid_workflow_products.md) — sample-scale inspection
+  once results are pulled.
+- [`wiki/core/operations/hpc_infrastructure.md`](../wiki/core/operations/hpc_infrastructure.md)
+  — the assistant's generic HPC template/sync machinery, of which the pipeline's
+  `hpc/` folder is the Euclid instance.
+
+## Further reading
+
+- **General reference** — [euclid_strong_lens_modeling_pipeline README](https://github.com/PyAutoLabs/euclid_strong_lens_modeling_pipeline#readme):
+  the repo the batch templates run from.

skills/euclid_model_lens.md

@@ -0,0 +1,122 @@
+---
+name: euclid_model_lens
+description: Model a Euclid strong lens through the pipeline's staged scripts — the initial MGE + SIE fit (start_here.py), Sersic lens/source photometry for SED fitting (sersic_lens_model.py), lens-only MGE subtraction (mge_lens_only.py), multi-waveband NIR/EXT fits with the VIS model fixed (lens_model_waveband.py), and the full SLaM pixelized-source + PowerLaw model (full_model.py). Use to choose and run the right Euclid pipeline stage for a science goal; NOT for setting up the repo (euclid_setup_pipeline) or hand-composed non-Euclid models (al_build_imaging_model / al_run_slam_pipeline).
+---
+
+# Modeling a Euclid lens: the staged pipelines
+
+The pipeline models every Euclid lens through the same staged progression. Each stage
+answers a different science need, and each is seeded by the previous stage's result so
+convergence stays robust — the same chained-search logic as PyAutoLens's SLaM pipelines
+([`wiki/core/concepts/slam_pipeline.md`](../wiki/core/concepts/slam_pipeline.md)),
+specialised to Euclid data. All stages run with one CLI:
+
+```bash
+python <script> --sample=<sample> --dataset=<dataset_name> --iterations_per_quick_update=50000
+```
+
+and every script is also importable (each exposes `fit(...)` functions), so batch
+drivers and HPC submissions reuse the same code paths.
+
+The papers behind these choices — why MGE + SIE is the standard Q1/DR1 fit, what the
+discovery samples look like — are in
+[`wiki/euclid/sources/euclid-strong-lensing.md`](../wiki/euclid/sources/euclid-strong-lensing.md).
+
+## Ask
+
+- *"What do you need from this lens?"* — a first model / photometry for SED fitting /
+  a clean arc image / colour information / a publication mass model. The answer picks
+  the stage below.
+- *"Has the initial fit already run?"* — every later stage fixes parameters to the
+  `initial_lens_model` result; run [`euclid_setup_pipeline`](./euclid_setup_pipeline.md)
+  first if `output/<sample>/<dataset>/initial_lens_model/` doesn't exist.
+- *"One lens interactively, or a sample?"* — for samples, plan the run through
+  [`euclid_hpc_runs`](./euclid_hpc_runs.md) and inspect with
+  [`euclid_workflow_products`](./euclid_workflow_products.md).
+
+## Branch — initial lens model (`start_here.py`)
+
+The standard first fit: MGE lens light (2×20 Gaussians) + Isothermal + shear + MGE
+source, ~15 non-linear parameters, Nautilus. The MGE's linear light profiles keep the
+non-linear space small and well-conditioned, which is what makes ~10 min/lens on a GPU
+possible. Mass centre is anchored to the brightest central pixel; the MGE intensities
+are solved by linear inversion at every likelihood call. Outputs include the deblended
+lens/source images and the Euclid latent variables (aperture fluxes → AB magnitudes via
+the header MAGZERO, magnification) from
+`euclid_strong_lens_modeling_pipeline:util.py` (`LatentEuclid`).
+`scripts/initial_lens_model.py` is the same fit as an importable module.
+
+## Branch — Sersic photometry (`scripts/sersic_lens_model.py`)
+
+When the goal is **photometry for SED fitting** (and photometric redshifts), Sersic
+profiles beat an MGE: total flux is a model parameter rather than a sum of Gaussians.
+`fit_sersic()` fits a linear Sersic bulge to lens and source in VIS with the **mass
+model fixed** to the initial result; `fit_waveband()` then carries the Sersic model
+across every non-VIS band. The Sersic centre diverges, so the script derives a
+higher-order over-sampling scheme from the prior fit's tracer. Multi-band Sersic
+fluxes → SEDs → photo-zs is the standard chain
+([`wiki/euclid/concepts/euclid-photo-z.md`](../wiki/euclid/concepts/euclid-photo-z.md)).
+
+## Branch — lens-only subtraction (`scripts/mge_lens_only.py`)
+
+Fits an MGE to the **lens light only** — no mass, no source. The residual image is the
+cleanest quick view of the lensed arcs, for visual inspection, discovery vetting, or as
+input to downstream tools. `fit()` runs VIS; `fit_waveband()` repeats per band with the
+VIS lens model fixed, giving lens-subtracted images in every band.
+
+## Branch — multi-waveband model (`scripts/lens_model_waveband.py`)
+
+Carries the full VIS lens model (light + mass + source) onto every non-VIS band, fixed,
+with only a per-band sub-pixel astrometric offset (`DatasetModel`) free to absorb
+residual band-to-band alignment errors. This is how colour information for the lens and
+source is extracted without re-fitting the whole model at lower resolution — the NIR
+bands trade resolution for colour ([`wiki/euclid/entities/nisp.md`](../wiki/euclid/entities/nisp.md),
+[`wiki/euclid/entities/ext-surveys.md`](../wiki/euclid/entities/ext-surveys.md)).
+Called internally by the Sersic and lens-only pipelines' `fit_waveband()` stages.
+
+## Branch — full SLaM model (`scripts/full_model.py`)
+
+The publication-quality model, chaining five searches on VIS:
+
+1. **SOURCE LP** — parametric MGE source + isothermal mass (fast initialisation).
+2. **SOURCE PIX 1** — pixelized source on a rectangular adaptive mesh, creating the
+   adapt image (`mesh_pixels_yx` is fixed — JAX needs statically shaped arrays).
+3. **SOURCE PIX 2** — refined pixelization with adapt-image regularisation.
+4. **LIGHT LP** — lens-light refinement with source/mass held.
+5. **MASS TOTAL** — final PowerLaw mass model.
+
+Image positions are computed automatically from the SOURCE LP result to penalise
+unphysical demagnified reconstructions. Expect hours, not minutes — plan sample-scale
+runs on HPC. The generic SLaM concepts are in
+[`al_run_slam_pipeline`](./al_run_slam_pipeline.md); the Euclid script is the tuned,
+ready-to-run instance.
+
+## Choosing a stage
+
+| Science goal | Stage |
+|---|---|
+| First model / mass + Einstein radius estimate | `start_here.py` |
+| Fluxes and magnitudes for SED fitting / photo-z | `sersic_lens_model.py` |
+| Clean image of the arcs | `mge_lens_only.py` |
+| Lens + source colours across bands | `lens_model_waveband.py` (or the `fit_waveband` stages) |
+| Publication mass model, pixelized source | `full_model.py` |
+
+## Combine
+
+- [`euclid_workflow_products`](./euclid_workflow_products.md) — turn many fits into
+  .csv catalogues, .fits stacks and one-line .png summaries.
+- [`al_load_results`](./al_load_results.md) / [`al_plot_fit_residuals`](./al_plot_fit_residuals.md)
+  — inspect an individual fit in depth; the pipeline's outputs are standard PyAutoLens
+  results.
+- [`al_debug_fit_failure`](./al_debug_fit_failure.md) — post-mortem for a stage that
+  converged badly; check the initial fit before blaming a later stage.
+- If key output for your science case is missing from the pipeline, that is by design a
+  collaboration conversation — the pipeline grows by request so outputs stay standard
+  across the data release (contact James Nightingale on the Euclid Consortium Slack).
+
+## Further reading
+
+- **General reference** — [euclid_strong_lens_modeling_pipeline README](https://github.com/PyAutoLabs/euclid_strong_lens_modeling_pipeline#readme):
+  the pipeline inventory and run commands.
+- **Experienced PyAutoLens user** — [workspace/lens: guides/modeling/slam_start_here.py](https://github.com/PyAutoLabs/autolens_workspace/blob/main/scripts/guides/modeling/slam_start_here.py):
+  the generic SLaM template `full_model.py` specialises for Euclid.