Skip to content

Commit 06cc5b4

Browse files
Jammy2211claude
authored andcommitted
Initial commit: lensing_wiki + README
Bootstraps the PyAutoPaper repo with the strong-gravitational-lensing sub-wiki (~85 markdown files under lensing_wiki/), a short README, and a .gitignore that keeps PDFs and other binaries out of the repo. The wiki follows Karpathy's LLM Wiki pattern: raw PDFs as the immutable local layer, compiled topical concept / entity / source pages as the git-tracked layer, with a CLAUDE.md schema describing how an AI assistant should read and maintain it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0 parents  commit 06cc5b4

84 files changed

Lines changed: 4837 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.gitignore

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# PDFs are kept locally only — they are available from arXiv / journal sites
2+
*.pdf
3+
4+
# Other binary/large files we don't want in the repo
5+
*.exe
6+
*.zip
7+
*.tar
8+
*.tar.gz
9+
*.tgz
10+
*.7z
11+
*.odt
12+
*.doc
13+
*.docx
14+
*.ppt
15+
*.pptx
16+
*.xls
17+
*.xlsx
18+
*.jpg
19+
*.JPG
20+
*.jpeg
21+
*.png
22+
*.PNG
23+
*.gif
24+
*.tif
25+
*.tiff
26+
27+
# OS / editor cruft
28+
.DS_Store
29+
Thumbs.db
30+
desktop.ini
31+
.idea/
32+
.vscode/
33+
*.swp
34+
*~
35+
__pycache__/
36+
*.pyc
37+
38+
# LaTeX build artefacts
39+
*.aux
40+
*.bbl
41+
*.blg
42+
*.fdb_latexmk
43+
*.fls
44+
*.log
45+
*.out
46+
*.synctex.gz
47+
*.toc
48+
49+
# Exception — the wiki's own log.md is text and must be tracked
50+
!lensing_wiki/log.md

README.md

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# PyAutoPaper
2+
3+
A personal paper library and writing context repo for
4+
[PyAutoLens](https://github.com/Jammy2211/PyAutoLens) and related
5+
science. Holds source PDFs locally (gitignored) and compiled
6+
LLM-readable wikis in git.
7+
8+
## What's in here
9+
10+
- **`lensing_wiki/`** — the strong-gravitational-lensing sub-wiki.
11+
Topical concept pages, named-entity pages (surveys, lenses, software),
12+
and per-topic bibliographies covering ~190 papers across
13+
`Strong_Lens/`, `Substructure/`, `StrongLensCluster/`,
14+
`Dark_Matter_Detection/`, `DarkMatterModels/`. Built on Karpathy's
15+
"LLM Wiki" pattern: the PDFs are the raw layer, the wiki is the
16+
compiled layer an AI assistant reads at query time. Start at
17+
[`lensing_wiki/index.md`](lensing_wiki/index.md). See
18+
[`lensing_wiki/CLAUDE.md`](lensing_wiki/CLAUDE.md) for the schema.
19+
- **`Strong_Lens/`, `Substructure/`, …** (gitignored) — the PDF folders.
20+
Kept locally so the wiki can reference them; not pushed because the
21+
PDFs are available from arXiv / journal sites.
22+
23+
## For collaborators
24+
25+
Clone the repo and you have the wikis. You won't see the PDFs — fetch
26+
them from the URLs cited in each source page if you need them. The
27+
wikis are self-contained scientific context: every concept page is a
28+
synthesis, every source page lists the file path and a one-paragraph
29+
stub keyed to the cited PDF.
30+
31+
The wiki status of each source page is tracked in its frontmatter
32+
(`status: stub | drafted | reviewed`). In the initial build all
33+
per-paper stubs are `stub` — inferred from filenames and field
34+
knowledge, not yet verified against the PDFs. Anything contributed
35+
should upgrade stubs to `drafted` after the corresponding PDF has been
36+
read, and add an entry to `lensing_wiki/log.md`.
37+
38+
## Roadmap
39+
40+
- Verify and upgrade per-paper stubs from `stub``drafted`.
41+
- Add tangential folders (`WeakLensing/`, `Ellipticals/`,
42+
`Deep Learning/`, `Stats/`, …) to scope.
43+
- Add sibling sub-wikis as the library grows (e.g. `imaging_wiki/`,
44+
`cosmology_wiki/`) following the same schema.

lensing_wiki/CLAUDE.md

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
# PyAutoLens AI Assistant — Strong Lensing Wiki
2+
3+
This wiki gives a PyAutoLens AI assistant the broad scientific context of
4+
strong gravitational lensing. It follows Karpathy's "LLM Wiki" pattern: the
5+
PDFs in the surrounding `PyAutoPaper/` repo are the immutable `raw/` layer
6+
(gitignored — kept locally only), and this `lensing_wiki/` folder is the
7+
compiled, cross-linked knowledge layer the assistant reads at query time
8+
and that lives in git for collaborators.
9+
10+
## Layout
11+
12+
```
13+
PyAutoPaper/ # repo root
14+
├── lensing_wiki/ # this folder — the compiled wiki (in git)
15+
│ ├── CLAUDE.md # this file — schema + usage rules
16+
│ ├── index.md # top-level navigation
17+
│ ├── log.md # append-only compilation log
18+
│ ├── concepts/ # one topic per page — the science
19+
│ ├── entities/ # specific surveys, lenses, collaborations, software
20+
│ └── sources/ # per-topic bibliography pages (one paper = one section)
21+
├── Strong_Lens/ # raw/ layer — PDFs (gitignored)
22+
├── Substructure/
23+
├── StrongLensCluster/
24+
├── Dark_Matter_Detection/
25+
├── DarkMatterModels/
26+
└── ... # other PDF folders (gitignored)
27+
```
28+
29+
The PDFs are the ground truth. Wiki pages are syntheses. If a wiki page and a
30+
PDF disagree, the PDF wins; the wiki page should be updated and the change
31+
noted in `log.md`.
32+
33+
## Page types
34+
35+
| Type | Folder | Scope |
36+
|-------------|--------------|-------------------------------------------------------|
37+
| Concept | `concepts/` | One scientific concept (e.g. mass-sheet degeneracy) |
38+
| Entity | `entities/` | One named thing (survey, lens, code, collaboration) |
39+
| Sources | `sources/` | All paper stubs for one topic, one section per paper |
40+
| Index/log | root | Navigation and provenance |
41+
42+
## Naming
43+
44+
- File names are lowercase kebab-case: `mass-sheet-degeneracy.md`,
45+
`slacs.md`, `time-delay-cosmography.md`.
46+
- One concept per concept page. If a page tries to cover two ideas, split it.
47+
- Source-collection pages are named by topic: `sources/dark-matter-substructure.md`.
48+
49+
## Cross-references
50+
51+
Use `[[page-slug]]` for wiki-internal links — for example
52+
`[[mass-sheet-degeneracy]]` or `[[slacs]]`. Slugs match the filename without
53+
`.md`. A `[[link]]` that has no target file yet is fine — it marks a future
54+
page to write.
55+
56+
External references point at the PDF on disk using a relative path from
57+
the `PyAutoPaper/` repo root, e.g. `Strong_Lens/Suyu2016Holicow.pdf`.
58+
The PDFs themselves are kept locally (gitignored), so collaborators
59+
cloning the repo will see the wiki references but need to source the
60+
PDFs separately (most are on arXiv or the journal site).
61+
62+
## Frontmatter
63+
64+
Every wiki file starts with YAML frontmatter:
65+
66+
```yaml
67+
---
68+
title: Mass-sheet degeneracy
69+
type: concept # concept | entity | sources | meta
70+
topics: [degeneracies, cosmography]
71+
sources: # optional — papers most relevant to this page
72+
- Strong_Lens/Suyu2016Holicow.pdf
73+
- Strong_Lens/Birrer2020TDCOSMOSIVH0.pdf
74+
status: stub # stub | drafted | reviewed
75+
---
76+
```
77+
78+
Sources may be `[]` for pages that synthesize general field knowledge.
79+
80+
## Concept page structure
81+
82+
```
83+
# Title
84+
85+
## TL;DR
86+
One paragraph an assistant can quote back to a user.
87+
88+
## What it is
89+
The physics / definition.
90+
91+
## Why it matters for PyAutoLens
92+
How this concept shows up in lens-modeling decisions a PyAutoLens user makes.
93+
94+
## Key results from the literature
95+
Bullet list. Each bullet ends with `([[author-year-stub]])` so the LLM can
96+
follow the link to the per-paper section.
97+
98+
## See also
99+
- [[related-concept-1]]
100+
- [[related-concept-2]]
101+
```
102+
103+
## Entity page structure
104+
105+
Same idea but the headings are "What it is / Key facts / Papers / See also".
106+
Use entity pages for: surveys (SLACS, BELLS, H0liCOW), specific lenses
107+
(Abell 1201, the Cosmic Horseshoe), software (PyAutoLens, lenstronomy),
108+
collaborations (TDCOSMO, Space Warps).
109+
110+
## Source-collection page structure
111+
112+
```
113+
# Sources: <topic>
114+
115+
Bibliography of papers in `Papers/` covering this topic. Each paper has its
116+
own H2 section; cross-link from concept and entity pages with
117+
`[[sources-<topic>#author-year-slug]]`.
118+
119+
## Author Year — short tag
120+
121+
**File:** `Strong_Lens/Filename.pdf`
122+
**Concepts:** [[concept-1]], [[concept-2]]
123+
**Summary:** one-paragraph stub inferred from filename and field knowledge.
124+
Mark `(stub — verify against PDF)` if not yet read.
125+
```
126+
127+
## How the assistant should use this wiki
128+
129+
1. On a user question, first open `index.md`.
130+
2. Follow the relevant `concepts/` or `entities/` page.
131+
3. Cite specific results by linking the `[[sources-topic#author-year]]`
132+
anchor. If a claim is needed and the source stub is unread, open the
133+
PDF at the path in the stub's `File:` line.
134+
4. When a PDF is read in full, upgrade the stub's `status:` from `stub` to
135+
`drafted`, replace the inferred summary with what the paper actually
136+
says, and add a line to `log.md`.
137+
5. Never fabricate a citation. If a result is not on a wiki page, say so
138+
and offer to read the relevant PDF.
139+
140+
## Scope
141+
142+
In-scope folders for the current wiki build:
143+
144+
- `Strong_Lens/` — primary
145+
- `Substructure/`
146+
- `StrongLensCluster/`
147+
- `Dark_Matter_Detection/`
148+
- `DarkMatterModels/`
149+
150+
Other folders (`WeakLensing/`, `Ellipticals/`, `Deep Learning/`, etc.)
151+
contain papers that touch lensing tangentially but are out of scope until
152+
explicitly added — see `log.md` for the decision.
153+
154+
This wiki is the **lensing** sub-wiki of `PyAutoPaper/`. Future sub-wikis
155+
(e.g. `imaging_wiki/`, `cosmology_wiki/`) may sit alongside it at the
156+
repo root following the same schema.
Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
---
2+
title: Bayesian inference for strong lensing
3+
type: concept
4+
topics: [methods]
5+
sources: []
6+
status: drafted
7+
---
8+
9+
# Bayesian inference for strong lensing
10+
11+
## TL;DR
12+
13+
Strong-lens modelling is a high-dimensional, multimodal, often
14+
ill-conditioned inverse problem. The community has converged on Bayesian
15+
inference with nested sampling for posterior + evidence estimation.
16+
PyAutoLens uses the [[pyautofit]] library for this, supporting nested
17+
samplers like Dynesty, UltraNest, and Nautilus, and MCMC samplers like
18+
emcee. Pixel-source likelihoods carry an analytic source-marginalisation
19+
term (the "Bayesian evidence" of Suyu / Koopmans).
20+
21+
## Why nested sampling
22+
23+
- Returns the **evidence Z** — central for [[mass-models|model
24+
comparison]] (with vs. without external shear, EPL vs. composite, etc.).
25+
- Handles multimodality (mirror images, parity ambiguities) better than
26+
vanilla MCMC.
27+
- Doesn't need an initial guess close to the peak.
28+
29+
## The evidence for pixelised sources
30+
31+
For a pixelised source under Gaussian regularisation, the marginal
32+
likelihood over source pixels is analytic:
33+
34+
```
35+
log Z(θ_mass, λ) = −½ χ²(ŝ) − ½ log|LᵀC⁻¹L + λH| + ½ log|λH| − ½ N log(2π)
36+
```
37+
38+
This is the quantity nested sampling explores over **mass-model
39+
parameters and regularisation hyperparameters**, with the source itself
40+
already marginalised out.
41+
42+
## Pipelines and hierarchical fits
43+
44+
A modern fit (e.g. SLaM; [[slam-pipeline]]) chains together a sequence of
45+
sub-fits: first an SIE + Sérsic, then a power-law with a parametric
46+
source, then the same with a pixelised source, then optionally a
47+
composite stellar + dark fit. Each phase passes its posterior as a prior
48+
to the next via PyAutoFit's "search chaining". This is essential for
49+
robustness on real data.
50+
51+
## Model comparison
52+
53+
Bayesian evidence is the canonical metric. In practice:
54+
55+
- Two-sided log-evidence differences > a few are decisive.
56+
- Cross-check with [[bayesian-inference-lensing|posterior predictive
57+
checks]] and residuals.
58+
- For [[dark-matter-substructure|subhalo detection]] the
59+
evidence-difference threshold is calibrated against
60+
[[sources-dark-matter-substructure|Despali 2018/2022 sensitivity
61+
studies]].
62+
63+
## See also
64+
65+
- [[regularization]]
66+
- [[source-reconstruction]]
67+
- [[pyautofit]]
68+
- [[pyautolens]]
Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
---
2+
title: Bulge–halo (stellar + dark matter) decomposition
3+
type: concept
4+
topics: [galaxy-structure]
5+
sources:
6+
- Strong_Lens/Etherington2022BulgeHalo.pdf
7+
- Strong_Lens/Etherington2022SLaM.pdf
8+
- Strong_Lens/Etherington2023aBeyondBulge.pdf
9+
- Strong_Lens/Shajib2021MassSlopeNFW.pdf
10+
- Strong_Lens/Oldham2018StellarAndDark.pdf
11+
status: drafted
12+
---
13+
14+
# Bulge–halo decomposition
15+
16+
## TL;DR
17+
18+
Decompose a lens galaxy's mass into a stellar component (mass-traces-
19+
light Sérsic or MGE) and a dark-matter component (NFW or gNFW). This
20+
splits what the elliptical-power-law model conflates into a single slope
21+
into physically meaningful components — stellar M/L and DM concentration
22+
— at the cost of more parameters and a stronger sensitivity to the
23+
[[radial-angular-degeneracy]].
24+
25+
## What it is
26+
27+
Inputs:
28+
29+
- A high-resolution image of the lens light → MGE / Sérsic / multi-Sérsic
30+
decomposition (see [[sources-bulge-halo|Etherington series]]).
31+
- A free overall M/L or per-component M/L.
32+
- An NFW (or gNFW) DM halo.
33+
- Optional adiabatic contraction.
34+
- [[external-convergence-shear|External shear]].
35+
36+
Outputs:
37+
38+
- Stellar M/L, often inverted to **the IMF**
39+
([[microlensing-imf|complementary to microlensing-IMF measurements]]).
40+
- DM concentration and mass.
41+
- Stellar-to-halo mass ratio inside the [[einstein-radius]].
42+
43+
## Why it matters
44+
45+
- Tests galaxy formation: do massive ellipticals have NFW-consistent
46+
haloes or contracted ones?
47+
- Calibrates the **dark-matter fraction inside R_eff**, a benchmark
48+
galaxy-evolution quantity.
49+
- Disentangles IMF from DM concentration, the long-standing
50+
bulge-vs-halo degeneracy.
51+
52+
## In PyAutoLens
53+
54+
The SLaM pipeline ([[slam-pipeline]]) chains:
55+
56+
1. Sérsic-mass + SIE phase.
57+
2. Power-law phase with parametric source.
58+
3. Power-law phase with pixelated source.
59+
4. Composite Sérsic + NFW phase — the bulge-halo step.
60+
61+
Each phase informs the next via PyAutoFit search chaining
62+
([[bayesian-inference-lensing]]).
63+
64+
## See also
65+
66+
- [[mass-models]]
67+
- [[radial-angular-degeneracy]]
68+
- [[microlensing-imf]]
69+
- [[slam-pipeline]]
70+
- [[sources-bulge-halo]]

0 commit comments

Comments
 (0)