Restructure: wiki/ + bibliography consolidation, structure lint, template sync

[Jammy2211]

Overview

PyAutoMemory's root has accumulated pre-wiki-era debris: 9 loose legacy .bib files (~12MB), orphan paper PDFs with ADS filenames, five one-file topic folders, and LaTeX hangovers — while the five *_wiki/ sub-wikis sit scattered at the root. This task restructures the repo around two clean top-level homes (wiki/ + the existing bibliography/), audits-then-retires the legacy bib files without losing any citation metadata, adds a structure lint so the state cannot recur, and syncs the fresh-slate template so every PyAutoScientist fork starts with the clean layout. It carries a research tail: a design note on longer-term PyAutoMemory structure.

Plan

• Move the five sub-wikis under wiki/, dropping the redundant suffix: wiki/lensing/, wiki/smbh/, wiki/cti/, wiki/methods/, wiki/galaxies/; promote the shared schema from lensing_wiki/CLAUDE.md to wiki/CLAUDE.md. Keep bibliography/ as the single canonical bib home (no rename to bib/).
• Key-level audit of the 9 root legacy .bib files + Euclid/Paykari2020.bib against bibliography/pyautomemory.bib; fold missing verified entries into canonical, then delete the legacy files (git history retains them). Delete the pyautopaper.bib back-compat symlink (no live references).
• Retire stray artifacts: confirm canonical bib entries + source stubs for the orphan papers (Hubble 1926, de Vaucouleurs 1948, Navarro 1996, Hall 1952, Paykari 2020, Liu 2024), then delete the two root PDFs, the five topic folders (CTI/, DarkMatterModels/, Euclid/, LightProFFits/, Medical/), cticomments, euclid.sty. Fix the index.md instruction that contradicts the "source PDFs live off-repo" policy.
• Add a structure lint (top-level allowlist; no .bib outside bibliography/; no PDFs/binaries) wired into make validate + minimal CI; write the rules into AGENTS.md/CONTRIBUTING.md.
• Sync tooling that hardcodes the old layout: PyAutoMemory validator globs, PyAutoMind/scripts/spawn.py template manifest (template ships empty wiki/ + bibliography/), PyAutoBrain memory-faculty glob (small parallel PR — repo claimed by workspace-agent; no file overlap, user-acknowledged), workspace-root AGENTS.md science pointer, admin_jammy doc mentions.
• Research tail: design note on longer-term structure (canonical-key TODO sweep, memory-surface manifest, reading-queue lifecycle, single-vs-sharded bib, repo-weight reality); accepted follow-ups filed via intake.

Detailed implementation plan

Affected Repositories

• PyAutoMemory (primary)
• PyAutoMind (scripts/spawn.py template manifest + prose)
• PyAutoBrain (one-line memory-faculty glob + one doc mention — small parallel PR; repo is claimed by the in-flight workspace-agent task, parallel PR explicitly user-approved, no file overlap)
• PyAutoLabs root AGENTS.md + admin_jammy docs (trivial pointer edits at ship time)
• PyAutoScientist (docs follow-up only; not part of this PR set)

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoMemory	main	1 modified (reading-queue.md, user's local edit — leave untouched)
./PyAutoMind	main	clean
./PyAutoBrain	main	1 modified (agents/conductors/intake/_intake.py, unregistered prior-session work — warned)

Suggested branch: feature/memory-structure-cleanup
Worktree root: ~/Code/PyAutoLabs-wt/memory-structure-cleanup/ (created by /start_library)
Work classification: infrastructure (organ repos; library-dev worktree mechanics)

Implementation Steps

1. Bib audit (read-only first). Script a key extraction over the 9 root .bib files + Euclid/Paykari2020.bib; diff key sets (and DOI/arXiv/title matches via bibliography/bibkey_aliases.yaml conventions) against bibliography/pyautomemory.bib. Report: per-file unique-key count. Fold verified missing entries into canonical (respecting the existing "conflicts never replace the library.bib-derived entry" precedent). Run make validate-literature-citations.
2. Delete legacy bib files (post-fold): cooper.bib, library.bib, library2.bib, library_jam.bib, libraryipft.bib, manual.bib, pyautofit.bib, pyautolens.bib, qiuhan.bib, bibliography/pyautopaper.bib symlink.
3. Retire strays: verify/add canonical entries + source stubs for the six orphan papers; delete Hubble1926.321H, devaucoleurs1948.247D, CTI/, DarkMatterModels/, Euclid/, LightProFFits/, Medical/, cticomments, euclid.sty.
4. Wiki move: git mv <domain>_wiki wiki/<domain> for lensing/smbh/cti/methods/galaxies; git mv wiki/lensing/CLAUDE.md wiki/CLAUDE.md (schema is shared, sub-wikis inherit); scripted rewrite of relative links (../<d>_wiki/ → ../<d>/, <d>_wiki/ → wiki/<d>/ in root files); update index.md, README.md, AGENTS.md, bibliography/README.md.
5. Validator + lint: update scripts/validate_literature_citations.py glob (*_wiki/sources → wiki/*/sources); add structure lint (new scripts/validate_structure.py or extend existing): top-level allowlist, no .bib outside bibliography/, no PDF/binary blobs; Makefile target; minimal GitHub Actions workflow running make validate-literature-citations + the lint; update tests/.
6. Rules prose: AGENTS.md + CONTRIBUTING.md — the layout contract (wiki/ + bibliography/ + allowlisted root files), PDFs off-repo, new bib entries only in canonical. Fix index.md reading-queue instruction (stub + cite; never commit the PDF).
7. PyAutoMind: update scripts/spawn.py manifest (*_wiki/* → wiki/* rules; keep bibliography/* EMPTY, *.bib DROP) + generated prose (example_wiki/ → wiki/example/); run /spawn drift check (dry-run) — actual template republish is a separate human-confirmed step.
8. PyAutoBrain (small parallel PR): agents/faculties/memory/_memory.py glob("*_wiki") → also/instead wiki/* (tolerate both layouts for one release); update skills/sampler_pipeline/reference.md mention (PyAutoMemory/methods_wiki → PyAutoMemory/wiki/methods).
9. Ship-time pointer edits: workspace-root AGENTS.md science pointer (sub-wiki names), admin_jammy/claude.md + README.md mentions.
10. Research tail: design-note comment on this issue — canonical-key TODO sweep (345 open TODOs; the legacy fold should resolve some), a memory-surface manifest for the faculty ("no layout coupling" promise), reading-queue lifecycle, single canonical bib vs per-domain shards (recommend single), repo weight (legacy blobs remain in git history; no rewrites, template spawn is the clean-slate path). File accepted follow-ups via intake, one at a time.

Key Files

• PyAutoMemory/bibliography/pyautomemory.bib — canonical bib; fold target
• PyAutoMemory/bibliography/README.md — bib policy; update legacy-inputs paragraph once legacy files are gone
• PyAutoMemory/scripts/validate_literature_citations.py — glob("*_wiki/sources") layout coupling
• PyAutoMemory/index.md, README.md, AGENTS.md — navigation + rules
• PyAutoMemory/lensing_wiki/CLAUDE.md — shared schema (promote to wiki/CLAUDE.md)
• PyAutoMind/scripts/spawn.py — fresh-slate template manifest (*_wiki/* DROP, bibliography/* EMPTY, *.bib DROP) + template prose
• PyAutoBrain/agents/faculties/memory/_memory.py — glob("*_wiki") (line 35)
• PyAutoBrain/skills/sampler_pipeline/reference.md — methods_wiki mention

Original Prompt

Click to expand starting prompt

PyAutoMemory, This feels like it could be cleaned up:

• Have a wiki/ folder with the wikis, analgous to autolens_assistant, at the moment the wiki fodlers seem scattered.
• Fodlers like CTI, DarkMatterModels, Euclid and LightProfFits contain individual .pdf files or .bib files. These must be
  some sort of unused hangover from previous work which can be removed?
• Is bibliography/pyautopaper.bib now redundant?
• There are like 9 .bib files in the main repo, I am sure I just dumped them here but its time to clean them up. Either
  consolidate to one .bib file or have a bib folder next to wiki which puts the .bib entries in sensible standalone files.
  I know bib referencing is hard for AI so these do contain important info so dont just delete some, but also remove duplication.
• devaucoleurs1948.247D and Hubble1926.321H surely
  dont belong here?

In general, it feels like some rules need to be put in place across PyAutoScientist to prevent PyAutoMemory
from ending up in this statem and it would benefit from a bit more structure (e.g. the wikis folder, bib folder)
to make it more navigateable for a random user. This structure would also generalize it and make it more useable
for anyone in the PyAutoScientist ecosystem, especially as then theyd just need empty wiki, bib folders and
whatnot.

Finally do deep research and think hard about any other longer term changes we can make to the structure and
design of PyAutoMemory.

[Jammy2211]

Design note — longer-term PyAutoMemory structure (the research tail)

Findings from the deep-dive that go beyond this PR set, with recommendations:

1. The real long-term gap is coverage, not layout. 856 of 1099 canonical bib entries are not yet referenced by any wiki source page (the wikis cite 254), and 345 source sections still carry Canonical BibTeX key: TODO markers. The audit showed canonical already holds every legacy key, so most TODOs are resolvable by DOI/title matching against canonical — the Hall 1952 resolution in #25 is the proof of pattern. Recommend: one follow-up task ("canonical-key TODO sweep") pairing a matching script with human verification, filed when this ships; the 856 unreferenced entries fill paper-by-paper via the reading queue, not a bulk job.

2. Keep one canonical bib; don't shard per domain. Cross-domain reuse is real (methods papers cited from lensing pages), the alias table stays flat, and the validator stays simple. At 708K the file is nowhere near a practical limit. The 9-legacy-file problem was duplication, not size.

3. Layout coupling is now contained, not abstracted. The memory faculty discovers both layouts (Brain#121); the validator globs wiki/*/sources. A "memory-surface manifest" would honor the no-coupling promise more formally, but it's machinery for a problem two globs solve — recommend against (lean-lever principle). Revisit only if a third consumer appears.

4. Repo weight is permanent but bounded. The deleted blobs (~12MB bibs + ~4.5MB PDFs) remain in git history; we never rewrite pushed history. Fresh clones stay heavy; spawned templates are the clean-slate path for everyone else. Acceptable — noted so nobody proposes a history rewrite later.

5. Prevention now has teeth in three layers. CI structure lint (top-level allowlist, no loose .bib, PDF magic-byte detection), the AGENTS.md write contract, and the template itself (a PyAutoScientist fork starts with empty wiki/ + bibliography/ and green validators — exactly the generalization goal).

6. reading-queue.md stays at root. It's the one instance-state file a random user should see first; it's allowlisted. Its lifecycle instruction now matches the off-repo PDF policy (the index.md contradiction that produced the strays is fixed).

[Jammy2211]

Shipped: PyAutoMemory#25, PyAutoMind#82 and PyAutoBrain#121 all merged. Post-merge pointer syncs (workspace-root AGENTS.md science pointer, admin_jammy doc mentions) done locally; design-note follow-up (canonical-key TODO sweep) to be filed via intake when picked up.