From fa1aaa687cec56f8dabe0308f42a35b95fe1df9c Mon Sep 17 00:00:00 2001 From: Jammy2211 Date: Wed, 15 Jul 2026 10:41:12 +0100 Subject: [PATCH] docs: JOSS paper for PyAutoLens-Assistant Add paper/ holding the JOSS manuscript, hosted with the software it describes and mirroring the PyAutoLens-JAX sibling in PyAutoLens/paper_jax/. Consolidates the author's drafted Summary, Statement of need, software design and benchmark sections into a single manuscript, adds the JOSS front-matter, and stubs the State of the field and Research impact sections that require author judgement. Bibliography entries are copied verbatim from the assistant literature bibliography so the paper and the wiki cannot drift apart. Co-Authored-By: Claude Opus 4.8 --- paper/.gitignore | 1 + paper/README.md | 51 ++++++++++++++++++++++++++ paper/paper.bib | 86 +++++++++++++++++++++++++++++++++++++++++++ paper/paper.md | 95 ++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 233 insertions(+) create mode 100644 paper/.gitignore create mode 100644 paper/README.md create mode 100644 paper/paper.bib create mode 100644 paper/paper.md diff --git a/paper/.gitignore b/paper/.gitignore new file mode 100644 index 0000000..012aee3 --- /dev/null +++ b/paper/.gitignore @@ -0,0 +1 @@ +/paper.pdf diff --git a/paper/README.md b/paper/README.md new file mode 100644 index 0000000..832f3b0 --- /dev/null +++ b/paper/README.md @@ -0,0 +1,51 @@ +# PyAutoLens-Assistant JOSS paper + +This directory contains the PyAutoLens-Assistant paper for submission to the +[Journal of Open Source Software](https://joss.theoj.org/). It is hosted with the +software it describes, and is the sibling of the PyAutoLens-JAX paper in +`PyAutoLens/paper_jax/`. + +## Files + +- `paper.md` — manuscript and JOSS metadata. +- `paper.bib` — bibliography cited by the manuscript. Entries are copied verbatim + from `../wiki/literature/bibliography/autolens_literature.bib` so the paper and + the literature wiki cannot drift apart. +- `paper.pdf` — local build output; do not commit it. + +## Drafting checklist + +- Confirm the full author list, affiliations, ORCIDs, corresponding author, and + submission date. The current block mirrors the PyAutoLens-JAX paper. +- Keep the manuscript within the current JOSS target of 750–1750 words. +- Replace every drafting comment (`State of the field`, `Research impact + statement`, `Acknowledgements`) with specific, evidenced prose. +- Compare against other domain-specific scientific AI assistants and + general-purpose coding assistants in “State of the field”. +- **Report real benchmark results.** The `Benchmark examples` section is written + in the future tense because `../benchmarks/RESULTS.md` currently records no runs + for any benchmark. Once the suite has been run, replace that prose with measured + outcomes — do not claim results the repository cannot evidence. +- The paper describes **three representative** benchmarks; the repository ships + **four** prompts (the fourth is `hard_group_multi.md`). This framing is + deliberate. If the fourth is later described, update the wording in both places. +- Verify every bibliography entry resolves and is the intended paper. +- Keep the AI usage disclosure accurate as the manuscript evolves. + +The current format requirements are documented in the +[JOSS paper guide](https://joss.readthedocs.io/en/latest/paper.html). + +## Build the paper + +From the `autolens_assistant` repository root, compile with the official JOSS +Inara image: + +```bash +docker run --rm \ + --volume "$PWD/paper:/data" \ + --user "$(id -u):$(id -g)" \ + --env JOURNAL=joss \ + openjournals/inara +``` + +The generated PDF is written to `paper/paper.pdf`. diff --git a/paper/paper.bib b/paper/paper.bib new file mode 100644 index 0000000..93dabfa --- /dev/null +++ b/paper/paper.bib @@ -0,0 +1,86 @@ +% Bibliography for the PyAutoLens-Assistant JOSS paper. +% Entries are copied verbatim from the assistant literature bibliography +% (wiki/literature/bibliography/autolens_literature.bib) so they stay consistent +% with the wiki. Verify every entry before submission. + +@article{Casey2023, +archivePrefix = {arXiv}, +arxivId = {2211.07865}, +author = {Casey, Caitlin M. and Kartaltepe, Jeyhan S. and Drakos, Nicole E. and Franco, Maximilien and Harish, Santosh and Paquereau, Louise and Ilbert, Olivier and Rose, Caitlin and Cox, Isabella G. and Nightingale, James W. and Robertson, Brant E. and Silverman, John D. and Koekemoer, Anton M. and Massey, Richard and McCracken, Henry Joy and Rhodes, Jason and Akins, Hollis B. and Allen, Natalie and Amvrosiadis, Aristeidis and Arango-Toro, Rafael C. and Bagley, Micaela B. and Bongiorno, Angela and Capak, Peter L. and Champagne, Jaclyn B. and Chartab, Nima and {Ch{\'{a}}vez Ortiz}, {\'{O}}scar A. and Chworowsky, Katherine and Cooke, Kevin C. and Cooper, Olivia R. and Darvish, Behnam and Ding, Xuheng and Faisst, Andreas L. and Finkelstein, Steven L. and Fujimoto, Seiji and Gentile, Fabrizio and Gillman, Steven and Gould, Katriona M. L. and Gozaliasl, Ghassem and Hayward, Christopher C. and He, Qiuhan and Hemmati, Shoubaneh and Hirschmann, Michaela and Jahnke, Knud and Jin, Shuowen and Khostovan, Ali Ahmad and Kokorev, Vasily and Lambrides, Erini and Laigle, Clotilde and Larson, Rebecca L. and Leung, Gene C. K. and Liu, Daizhong and Liaudat, Tobias and Long, Arianna S. and Magdis, Georgios and Mahler, Guillaume and Mainieri, Vincenzo and Manning, Sinclaire M. and Maraston, Claudia and Martin, Crystal L. and McCleary, Jacqueline E. and McKinney, Jed and McPartland, Conor J. R. and Mobasher, Bahram and Pattnaik, Rohan and Renzini, Alvio and Rich, R. Michael and Sanders, David B. and Sattari, Zahra and Scognamiglio, Diana and Scoville, Nick and Sheth, Kartik and Shuntov, Marko and Sparre, Martin and Suzuki, Tomoko L. and Talia, Margherita and Toft, Sune and Trakhtenbrot, Benny and Urry, C. Megan and Valentino, Francesco and Vanderhoof, Brittany N. and Vardoulaki, Eleni and Weaver, John R. and Whitaker, Katherine E. and Wilkins, Stephen M. and Yang, Lilan and Zavala, Jorge A.}, +doi = {10.3847/1538-4357/acc2bc}, +eprint = {2211.07865}, +issn = {0004-637X}, +journal = {ApJ}, +number = {1}, +pages = {31}, +title = {{COSMOS-Web: An Overview of the JWST Cosmic Origins Survey}}, +volume = {954}, +year = {2023} +} + +@article{Nightingale2021, +author = {Nightingale, James. and Hayes, Richard and Kelly, Ashley and Amvrosiadis, Aristeidis and Etherington, Amy and He, Qiuhan and Li, Nan and Cao, XiaoYue and Frawley, Jonathan and Cole, Shaun and Enia, Andrea and Frenk, Carlos and Harvey, David and Li, Ran and Massey, Richard and Negrello, Mattia and Robertson, Andrew}, +doi = {10.21105/joss.02825}, +issn = {2475-9066}, +journal = {J. Open Source Softw.}, +number = {58}, +pages = {2825}, +title = {{PyAutoLens: Open-Source Strong Gravitational Lensing}}, +volume = {6}, +year = {2021} +} + +@article{EuclidCollaboration2025, +archivePrefix = {arXiv}, +arxivId = {2503.15324}, +author = {{Euclid Collaboration} and Walmsley, M. and Holloway, P. and Lines, N. E. P. and Rojas, K. and Collett, T. E. and Verma, A. and Li, T. and Nightingale, J. W. and Despali, G. and Schuldt, S. and Gavazzi, R. and Melo, A. and Metcalf, R. B. and Andika, I. T. and Leuzzi, L. and Manj{\'{o}}n-Garc{\'{i}}a, A. and Pearce-Casey, R. and Vincken, S. H. and Wilde, J. and Busillo, V. and Tortora, C. and Barroso, J. A. Acevedo and Dole, H. and Ecker, L. R. and Pearson, J. and Marshall, P. J. and More, A. and Saifollahi, T. and Gracia-Carpio, J. and Baeten, E. and Cornen, C. and Johnson, L. C. and Macmillan, C. and Kruk, S. and Remmelgas, K. A. and Cl{\'{e}}ment, B. and Degaudenzi, H. and Courbin, F. and Bovy, J. and Casas, S. and Dannerbauer, H. and Diego, J. M. and Finner, K. and Galan, A. and Giocoli, C. and Hogg, N. B. and Jahnke, K. and Katona, J. and Kov{\'{a}}cs, A. and {De Leo}, C. and Mahler, G. and Millon, M. and Nagam, B. C. and Nugent, P. and de Murieta, A. Sainz and O'Riordan, C. M. and Sluse, D. and Sonnenfeld, A. and Spiniello, C. and Serjeant, S. and Thai, T. T. and Ulivi, L. and Walth, G. L. and Weisenbach, L. and Zumalacarregui, M. and Aghanim, N. and Altieri, B. and Amara, A. and Andreon, S. and Auricchio, N. and Aussel, H. and Baccigalupi, C. and Baldi, M. and Balestra, A. and Bardelli, S. and Battaglia, P. and Bernardeau, F. and Biviano, A. and Bonchi, A. and Bonino, D. and Branchini, E. and Brescia, M. and Brinchmann, J. and Camera, S. and Ca{\~{n}}as-Herrera, G. and Capobianco, V. and Carbone, C. and Cardone, V. F. and Carretero, J. and Castander, F. J. and Castellano, M. and Castignani, G. and Cavuoti, S. and Chambers, K. C. and Cimatti, A. and Colodro-Conde, C. and Congedo, G. and Conselice, C. J. and Conversi, L. and Copin, Y. and Corcione, L. and Courtois, H. M. and Cropper, M. and {Da Silva}, A. and {De Lucia}, G. and {Di Giorgio}, A. M. and Dolding, C. and Dubath, F. and Duncan, C. A. J. and Dupac, X. and Ealet, A. and Escoffier, S. and Fabricius, M. and Farina, M. and Farinelli, R. and Faustini, F. and Finelli, F. and Fotopoulou, S. and Frailis, M. and Franceschi, E. and Fumana, M. and Galeotta, S. and George, K. and Gillard, W. and Gillis, B. and G{\'{o}}mez-Alvarez, P. and Granett, B. R. and Grazian, A. and Grupp, F. and Guzzo, L. and Gwyn, S. and Haugan, S. V. H. and Hoekstra, H. and Holmes, W. and Hook, I. M. and Hormuth, F. and Hornstrup, A. and Hudelot, P. and Jhabvala, M. and Joachimi, B. and Keih{\"{a}}nen, E. and Kermiche, S. and Kiessling, A. and Kubik, B. and K{\"{u}}mmel, M. and Kunz, M. and Kurki-Suonio, H. and Lahav, O. and Boulc'h, Q. Le and Brun, A. M. C. Le and Mignant, D. Le and Ligori, S. and Lilje, P. B. and Lindholm, V. and Lloro, I. and Mainetti, G. and Maino, D. and Maiorano, E. and Mansutti, O. and Marcin, S. and Marggraf, O. and Martinelli, M. and Martinet, N. and Marulli, F. and Massey, R. and Maurogordato, S. and McCracken, H. J. and Medinaceli, E. and Mei, S. and Mellier, Y. and Meneghetti, M. and Merlin, E. and Meylan, G. and Mora, A. and Moresco, M. and Moscardini, L. and Nakajima, R. and Neissner, C. and Nichol, R. C. and Niemi, S. -M. and Padilla, C. and Paltani, S. and Pasian, F. and Pedersen, K. and Percival, W. J. and Pettorino, V. and Pires, S. and Polenta, G. and Poncet, M. and Popa, L. A. and Pozzetti, L. and Raison, F. and Rebolo, R. and Renzi, A. and Rhodes, J. and Riccio, G. and Romelli, E. and Roncarelli, M. and Saglia, R. and Sakr, Z. and S{\'{a}}nchez, A. G. and Sapone, D. and Sartoris, B. and Schewtschenko, J. A. and Schirmer, M. and Schneider, P. and Schrabback, T. and Secroun, A. and Seidel, G. and Seiffert, M. and Serrano, S. and Simon, P. and Sirignano, C. and Sirri, G. and Mancini, A. Spurio and Stanco, L. and Steinwagner, J. and Tallada-Cresp{\'{i}}, P. and Taylor, A. N. and Tereno, I. and Tessore, N. and Toft, S. and Toledo-Moreo, R. and Torradeflot, F. and Tutusaus, I. and Valentijn, E. A. and Valenziano, L. and Valiviita, J. and Vassallo, T. and Kleijn, G. Verdoes and Veropalumbo, A. and Wang, Y. and Weller, J. and Zacchei, A. and Zamorani, G. and Zerbi, F. M. and Zucca, E. and Allevato, V. and Ballardini, M. and Bolzonella, M. and Bozzo, E. and Burigana, C. and Cabanac, R. and Cappi, A. and {Di Ferdinando}, D. and Vigo, J. A. Escartin and Gabarra, L. and Huertas-Company, M. and Mart{\'{i}}n-Fleitas, J. and Matthew, S. and Mauri, N. and Pezzotta, A. and P{\"{o}}ntinen, M. and Porciani, C. and Risso, I. and Scottez, V. and Sereno, M. and Tenti, M. and Viel, M. and Wiesmann, M. and Akrami, Y. and Anselmi, S. and Archidiacono, M. and Atrio-Barandela, F. and Benoist, C. and Benson, K. and Bergamini, P. and Bertacca, D. and Bethermin, M. and Blot, L. and Borgani, S. and Brown, M. L. and Bruton, S. and Calabro, A. and Quevedo, B. Camacho and Caro, F. and Carvalho, C. S. and Castro, T. and Charles, Y. and Cogato, F. and Cooray, A. R. and Cucciati, O. and Davini, S. and {De Paolis}, F. and Desprez, G. and D{\'{i}}az-S{\'{a}}nchez, A. and Diaz, J. J. and {Di Domizio}, S. and Enia, A. and Fang, Y. and Ferrari, A. G. and Finoguenov, A. and Fontana, A. and Franco, A. and Ganga, K. and Garc{\'{i}}a-Bellido, J. and Gasparetto, T. and Gautard, V. and Gaztanaga, E. and Giacomini, F. and Gozaliasl, G. and Guidi, M. and Gutierrez, C. M. and Hall, A. and Hartley, W. G. and Hemmati, S. and Hern{\'{a}}ndez-Monteagudo, C. and Hildebrandt, H. and Hjorth, J. and Kajava, J. J. E. and Kang, Y. and Kansal, V. and Karagiannis, D. and Kiiveri, K. and Kirkpatrick, C. C. and Graet, J. Le and Legrand, L. and Lembo, M. and Lepori, F. and Leroy, G. and Lesci, G. F. and Lesgourgues, J. and Liaudat, T. I. and Loureiro, A. and Macias-Perez, J. and Maggio, G. and Magliocchetti, M. and Magnier, E. A. and Mannucci, F. and Maoli, R. and Martins, C. J. A. P. and Maurin, L. and Miluzio, M. and Monaco, P. and Moretti, C. and Morgante, G. and Murray, C. and Nadathur, S. and Naidoo, K. and Navarro-Alsina, A. and Nesseris, S. and Passalacqua, F. and Paterson, K. and Patrizii, L. and Pisani, A. and Potter, D. and Quai, S. and Radovich, M. and Rocci, P. -F. and Sacquegna, S. and Sahl{\'{e}}n, M. and Sanders, D. B. and Sarpa, E. and Scarlata, C. and Schaye, J. and Schneider, A. and Sciotti, D. and Sellentin, E. and Shankar, F. and Smith, L. C. and Tanidis, K. and Testera, G. and Teyssier, R. and Tosi, S. and Troja, A. and Tucci, M. and Valieri, C. and Venhola, A. and Vergani, D. and Vernardos, G. and Verza, G. and Vielzeuf, P. and Walton, N. A. and Scott, D.}, +eprint = {2503.15324}, +journal = {https://arxiv.org/abs/2503.15324}, +title = {{Euclid Quick Data Release (Q1): The Strong Lensing Discovery Engine A -- System overview and lens catalogue}}, +url = {http://arxiv.org/abs/2503.15324}, +year = {2025} +} + +@article{Minor2021, +archivePrefix = {arXiv}, +arxivId = {2011.10627}, +author = {Minor, Quinn and Gad-Nasr, Sophia and Kaplinghat, Manoj and Vegetti, Simona}, +doi = {10.1093/mnras/stab2247}, +eprint = {2011.10627}, +issn = {13652966}, +journal = {MNRAS}, +number = {2}, +pages = {1662--1683}, +title = {{An unexpected high concentration for the dark substructure in the gravitational lens SDSSJ0946+1006}}, +volume = {507}, +year = {2021} +} + +@article{LSSTDarkEnergyScienceCollaboration2012, +archivePrefix = {arXiv}, +arxivId = {1211.0310}, +author = {{LSST Dark Energy Science Collaboration}}, +eprint = {1211.0310}, +journal = {arXiv preprint arXiv:1211.0310}, +pages = {133}, +title = {{Large Synoptic Survey Telescope: Dark Energy Science Collaboration}}, +url = {http://arxiv.org/abs/1211.0310}, +year = {2012} +} + +@article{Vegetti2010, +archivePrefix = {arXiv}, +arxivId = {0910.0760}, +author = {Vegetti, S. and Koopmans, L. V.E. and Bolton, A. and Treu, T. and Gavazzi, R.}, +doi = {10.1111/j.1365-2966.2010.16865.x}, +eprint = {0910.0760}, +isbn = {063114806X}, +issn = {00358711}, +journal = {MNRAS}, +month = {nov}, +number = {4}, +pages = {1969--1981}, +title = {{Detection of a dark substructure through gravitational imaging}}, +volume = {408}, +year = {2010} +} diff --git a/paper/paper.md b/paper/paper.md new file mode 100644 index 0000000..fe4fd82 --- /dev/null +++ b/paper/paper.md @@ -0,0 +1,95 @@ +--- +title: "PyAutoLens-Assistant: Using Natural Language and AI to Analyse Gravitational Lenses" +tags: + - Python + - astronomy + - gravitational lensing + - artificial intelligence + - large language models + - natural language interfaces +authors: + - name: James W. Nightingale + orcid: 0000-0002-8987-7401 + affiliation: 1 + corresponding: true +affiliations: + - name: Institute for Computational Cosmology, Durham University, United Kingdom + index: 1 +date: 15 July 2026 +bibliography: paper.bib +--- + +# Summary + +Stage IV weak-lensing surveys, such as Euclid [@EuclidCollaboration2025] and the Vera C. Rubin Observatory [@LSSTDarkEnergyScienceCollaboration2012], are measuring increasingly large samples of galaxies, while strong-lensing searches are discovering rapidly growing numbers of galaxy-, group-, and cluster-scale lenses. These systems are observed through optical and infrared imaging, radio interferometry, point-source measurements of lensed quasars and supernovae, and weak-lensing shear catalogues, enabling studies of cosmology, dark matter, galaxy formation, and the early Universe. Mature open-source software such as PyAutoLens [@Nightingale2021] supports simulations, lensing calculations, and strong- and weak-lensing modelling across these datasets, but constructing a bespoke analysis can still require substantial effort to locate, adapt, and combine the relevant examples using the correct Python API and syntax. + +PyAutoLens-Assistant allows scientists to use natural language to describe the gravitational-lens analysis they want to perform. It provides a domain-specific interface to the documented and tested capabilities of PyAutoLens, supporting simulations, ray-tracing calculations, probabilistic modelling, data preparation, result interpretation, and visualization. Researchers can use it through a conversational AI assistant, such as ChatGPT, to ask questions and develop workflows interactively, or through agentic coding tools, such as Claude Code or Codex, which can inspect data, write and execute scripts, diagnose errors, analyse outputs, and iteratively refine an analysis. PyAutoLens-Assistant is grounded in curated, version-controlled documentation, examples, scientific reference material, and task-specific instructions, and produces explicit Python code and inspectable analysis products. + +# Statement of need + +Experienced PyAutoLens users often know exactly which scientific analysis they want to perform, but implementing it still requires substantial time assembling the appropriate Python workflow. An expert can quickly specify: "Perform multi-wavelength lens modelling of the F115W, F150W, F277W, and F444W JWST imaging of the COSMOS-Web Ring [@Casey2023] using a multi-Gaussian expansion lens-light model, a singular isothermal ellipsoid plus external shear mass model, and a Delaunay pixelized source reconstruction." Translating this concise scientific specification into executable code requires locating and combining several examples, loading and configuring each dataset, composing the model components with the correct API, and adapting the workflow to the system being analysed. As models incorporate more datasets, cluster-scale mass distributions, or joint strong- and weak-lensing constraints, this implementation burden increases even when the underlying scientific choices are already clear. PyAutoLens-Assistant reduces this overhead by translating natural-language specifications into explicit, executable, and reproducible PyAutoLens workflows. + +New users face a complementary challenge: they may not yet know which modelling approach, software abstractions, or examples are appropriate for the task they are learning. PyAutoLens has grown from galaxy-scale imaging analyses to support point-source lenses, group- and cluster-scale systems, weak lensing, interferometry, simulations, and joint analyses, accompanied by well over one hundred worked examples across the `autolens_workspace`. Navigating this material while simultaneously learning gravitational-lensing science, Bayesian inference, and the PyAutoLens API can be overwhelming. PyAutoLens-Assistant enables users to describe their immediate goal in natural language and receive targeted explanations, example code, and pointers to the relevant documentation. Its teaching mode also explains the underlying science and numerical methods, encourages follow-up questions, and supports learning rather than simply returning code. + +# State of the field + + + +# Software design + +PyAutoLens-Assistant is a version-controlled knowledge and workflow layer that enables general-purpose AI systems to use PyAutoLens reliably. Its architecture separates three components: instructions define assistant behaviour, skills describe how to perform specific tasks, and wiki pages provide the underlying technical and scientific knowledge. For a given request, the assistant selects the relevant skill, consults the associated wiki material, and adapts tested examples from the `autolens_workspace` rather than generating PyAutoLens code from memory. Generated scripts follow the established workspace structure and can be checked against the installed API, reducing the risk of outdated or invented syntax. + +## Reference wikis + +Two reference wikis provide complementary context. The core wiki organizes the PyAutoLens API, modelling concepts, datasets, inference methods, and operational guidance, linking these to procedural skills and relevant workspace examples. The literature wiki provides scientific context through pages on lensing concepts, named surveys and systems, and bibliographies of published papers. Users can also ingest papers relevant to a project, after which they become part of the assistant's persistent scientific context. + +## Access modes + +PyAutoLens-Assistant can be used through a browser-based conversational assistant or a local agentic coding tool. For systems such as ChatGPT or Claude, `llms.txt` acts as the machine-readable entry point: it asks the assistant to verify repository access and directs it through the canonical read order of instructions, skills, relevant wiki pages, and runnable workspace examples. In this mode, users can ask questions, receive scientific explanations, locate examples, interpret errors and figures, and generate draft end-to-end scripts, although the assistant cannot normally inspect local files or execute code. + +For full computational workflows, PyAutoLens-Assistant can instead be used with agentic tools such as Claude Code or Codex. These tools load the repository instructions directly and can inspect datasets, write and run scripts, generate diagnostic plots, debug failures, and iteratively refine an analysis. The resulting Python code, configuration, outputs, and modelling decisions remain explicit and inspectable. + +## Interaction modes and project structure + +The assistant operates in two interaction modes. **Assistant mode** is intended for users who want a task completed efficiently, with concise explanations and support ranging from interactive coding to phased end-to-end analysis. **Teacher mode** prioritizes learning by explaining what each stage does and why, making assumptions explicit, and directing users to relevant documentation and examples. Both modes use the same scientific capabilities, reproducibility requirements, and safety checks. + +For agentic work, each analysis can be stored in a separate project repository containing its data, configuration, scripts, results, and project journal. This separates the shared assistant knowledge base from the scientific project while preserving a complete record that can be shared with collaborators or released alongside a publication. + +# Benchmark examples + +PyAutoLens-Assistant is evaluated using a suite of frozen benchmark prompts distributed with the repository. We describe three representative examples here, which span progressively more demanding scientific workflows and are run using multiple conversational and agentic AI systems. Each benchmark records the full interaction, generated code, executed analysis where applicable, scientific outputs, and a rubric-based score, enabling direct comparison between different models, tools, and interaction modes. + +The first benchmark uses **Teacher mode** to simulate Euclid-like imaging of a simple strong lens, fit the simulated data, and recover the lens model. The assistant must explain the purpose of each stage, including model composition, simulation, masking, non-linear inference, and interpretation of the recovered parameters. This benchmark tests whether the assistant can provide scientifically accurate guidance while helping a new user understand an end-to-end PyAutoLens workflow. + +The second benchmark uses **Assistant mode** to model JWST imaging of the COSMOS-Web Ring [@Casey2023]. The assistant must inspect the supplied dataset, perform the required data-preparation steps, construct an appropriate lens-light and mass model with a pixelized source reconstruction, run the analysis, and present the reconstructed source and fit residuals. This benchmark tests the assistant's ability to convert a concise scientific request into a complete and reproducible modelling workflow with limited user intervention. + +The third benchmark requests a more autonomous analysis of the strong lens SDSSJ0946+1006. The assistant must reproduce a reported dark-matter subhalo detection [@Vegetti2010] through Bayesian model comparison, compare alternative subhalo mass profiles to test the reported high concentration of the perturber [@Minor2021], preserve all intermediate models and results for inspection, and determine whether the analysis should run locally or on high-performance computing resources. This benchmark tests long-horizon planning, scientific decision-making, project-state management, and the ability to execute a complex analysis across multiple stages. + +The benchmark suite is run across different AI systems and access modes, including browser-based conversational assistants and local agentic coding tools. Results will be reported using metrics such as task completion, scientific correctness, API validity, reproducibility, degree of autonomy, number of user interventions, wall-clock time, and computational cost. Together, the benchmarks test the two principal use cases of PyAutoLens-Assistant: teaching new users how to perform gravitational-lens analyses and enabling experienced users to execute complex workflows efficiently from natural-language specifications. + +# Research impact statement + + + +# AI usage disclosure + +Generative AI tools were used to scaffold this manuscript and may be used to assist drafting. All scientific claims, citations, and prose are reviewed and verified by the authors. + +# Acknowledgements + + + +# References