diff --git a/CLAUDE.md b/CLAUDE.md index 80c94557..b56c2190 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -96,9 +96,11 @@ Sub-packages within `scripts/` and `notebooks/`: - `interferometer/` — ALMA/JVLA uv-plane modeling - `multi/` — Multi-wavelength simultaneous modeling - `guides/` — API guides: modeling, results, plotting, over-sampling, HPC -- `howtogalaxy/` — Tutorial lecture series - `ellipse/` — Ellipse fitting (non-parametric morphology) +The tutorial lecture series now lives in its own repository at +`https://github.com/PyAutoLabs/HowToGalaxy`. + Each data-type package contains: `modeling.py`, `simulator.py`, `fit.py`, `features/`, `data_preparation/`, `likelihood_function.py`. ## Core API Patterns diff --git a/README.rst b/README.rst index 87c5b684..c9d86ce6 100644 --- a/README.rst +++ b/README.rst @@ -8,8 +8,8 @@ PyAutoGalaxy Workspace `Installation Guide `_ | `readthedocs `_ | -`Introduction on Colab `_ -`HowToGalaxy `_ +`Introduction on Colab `_ | +`HowToGalaxy `_ Welcome to the **PyAutoGalaxy** Workspace! @@ -38,18 +38,10 @@ to navigate the workspace for your science case. HowToGalaxy ----------- -For experienced scientists, the run through above will have been a breeze. Concepts surrounding galaxy structure and -morphology were already familiar and the statistical techniques used for fitting and modeling already understood. +If you are new to galaxy modeling or the statistical techniques it relies on, the **HowToGalaxy** lecture series +takes you from first principles through to modeling real galaxy imaging data. It now lives in its own repository: -For those less familiar with these concepts (e.g. undergraduate students, new PhD students or interested members of the -public), things may have been less clear and a slower more detailed explanation of each concept would be beneficial. - -The **HowToGalaxy** Jupyter Notebook lectures are provide exactly this They are a 3+ chapter guide which thoroughly -take you through the core concepts of galaxy light profiles, teach you the principles ofthe statistical techniques -used in modeling and ultimately will allow you to undertake scientific research like a professional astronomer. - -If this sounds like it suits you, checkout the `autogalaxy_workspace/notebooks/howtogalaxy` package now, its it -recommended you go here before anywhere else! +https://github.com/PyAutoLabs/HowToGalaxy Workspace Structure ------------------- @@ -68,7 +60,6 @@ The examples in the ``notebooks`` and ``scripts`` folders are structured as foll - ``imaging``: Examples for galaxy modeling using CCD imaging (e.g. Hubble, James Webb, Euclid). - ``interferometer``: Examples for galaxies observed with an interferometer (e.g. ALMA, JVLA). - ``multi``: Examples for modeling galaxies observed in multiple wavebands. -- ``howtogalaxy``: Conceptual tutorials introducing galaxy modeling and inference with **PyAutoGalaxy**. The dataset packages (e.g. ``imaging``, ``interferometer`` and ``multi``) include the following types of examples: diff --git a/config/build/env_vars.yaml b/config/build/env_vars.yaml index bf40d72c..1226999c 100644 --- a/config/build/env_vars.yaml +++ b/config/build/env_vars.yaml @@ -23,8 +23,6 @@ defaults: overrides: # Non-simulator scripts that load committed FITS data need full-size # datasets to avoid shape mismatch with pre-existing 100x100 data. - - pattern: "howtogalaxy/" - unset: [PYAUTO_SMALL_DATASETS] - pattern: "guides/" unset: [PYAUTO_SMALL_DATASETS] # guides/results/start_here.py must produce real samples so the example diff --git a/config/build/no_run.yaml b/config/build/no_run.yaml index 4e9e10f1..f2bc3915 100644 --- a/config/build/no_run.yaml +++ b/config/build/no_run.yaml @@ -17,7 +17,6 @@ # banner. Investigate the failure, fix the underlying bug, and remove # the NEEDS_FIX marker. -- tutorial_searches - guides/results/start_here # SLOW 2026-04-10 - exceeds 60s test timeout; unsets TEST_MODE to produce real samples for downstream examples - guides/results/examples/galaxies_fit # SLOW 2026-04-10 - exceeds 60s test timeout; unsets TEST_MODE for downstream examples - guides/results/examples/samples # SLOW 2026-04-10 - exceeds 60s test timeout; unsets TEST_MODE for downstream examples @@ -35,8 +34,6 @@ - ellipse/database # Ellipse model needs refactor and JAX support - ellipse/modeling # NEEDS_FIX 2026-04-10 - KeyError on 'ellipses.0.centre_0' kwargs after API drift in ellipse model - guides/advanced/over_sampling # NEEDS_FIX 2026-04-10 - plot_grid() got unexpected kwarg 'plot_grid_lines' after plotter API drift -- howtogalaxy/chapter_4_pixelizations/tutorial_2_mappers # NEEDS_FIX 2026-04-10 - IndexError in mapper tutorial, likely empty mapping array -- howtogalaxy/chapter_4_pixelizations/tutorial_5_model_fit # NEEDS_FIX 2026-04-10 - LinAlgError: matrix not positive definite in pixelization fit - imaging/data_preparation/manual/mask_irregular # NEEDS_FIX 2026-04-10 - silent failure, needs investigation - imaging/features/pixelization/modeling # NEEDS_FIX 2026-04-10 - LinAlgError: matrix not positive definite in pixelization modeling - autogalaxy_workspace/scripts/imaging/modeling # NEEDS_FIX 2026-04-10 - KeyError on ('galaxies','galaxy','bulge','ell_comps'...) kwargs after API drift in top-level imaging/modeling.py diff --git a/notebooks/README.rst b/notebooks/README.rst index 61bd965c..35be7dd6 100644 --- a/notebooks/README.rst +++ b/notebooks/README.rst @@ -15,4 +15,6 @@ Folders - ``interferometer``: Examples for galaxies observed with an interferometer (e.g. ALMA, JVLA). - ``ellipse``: Examples for perform ellipse isophote fitting on galaxy images. - ``multi``: Examples for multiple datasets simultaneously (E.g. multi-wavelength imaging, imaging and interferometry). -- ``howtogalaxy``: The **HowToGalaxy** lectures which teach inexperienced scientists what galaxy fitting is and how to use **PyAutoGalaxy**. \ No newline at end of file + +The **HowToGalaxy** lecture series, which teaches inexperienced scientists what galaxy fitting is and how to use +**PyAutoGalaxy**, now lives in its own repository: https://github.com/PyAutoLabs/HowToGalaxy \ No newline at end of file diff --git a/notebooks/ellipse/modeling.ipynb b/notebooks/ellipse/modeling.ipynb index 4b441f98..03b47db4 100644 --- a/notebooks/ellipse/modeling.ipynb +++ b/notebooks/ellipse/modeling.ipynb @@ -1,826 +1,826 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Modeling\n", - "========\n", - "\n", - "This guide shows how to perform ellipse fitting modeling on data using a non-linear search, including visualizing and\n", - "interpreting its results.\n", - "\n", - "__Fit__\n", - "\n", - "The non-linear search in this example calls a `log_likelihood_function` using the `Analysis` class many times, in\n", - "order to determine ellipse parameters and therefore overall distribution of ellipses that best-fit the data.\n", - "\n", - "The `log_likelihood_function` and how the ellipses are used to fit the data are described in the `fit.py` script,\n", - "which you should read first in order to better understand how ellipse fitting works.\n", - "\n", - "__Units__\n", - "\n", - "In this example, all quantities are **PyAutoGalaxy**'s internal unit coordinates, with spatial coordinates in\n", - "arc seconds, luminosities in electrons per second and mass quantities (e.g. convergence) are dimensionless.\n", - "\n", - "The guide `guides/units_and_cosmology.ipynb` illustrates how to convert these quantities to physical units like\n", - "kiloparsecs, magnitudes and solar masses.\n", - "\n", - "__Data Structures__\n", - "\n", - "Quantities inspected in this example script use **PyAutoGalaxy** bespoke data structures for storing arrays, grids,\n", - "vectors and other 1D and 2D quantities. These use the `slim` and `native` API to toggle between representing the\n", - "data in 1D numpy arrays or high dimension numpy arrays.\n", - "\n", - "This tutorial will only use the `slim` properties which show results in 1D numpy arrays of\n", - "shape [total_unmasked_pixels]. This is a slimmed-down representation of the data in 1D that contains only the\n", - "unmasked data points\n", - "\n", - "These are documented fully in the `autogalaxy_workspace/*/guides/data_structures.ipynb` guide.\n", - "\n", - "__Contents__\n", - "\n", - "**Loading Data:** Loading the imaging dataset from FITS files for ellipse fitting.\n", - "**Dataset Auto-Simulation:** Automatically simulating data if it does not exist.\n", - "**Mask:** Applying a circular mask to the dataset.\n", - "**Model Composition:** Composing an ellipse model with free centre and elliptical components.\n", - "**Search:** Configuring the Dynesty non-linear search for ellipse fitting.\n", - "**Analysis:** Setting up the AnalysisEllipse object for the model fit.\n", - "**Run Times:** Estimating the computational cost of the ellipse model fit.\n", - "**Model-Fit:** Running the non-linear search to fit the ellipse to data.\n", - "**Output Folder:** Description of the results written to hard-disk during and after the fit.\n", - "**Result:** Inspecting the result object and maximum likelihood ellipse parameters.\n", - "**Multiple Ellipses:** Fitting many ellipses of increasing size to trace the full galaxy.\n", - "**Final Fit:** Combining all ellipses into a single final fit.\n", - "**Masking:** Applying an extra galaxies mask and repeating the ellipse fitting.\n", - "**Data Preparation:** Pointers to data preparation scripts for your own data.\n", - "**HowToGalaxy:** Pointers to the HowToGalaxy lecture series for light profile modeling." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autoconf import setup_notebook; setup_notebook()\n", - "\n", - "import numpy as np\n", - "from pathlib import Path\n", - "import autofit as af\n", - "import autofit.plot as aplt_af\n", - "import autogalaxy as ag\n", - "import autogalaxy.plot as aplt" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Loading Data__\n", - "\n", - "We we begin by loading the galaxy dataset `simple` from .fits files, which is the dataset we will use to demonstrate \n", - "ellipse fitting.\n", - "\n", - "This uses the `Imaging` object used in other examples.\n", - "\n", - "Ellipse fitting does not use the Point Spread Function (PSF) of the dataset, so we do not need to load it." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_name = \"ellipse\"\n", - "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset Auto-Simulation__\n", - "\n", - "If the dataset does not already exist on your system, it will be created by running the corresponding\n", - "simulator script. This ensures that all example scripts can be run without manually simulating data first." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "if not dataset_path.exists():\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.run(\n", - " [sys.executable, \"scripts/ellipse/simulator.py\"],\n", - " check=True,\n", - " )\n", - "\n", - "\n", - "dataset = ag.Imaging.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " pixel_scales=0.1,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can use the `Imaging` to plot the image and noise-map of the dataset." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(array=dataset.data, title=\"Data\")\n", - "aplt.plot_array(array=dataset.noise_map, title=\"Noise Map\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can also plot a subplot which shows all these properties simultaneously." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_imaging_dataset(dataset=dataset)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Mask__\n", - "\n", - "We now mask the data, so that regions where there is no signal (e.g. the edges) are omitted from the fit.\n", - "\n", - "We use a `Mask2D` object, which for this example is 4.0\" circular mask.\n", - "\n", - "For ellipse fitting, the mask radius defines the region of the image that the ellipses are fitted over. We therefore\n", - "define the `mask_radius` as a variable which is used below to define the sizes of the ellipses in the model fitting." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "mask_radius = 5.0\n", - "\n", - "mask = ag.Mask2D.circular(\n", - " shape_native=dataset.shape_native,\n", - " pixel_scales=dataset.pixel_scales,\n", - " radius=mask_radius,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We now combine the imaging dataset with the mask." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset = dataset.apply_mask(mask=mask)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We now plot the image with the mask applied, where the image automatically zooms around the mask to make the galaxy\n", - "appear bigger." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(array=dataset.data, title=\"Image Data With Mask Applied\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The mask is also used to compute a `Grid2D`, where the (y,x) arc-second coordinates are only computed in unmasked\n", - "pixels within the masks' circle.\n", - "\n", - "As shown in the previous overview example, this grid will be used to perform galaxying calculations when fitting the\n", - "data below." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_grid(grid=dataset.grid, title=\"Grid2D of Masked Dataset\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Model Composition__\n", - "\n", - "The API below for composing a model uses the `Model` and `Collection` objects, which are imported from the \n", - "parent project **PyAutoFit** \n", - "\n", - "The API is fairly self explanatory and is straight forward to extend, for example adding more ellipses\n", - "to the galaxy.\n", - "\n", - "Ellipse fitting fits ellispes of increasing size to the data, one after another, with the properties of each ellipse\n", - "as a function of size being the main results of the model-fit.\n", - "\n", - "We therefore compose a model consistent of a single ellise to demonstrate this fitting process, and then towards\n", - "the end of the script we will extend the model to fit multiple ellipses.\n", - "\n", - "The model is composed of 1 ellipses as follows:\n", - "\n", - "1) The ellipse has a fixed sizes that is input manually. When multiple ellipses are fitted, this size will \n", - " incrementally grow in size in order to cover the entire galaxy.\n", - "\n", - "2) The centre and elliptical components of the ellipse are free, meaning that the model has N=4 free parameters.\n", - "\n", - "The model composition below uses a list even though there is one ellipse, as this format allows us to fit\n", - "multiple ellipses in the model-fit at once, albeit its rare we would want to do this.\n", - "\n", - "__Model Cookbook__\n", - "\n", - "A full description of model composition is provided by the model cookbook: \n", - "\n", - "https://pyautogalaxy.readthedocs.io/en/latest/general/model_cookbook.html\n", - "\n", - "__Coordinates__\n", - "\n", - "The model fitting default settings assume that the galaxy centre is near the coordinates (0.0\", 0.0\"). \n", - "\n", - "If for your dataset the galaxy is not centred at (0.0\", 0.0\"), we recommend that you either: \n", - "\n", - " - Reduce your data so that the centre is (`autogalaxy_workspace/*/preprocess`). \n", - " - Manually override the model priors (`autogalaxy_workspace/*/modeling/imaging/customize/priors.py`)." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "ellipse = af.Model(ag.Ellipse)\n", - "\n", - "ellipse.centre.centre_0 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1)\n", - "ellipse.centre.centre_1 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1)\n", - "\n", - "ellipse.ell_comps.ell_comps_0 = af.UniformPrior(lower_limit=-0.6, upper_limit=0.6)\n", - "ellipse.ell_comps.ell_comps_1 = af.UniformPrior(lower_limit=-0.6, upper_limit=0.6)\n", - "\n", - "ellipse.major_axis = 0.3\n", - "\n", - "model = af.Collection(ellipses=[ellipse])" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `info` attribute shows the model in a readable format.\n", - "\n", - "[The `info` below may not display optimally on your computer screen, for example the whitespace between parameter\n", - "names on the left and parameter priors on the right may lead them to appear across multiple lines. This is a\n", - "common issue in Jupyter notebooks.\n", - "\n", - "The`info_whitespace_length` parameter in the file `config/generag.yaml` in the [output] section can be changed to \n", - "increase or decrease the amount of whitespace (The Jupyter notebook kernel will need to be reset for this change to \n", - "appear in a notebook).]" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(model.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Search__\n", - "\n", - "The model is fitted to the data using a non-linear search. \n", - "\n", - "This example uses the nested sampling algorithm Dynesty (https://dynesty.readthedocs.io/en/stable/), which extensive \n", - "testing has revealed gives the most accurate and efficient modeling results for ellipse fitting.\n", - "\n", - "Dynesty has one main setting that trades-off accuracy and computational run-time, the number of `live_points`. \n", - "A higher number of live points gives a more accurate result, but increases the run-time. A lower value may give \n", - "less reliable modeling (e.g. the fit may infer a local maxima), but is faster. \n", - "\n", - "The suitable value depends on the model complexity whereby models with more parameters require more live points. \n", - "The default value of 200 is sufficient for the vast majority of ellipse fitting problems. Lower values often given \n", - "reliable results though, and speed up the run-times. \n", - "\n", - "__Unique Identifier__\n", - "\n", - "In the path above, the `unique_identifier` appears as a collection of characters, where this identifier is generated \n", - "based on the model, search and dataset that are used in the fit.\n", - "\n", - "An identical combination of model and search generates the same identifier, meaning that rerunning the script will use \n", - "the existing results to resume the model-fit. In contrast, if you change the model or search, a new unique identifier \n", - "will be generated, ensuring that the model-fit results are output into a separate folder.\n", - "\n", - "We additionally want the unique identifier to be specific to the dataset fitted, so that if we fit different datasets\n", - "with the same model and search results are output to a different folder. We achieve this below by passing \n", - "the `dataset_name` to the search's `unique_tag`.\n", - "\n", - "__Parallel Script__\n", - "\n", - "Depending on the operating system (e.g. Linux, Mac, Windows), Python version, if you are running a Jupyter notebook \n", - "and other factors, this script may not run a successful parallel fit (e.g. running the script \n", - "with `number_of_cores` > 1 will produce an error). It is also common for Jupyter notebooks to not run in parallel \n", - "correctly, requiring a Python script to be run, often from a command line terminal.\n", - "\n", - "To fix these issues, the Python script needs to be adapted to use an `if __name__ == \"__main__\":` API, as this allows\n", - "the Python `multiprocessing` module to allocate threads and jobs correctly. An adaptation of this example script \n", - "is provided at `autolens_workspace/scripts/modeling/imaging/customize/parallel.py`, which will hopefully run \n", - "successfully in parallel on your computer!\n", - "\n", - "Therefore if paralellization for this script doesn't work, check out the `parallel.py` example. You will need to update\n", - "all scripts you run to use the this format and API. \n", - "\n", - "__Iterations Per Update__\n", - "\n", - "Every N iterations, the non-linear search outputs the current results to the folder `autogalaxy_workspace/output`,\n", - "which includes producing visualization. \n", - "\n", - "Depending on how long it takes for the model to be fitted to the data (see discussion about run times below), \n", - "this can take up a large fraction of the run-time of the non-linear search.\n", - "\n", - "For this fit, the fit is very fast, thus we set a high value of `iterations_per_quick_update=10000` to ensure these updates\n", - "so not slow down the overall speed of the model-fit. \n", - "\n", - "**If the iteration per update is too low, the model-fit may be significantly slowed down by the time it takes to\n", - "output results and visualization frequently to hard-disk. If your fit is consistent displaying a log saying that it\n", - "is outputting results, try increasing this value to ensure the model-fit runs efficiently.**" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "search = af.DynestyStatic(\n", - " path_prefix=Path(\"ellipse\"),\n", - " name=f\"fit_start\",\n", - " unique_tag=dataset_name,\n", - " sample=\"rwalk\",\n", - " n_live=50,\n", - " iterations_per_quick_update=10000,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Analysis__\n", - "\n", - "We next create an `AnalysisEllipse` object, which can be given many inputs customizing how the model is fitted to the \n", - "data (in this example they are omitted for simplicity).\n", - "\n", - "Internally, this object defines the `log_likelihood_function` used by the non-linear search to fit the model to \n", - "the `Imaging` dataset. \n", - "\n", - "It is not vital that you as a user understand the details of how the `log_likelihood_function` fits a model to \n", - "data, but interested readers can find a step-by-step guide of the likelihood \n", - "function at ``autogalaxy_workspace/*/imaging/log_likelihood_function`\n", - "\n", - "__JAX__\n", - "\n", - "PyAutoLens uses JAX under the hood for fast GPU/CPU acceleration. However, ellipse fitting does not support JAX\n", - "acceleration, so we disable it here by passing `use_jax=False`." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "analysis = ag.AnalysisEllipse(dataset=dataset, use_jax=False)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Run Times__\n", - "\n", - "Modeling can be a computationally expensive process. When fitting complex models to high resolution datasets \n", - "run times can be of order hours, days, weeks or even months.\n", - "\n", - "Run times are dictated by two factors:\n", - "\n", - " - The log likelihood evaluation time: the time it takes for a single `instance` of the model to be fitted to \n", - " the dataset such that a log likelihood is returned.\n", - "\n", - " - The number of iterations (e.g. log likelihood evaluations) performed by the non-linear search: more complex\n", - " models require more iterations to converge to a solution.\n", - "\n", - "For this analysis, the log likelihood evaluation time is ~0.04 seconds, which is extremely fast for modeling. For higher resolution datasets ellipse\n", - "fitting can slow down to a likelihood evaluation time of order 0.5 - 1.0 second, which is still reasonably fast.\n", - "\n", - "To estimate the expected overall run time of the model-fit we multiply the log likelihood evaluation time by an \n", - "estimate of the number of iterations the non-linear search will perform. \n", - "\n", - "Estimating this quantity is more tricky, as it varies depending on the model complexity (e.g. number of parameters)\n", - "and the properties of the dataset and model being fitted.\n", - "\n", - "For this example, we conservatively estimate that the non-linear search will perform ~10000 iterations per free \n", - "parameter in the model. This is an upper limit, with models typically converging in far fewer iterations.\n", - "\n", - "If you perform the fit over multiple CPUs, you can divide the run time by the number of cores to get an estimate of\n", - "the time it will take to fit the model. Parallelization with Nautilus scales well, it speeds up the model-fit by the \n", - "`number_of_cores` for N < 8 CPUs and roughly `0.5*number_of_cores` for N > 8 CPUs. This scaling continues \n", - "for N> 50 CPUs, meaning that with super computing facilities you can always achieve fast run times!\n", - "\n", - "__Model-Fit__\n", - "\n", - "We can now begin the model-fit by passing the model and analysis object to the search, which performs a non-linear\n", - "search to find which models fit the data with the highest likelihood.\n", - "\n", - "Checkout the output folder for live outputs of the results of the fit, including on-the-fly visualization of the best \n", - "fit model!\n", - "\n", - "**Run Time Error:** On certain operating systems (e.g. Windows, Linux) and Python versions, the code below may produce \n", - "an error. If this occurs, see the `autolens_workspace/guides/modeling/bug_fix` example for a fix." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "result = search.fit(model=model, analysis=analysis)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Output Folder__\n", - "\n", - "Now this is running you should checkout the `autogalaxy_workspace/output` folder. This is where the results of the \n", - "search are written to hard-disk (in the `start_here` folder), where all outputs are human readable (e.g. as .json,\n", - ".csv or text files).\n", - "\n", - "As the fit progresses, results are written to the `output` folder on the fly using the highest likelihood model found\n", - "by the non-linear search so far. This means you can inspect the results of the model-fit as it runs, without having to\n", - "wait for the non-linear search to terminate.\n", - "\n", - "The `output` folder includes:\n", - "\n", - " - `model.info`: Summarizes the model, its parameters and their priors discussed in the next tutorial.\n", - "\n", - " - `model.results`: Summarizes the highest likelihood model inferred so far including errors.\n", - "\n", - " - `images`: Visualization of the highest likelihood model-fit to the dataset, (e.g. a fit subplot showing the \n", - " galaxies, model data and residuals).\n", - "\n", - " - `files`: A folder containing .fits files of the dataset, the model as a human-readable .json file, \n", - " a `.csv` table of every non-linear search sample and other files containing information about the model-fit.\n", - "\n", - " - search.summary: A file providing summary statistics on the performance of the non-linear search.\n", - "\n", - " - `search_internal`: Internal files of the non-linear search (in this case Nautilus) used for resuming the fit and\n", - " visualizing the search.\n", - "\n", - "__Result__\n", - "\n", - "The search returns a result object, which whose `info` attribute shows the result in a readable format.\n", - "\n", - "[Above, we discussed that the `info_whitespace_length` parameter in the config files could b changed to make \n", - "the `model.info` attribute display optimally on your computer. This attribute also controls the whitespace of the\n", - "`result.info` attribute.]" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(result.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `Result` object also contains:\n", - "\n", - " - The model corresponding to the maximum log likelihood solution in parameter space.\n", - " - The corresponding maximum log likelihood `Ellipse` and `FitEllipse` objects." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "instance = result.max_log_likelihood_instance\n", - "\n", - "print(\"Max Log Likelihood Model:\")\n", - "print(instance)\n", - "\n", - "print(f\"First Ellipse Centre: {instance.ellipses[0].centre}\")\n", - "print(f\"First Ellipse Elliptical Components: {instance.ellipses[0].ell_comps}\")\n", - "print(f\"First Ellipse Major Axis: {instance.ellipses[0].major_axis}\")\n", - "print(f\"First Ellipse Axis Ratio: {instance.ellipses[0].axis_ratio}\")\n", - "print(f\"First Ellipse Angle: {instance.ellipses[0].angle}\")\n", - "\n", - "for i, ellipse in enumerate(result.max_log_likelihood_instance.ellipses):\n", - " print(f\"Ellipse {i} Minor Axis: {ellipse.minor_axis}\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The maximum log likelihood fit is also available via the result, which can visualize the fit." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(array=dataset.data, title=\"Data\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The result contains the full posterior information of our non-linear search, including all parameter samples, \n", - "log likelihood values and tools to compute the errors on the model. \n", - "\n", - "There are built in visualization tools for plotting this.\n", - "\n", - "The plot is labeled with short hand parameter names (e.g. `sersic_index` is mapped to the short hand \n", - "parameter `n`). These mappings ate specified in the `config/notation.yaml` file and can be customized by users.\n", - "\n", - "The superscripts of labels correspond to the name each component was given in the model (e.g. for the `Isothermal`\n", - "mass its name `mass` defined when making the `Model` above is used)." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt_af.corner_cornerpy(samples=result.samples)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Multiple Ellipses__" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "number_of_ellipses = 10\n", - "\n", - "major_axis_list = np.linspace(0.3, mask_radius * 0.9, number_of_ellipses)\n", - "\n", - "total_ellipses = len(major_axis_list)\n", - "\n", - "result_list = []\n", - "\n", - "for i in range(len(major_axis_list)):\n", - " ellipse = af.Model(ag.Ellipse)\n", - "\n", - " ellipse.centre.centre_0 = result.instance.ellipses[0].centre[0]\n", - " ellipse.centre.centre_1 = result.instance.ellipses[0].centre[1]\n", - "\n", - " ellipse.ell_comps.ell_comps_0 = af.UniformPrior(lower_limit=-0.6, upper_limit=0.6)\n", - " ellipse.ell_comps.ell_comps_1 = af.UniformPrior(lower_limit=-0.6, upper_limit=0.6)\n", - "\n", - " ellipse.major_axis = major_axis_list[i]\n", - "\n", - " model = af.Collection(ellipses=[ellipse])\n", - "\n", - " search = af.DynestyStatic(\n", - " path_prefix=Path(\"ellipse\"),\n", - " name=f\"fit_{i}\",\n", - " unique_tag=dataset_name,\n", - " sample=\"rwalk\",\n", - " n_live=50,\n", - " number_of_cores=4,\n", - " iterations_per_quick_update=10000,\n", - " )\n", - "\n", - " analysis = ag.AnalysisEllipse(dataset=dataset, use_jax=False)\n", - "\n", - " result = search.fit(model=model, analysis=analysis)\n", - "\n", - " result_list.append(result)\n" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Final Fit__\n", - "\n", - "A final fit is performed combining all ellipses." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "ellipses = [result.instance.ellipses[0] for result in result_list]\n", - "\n", - "model = af.Collection(ellipses=ellipses)\n", - "\n", - "model.dummy_0 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1)\n", - "\n", - "search = af.Drawer(\n", - " path_prefix=Path(\"ellipse\"),\n", - " name=f\"fit_all\",\n", - " unique_tag=dataset_name,\n", - " total_draws=1,\n", - ")\n", - "\n", - "result = search.fit(model=model, analysis=analysis)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Masking__" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset = dataset.apply_mask(mask=mask)\n", - "\n", - "\n", - "number_of_ellipses = 10\n", - "\n", - "major_axis_list = np.linspace(0.3, mask_radius * 0.9, number_of_ellipses)\n", - "\n", - "total_ellipses = len(major_axis_list)\n", - "\n", - "result_list = []\n", - "\n", - "for i in range(len(major_axis_list)):\n", - " ellipse = af.Model(ag.Ellipse)\n", - "\n", - " ellipse.centre.centre_0 = result.instance.ellipses[0].centre[0]\n", - " ellipse.centre.centre_1 = result.instance.ellipses[0].centre[1]\n", - "\n", - " ellipse.ell_comps.ell_comps_0 = af.UniformPrior(lower_limit=-0.6, upper_limit=0.6)\n", - " ellipse.ell_comps.ell_comps_1 = af.UniformPrior(lower_limit=-0.6, upper_limit=0.6)\n", - "\n", - " ellipse.major_axis = major_axis_list[i]\n", - "\n", - " model = af.Collection(ellipses=[ellipse])\n", - "\n", - " search = af.DynestyStatic(\n", - " path_prefix=Path(\"ellipse_mask\"),\n", - " name=f\"fit_{i}\",\n", - " unique_tag=dataset_name,\n", - " sample=\"rwalk\",\n", - " n_live=50,\n", - " number_of_cores=4,\n", - " iterations_per_quick_update=10000,\n", - " )\n", - "\n", - " analysis = ag.AnalysisEllipse(dataset=dataset, use_jax=False)\n", - "\n", - " result = search.fit(model=model, analysis=analysis)\n", - "\n", - " result_list.append(result)\n", - "\n", - "ellipses = [result.instance.ellipses[0] for result in result_list]\n", - "\n", - "model = af.Collection(ellipses=ellipses)\n", - "\n", - "model.dummy_0 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1)\n", - "\n", - "search = af.Drawer(\n", - " path_prefix=Path(\"ellipse_mask\"),\n", - " name=f\"fit_all\",\n", - " unique_tag=dataset_name,\n", - " total_draws=1,\n", - ")\n", - "\n", - "result = search.fit(model=model, analysis=analysis)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This script gives a concise overview of the ellipse fitting modeling API, fitting one the simplest models possible.\n", - "So, what next? \n", - "\n", - "__Data Preparation__\n", - "\n", - "If you are looking to fit your own CCD imaging data of a galaxy, checkout \n", - "the `autogalaxy_workspace/*/data_preparation/imaging/start_here.ipynb` script for an overview of how data should be \n", - "prepared before being modeled.\n", - "\n", - "__HowToGalaxy__\n", - "\n", - "This example script above explains ellipse fitting, but there are many other ways to model a galaxy, using\n", - "light profiles which represent its surface brightness. \n", - "\n", - "This is explained in the **HowToGalaxy** Jupyter notebook lectures, found at `autogalaxy_workspace/*/howtogalaxy`. \n", - "\n", - "I recommend that you check them out if you are interested in more details!" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Modeling\n", + "========\n", + "\n", + "This guide shows how to perform ellipse fitting modeling on data using a non-linear search, including visualizing and\n", + "interpreting its results.\n", + "\n", + "__Fit__\n", + "\n", + "The non-linear search in this example calls a `log_likelihood_function` using the `Analysis` class many times, in\n", + "order to determine ellipse parameters and therefore overall distribution of ellipses that best-fit the data.\n", + "\n", + "The `log_likelihood_function` and how the ellipses are used to fit the data are described in the `fit.py` script,\n", + "which you should read first in order to better understand how ellipse fitting works.\n", + "\n", + "__Units__\n", + "\n", + "In this example, all quantities are **PyAutoGalaxy**'s internal unit coordinates, with spatial coordinates in\n", + "arc seconds, luminosities in electrons per second and mass quantities (e.g. convergence) are dimensionless.\n", + "\n", + "The guide `guides/units_and_cosmology.ipynb` illustrates how to convert these quantities to physical units like\n", + "kiloparsecs, magnitudes and solar masses.\n", + "\n", + "__Data Structures__\n", + "\n", + "Quantities inspected in this example script use **PyAutoGalaxy** bespoke data structures for storing arrays, grids,\n", + "vectors and other 1D and 2D quantities. These use the `slim` and `native` API to toggle between representing the\n", + "data in 1D numpy arrays or high dimension numpy arrays.\n", + "\n", + "This tutorial will only use the `slim` properties which show results in 1D numpy arrays of\n", + "shape [total_unmasked_pixels]. This is a slimmed-down representation of the data in 1D that contains only the\n", + "unmasked data points\n", + "\n", + "These are documented fully in the `autogalaxy_workspace/*/guides/data_structures.ipynb` guide.\n", + "\n", + "__Contents__\n", + "\n", + "**Loading Data:** Loading the imaging dataset from FITS files for ellipse fitting.\n", + "**Dataset Auto-Simulation:** Automatically simulating data if it does not exist.\n", + "**Mask:** Applying a circular mask to the dataset.\n", + "**Model Composition:** Composing an ellipse model with free centre and elliptical components.\n", + "**Search:** Configuring the Dynesty non-linear search for ellipse fitting.\n", + "**Analysis:** Setting up the AnalysisEllipse object for the model fit.\n", + "**Run Times:** Estimating the computational cost of the ellipse model fit.\n", + "**Model-Fit:** Running the non-linear search to fit the ellipse to data.\n", + "**Output Folder:** Description of the results written to hard-disk during and after the fit.\n", + "**Result:** Inspecting the result object and maximum likelihood ellipse parameters.\n", + "**Multiple Ellipses:** Fitting many ellipses of increasing size to trace the full galaxy.\n", + "**Final Fit:** Combining all ellipses into a single final fit.\n", + "**Masking:** Applying an extra galaxies mask and repeating the ellipse fitting.\n", + "**Data Preparation:** Pointers to data preparation scripts for your own data.\n", + "**HowToGalaxy:** Pointers to the HowToGalaxy lecture series for light profile modeling." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "\n", + "from autoconf import setup_notebook; setup_notebook()\n", + "\n", + "import numpy as np\n", + "from pathlib import Path\n", + "import autofit as af\n", + "import autofit.plot as aplt_af\n", + "import autogalaxy as ag\n", + "import autogalaxy.plot as aplt" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Loading Data__\n", + "\n", + "We we begin by loading the galaxy dataset `simple` from .fits files, which is the dataset we will use to demonstrate \n", + "ellipse fitting.\n", + "\n", + "This uses the `Imaging` object used in other examples.\n", + "\n", + "Ellipse fitting does not use the Point Spread Function (PSF) of the dataset, so we do not need to load it." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset_name = \"ellipse\"\n", + "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Dataset Auto-Simulation__\n", + "\n", + "If the dataset does not already exist on your system, it will be created by running the corresponding\n", + "simulator script. This ensures that all example scripts can be run without manually simulating data first." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "if not dataset_path.exists():\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.run(\n", + " [sys.executable, \"scripts/ellipse/simulator.py\"],\n", + " check=True,\n", + " )\n", + "\n", + "\n", + "dataset = ag.Imaging.from_fits(\n", + " data_path=dataset_path / \"data.fits\",\n", + " noise_map_path=dataset_path / \"noise_map.fits\",\n", + " pixel_scales=0.1,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can use the `Imaging` to plot the image and noise-map of the dataset." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.plot_array(array=dataset.data, title=\"Data\")\n", + "aplt.plot_array(array=dataset.noise_map, title=\"Noise Map\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can also plot a subplot which shows all these properties simultaneously." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.subplot_imaging_dataset(dataset=dataset)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Mask__\n", + "\n", + "We now mask the data, so that regions where there is no signal (e.g. the edges) are omitted from the fit.\n", + "\n", + "We use a `Mask2D` object, which for this example is 4.0\" circular mask.\n", + "\n", + "For ellipse fitting, the mask radius defines the region of the image that the ellipses are fitted over. We therefore\n", + "define the `mask_radius` as a variable which is used below to define the sizes of the ellipses in the model fitting." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "mask_radius = 5.0\n", + "\n", + "mask = ag.Mask2D.circular(\n", + " shape_native=dataset.shape_native,\n", + " pixel_scales=dataset.pixel_scales,\n", + " radius=mask_radius,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We now combine the imaging dataset with the mask." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset = dataset.apply_mask(mask=mask)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We now plot the image with the mask applied, where the image automatically zooms around the mask to make the galaxy\n", + "appear bigger." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.plot_array(array=dataset.data, title=\"Image Data With Mask Applied\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The mask is also used to compute a `Grid2D`, where the (y,x) arc-second coordinates are only computed in unmasked\n", + "pixels within the masks' circle.\n", + "\n", + "As shown in the previous overview example, this grid will be used to perform galaxying calculations when fitting the\n", + "data below." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.plot_grid(grid=dataset.grid, title=\"Grid2D of Masked Dataset\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Model Composition__\n", + "\n", + "The API below for composing a model uses the `Model` and `Collection` objects, which are imported from the \n", + "parent project **PyAutoFit** \n", + "\n", + "The API is fairly self explanatory and is straight forward to extend, for example adding more ellipses\n", + "to the galaxy.\n", + "\n", + "Ellipse fitting fits ellispes of increasing size to the data, one after another, with the properties of each ellipse\n", + "as a function of size being the main results of the model-fit.\n", + "\n", + "We therefore compose a model consistent of a single ellise to demonstrate this fitting process, and then towards\n", + "the end of the script we will extend the model to fit multiple ellipses.\n", + "\n", + "The model is composed of 1 ellipses as follows:\n", + "\n", + "1) The ellipse has a fixed sizes that is input manually. When multiple ellipses are fitted, this size will \n", + " incrementally grow in size in order to cover the entire galaxy.\n", + "\n", + "2) The centre and elliptical components of the ellipse are free, meaning that the model has N=4 free parameters.\n", + "\n", + "The model composition below uses a list even though there is one ellipse, as this format allows us to fit\n", + "multiple ellipses in the model-fit at once, albeit its rare we would want to do this.\n", + "\n", + "__Model Cookbook__\n", + "\n", + "A full description of model composition is provided by the model cookbook: \n", + "\n", + "https://pyautogalaxy.readthedocs.io/en/latest/general/model_cookbook.html\n", + "\n", + "__Coordinates__\n", + "\n", + "The model fitting default settings assume that the galaxy centre is near the coordinates (0.0\", 0.0\"). \n", + "\n", + "If for your dataset the galaxy is not centred at (0.0\", 0.0\"), we recommend that you either: \n", + "\n", + " - Reduce your data so that the centre is (`autogalaxy_workspace/*/preprocess`). \n", + " - Manually override the model priors (`autogalaxy_workspace/*/modeling/imaging/customize/priors.py`)." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "ellipse = af.Model(ag.Ellipse)\n", + "\n", + "ellipse.centre.centre_0 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1)\n", + "ellipse.centre.centre_1 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1)\n", + "\n", + "ellipse.ell_comps.ell_comps_0 = af.UniformPrior(lower_limit=-0.6, upper_limit=0.6)\n", + "ellipse.ell_comps.ell_comps_1 = af.UniformPrior(lower_limit=-0.6, upper_limit=0.6)\n", + "\n", + "ellipse.major_axis = 0.3\n", + "\n", + "model = af.Collection(ellipses=[ellipse])" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The `info` attribute shows the model in a readable format.\n", + "\n", + "[The `info` below may not display optimally on your computer screen, for example the whitespace between parameter\n", + "names on the left and parameter priors on the right may lead them to appear across multiple lines. This is a\n", + "common issue in Jupyter notebooks.\n", + "\n", + "The`info_whitespace_length` parameter in the file `config/generag.yaml` in the [output] section can be changed to \n", + "increase or decrease the amount of whitespace (The Jupyter notebook kernel will need to be reset for this change to \n", + "appear in a notebook).]" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(model.info)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Search__\n", + "\n", + "The model is fitted to the data using a non-linear search. \n", + "\n", + "This example uses the nested sampling algorithm Dynesty (https://dynesty.readthedocs.io/en/stable/), which extensive \n", + "testing has revealed gives the most accurate and efficient modeling results for ellipse fitting.\n", + "\n", + "Dynesty has one main setting that trades-off accuracy and computational run-time, the number of `live_points`. \n", + "A higher number of live points gives a more accurate result, but increases the run-time. A lower value may give \n", + "less reliable modeling (e.g. the fit may infer a local maxima), but is faster. \n", + "\n", + "The suitable value depends on the model complexity whereby models with more parameters require more live points. \n", + "The default value of 200 is sufficient for the vast majority of ellipse fitting problems. Lower values often given \n", + "reliable results though, and speed up the run-times. \n", + "\n", + "__Unique Identifier__\n", + "\n", + "In the path above, the `unique_identifier` appears as a collection of characters, where this identifier is generated \n", + "based on the model, search and dataset that are used in the fit.\n", + "\n", + "An identical combination of model and search generates the same identifier, meaning that rerunning the script will use \n", + "the existing results to resume the model-fit. In contrast, if you change the model or search, a new unique identifier \n", + "will be generated, ensuring that the model-fit results are output into a separate folder.\n", + "\n", + "We additionally want the unique identifier to be specific to the dataset fitted, so that if we fit different datasets\n", + "with the same model and search results are output to a different folder. We achieve this below by passing \n", + "the `dataset_name` to the search's `unique_tag`.\n", + "\n", + "__Parallel Script__\n", + "\n", + "Depending on the operating system (e.g. Linux, Mac, Windows), Python version, if you are running a Jupyter notebook \n", + "and other factors, this script may not run a successful parallel fit (e.g. running the script \n", + "with `number_of_cores` > 1 will produce an error). It is also common for Jupyter notebooks to not run in parallel \n", + "correctly, requiring a Python script to be run, often from a command line terminal.\n", + "\n", + "To fix these issues, the Python script needs to be adapted to use an `if __name__ == \"__main__\":` API, as this allows\n", + "the Python `multiprocessing` module to allocate threads and jobs correctly. An adaptation of this example script \n", + "is provided at `autolens_workspace/scripts/modeling/imaging/customize/parallel.py`, which will hopefully run \n", + "successfully in parallel on your computer!\n", + "\n", + "Therefore if paralellization for this script doesn't work, check out the `parallel.py` example. You will need to update\n", + "all scripts you run to use the this format and API. \n", + "\n", + "__Iterations Per Update__\n", + "\n", + "Every N iterations, the non-linear search outputs the current results to the folder `autogalaxy_workspace/output`,\n", + "which includes producing visualization. \n", + "\n", + "Depending on how long it takes for the model to be fitted to the data (see discussion about run times below), \n", + "this can take up a large fraction of the run-time of the non-linear search.\n", + "\n", + "For this fit, the fit is very fast, thus we set a high value of `iterations_per_quick_update=10000` to ensure these updates\n", + "so not slow down the overall speed of the model-fit. \n", + "\n", + "**If the iteration per update is too low, the model-fit may be significantly slowed down by the time it takes to\n", + "output results and visualization frequently to hard-disk. If your fit is consistent displaying a log saying that it\n", + "is outputting results, try increasing this value to ensure the model-fit runs efficiently.**" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "search = af.DynestyStatic(\n", + " path_prefix=Path(\"ellipse\"),\n", + " name=f\"fit_start\",\n", + " unique_tag=dataset_name,\n", + " sample=\"rwalk\",\n", + " n_live=50,\n", + " iterations_per_quick_update=10000,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Analysis__\n", + "\n", + "We next create an `AnalysisEllipse` object, which can be given many inputs customizing how the model is fitted to the \n", + "data (in this example they are omitted for simplicity).\n", + "\n", + "Internally, this object defines the `log_likelihood_function` used by the non-linear search to fit the model to \n", + "the `Imaging` dataset. \n", + "\n", + "It is not vital that you as a user understand the details of how the `log_likelihood_function` fits a model to \n", + "data, but interested readers can find a step-by-step guide of the likelihood \n", + "function at ``autogalaxy_workspace/*/imaging/log_likelihood_function`\n", + "\n", + "__JAX__\n", + "\n", + "PyAutoLens uses JAX under the hood for fast GPU/CPU acceleration. However, ellipse fitting does not support JAX\n", + "acceleration, so we disable it here by passing `use_jax=False`." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "analysis = ag.AnalysisEllipse(dataset=dataset, use_jax=False)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Run Times__\n", + "\n", + "Modeling can be a computationally expensive process. When fitting complex models to high resolution datasets \n", + "run times can be of order hours, days, weeks or even months.\n", + "\n", + "Run times are dictated by two factors:\n", + "\n", + " - The log likelihood evaluation time: the time it takes for a single `instance` of the model to be fitted to \n", + " the dataset such that a log likelihood is returned.\n", + "\n", + " - The number of iterations (e.g. log likelihood evaluations) performed by the non-linear search: more complex\n", + " models require more iterations to converge to a solution.\n", + "\n", + "For this analysis, the log likelihood evaluation time is ~0.04 seconds, which is extremely fast for modeling. For higher resolution datasets ellipse\n", + "fitting can slow down to a likelihood evaluation time of order 0.5 - 1.0 second, which is still reasonably fast.\n", + "\n", + "To estimate the expected overall run time of the model-fit we multiply the log likelihood evaluation time by an \n", + "estimate of the number of iterations the non-linear search will perform. \n", + "\n", + "Estimating this quantity is more tricky, as it varies depending on the model complexity (e.g. number of parameters)\n", + "and the properties of the dataset and model being fitted.\n", + "\n", + "For this example, we conservatively estimate that the non-linear search will perform ~10000 iterations per free \n", + "parameter in the model. This is an upper limit, with models typically converging in far fewer iterations.\n", + "\n", + "If you perform the fit over multiple CPUs, you can divide the run time by the number of cores to get an estimate of\n", + "the time it will take to fit the model. Parallelization with Nautilus scales well, it speeds up the model-fit by the \n", + "`number_of_cores` for N < 8 CPUs and roughly `0.5*number_of_cores` for N > 8 CPUs. This scaling continues \n", + "for N> 50 CPUs, meaning that with super computing facilities you can always achieve fast run times!\n", + "\n", + "__Model-Fit__\n", + "\n", + "We can now begin the model-fit by passing the model and analysis object to the search, which performs a non-linear\n", + "search to find which models fit the data with the highest likelihood.\n", + "\n", + "Checkout the output folder for live outputs of the results of the fit, including on-the-fly visualization of the best \n", + "fit model!\n", + "\n", + "**Run Time Error:** On certain operating systems (e.g. Windows, Linux) and Python versions, the code below may produce \n", + "an error. If this occurs, see the `autolens_workspace/guides/modeling/bug_fix` example for a fix." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "result = search.fit(model=model, analysis=analysis)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Output Folder__\n", + "\n", + "Now this is running you should checkout the `autogalaxy_workspace/output` folder. This is where the results of the \n", + "search are written to hard-disk (in the `start_here` folder), where all outputs are human readable (e.g. as .json,\n", + ".csv or text files).\n", + "\n", + "As the fit progresses, results are written to the `output` folder on the fly using the highest likelihood model found\n", + "by the non-linear search so far. This means you can inspect the results of the model-fit as it runs, without having to\n", + "wait for the non-linear search to terminate.\n", + "\n", + "The `output` folder includes:\n", + "\n", + " - `model.info`: Summarizes the model, its parameters and their priors discussed in the next tutorial.\n", + "\n", + " - `model.results`: Summarizes the highest likelihood model inferred so far including errors.\n", + "\n", + " - `images`: Visualization of the highest likelihood model-fit to the dataset, (e.g. a fit subplot showing the \n", + " galaxies, model data and residuals).\n", + "\n", + " - `files`: A folder containing .fits files of the dataset, the model as a human-readable .json file, \n", + " a `.csv` table of every non-linear search sample and other files containing information about the model-fit.\n", + "\n", + " - search.summary: A file providing summary statistics on the performance of the non-linear search.\n", + "\n", + " - `search_internal`: Internal files of the non-linear search (in this case Nautilus) used for resuming the fit and\n", + " visualizing the search.\n", + "\n", + "__Result__\n", + "\n", + "The search returns a result object, which whose `info` attribute shows the result in a readable format.\n", + "\n", + "[Above, we discussed that the `info_whitespace_length` parameter in the config files could b changed to make \n", + "the `model.info` attribute display optimally on your computer. This attribute also controls the whitespace of the\n", + "`result.info` attribute.]" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(result.info)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The `Result` object also contains:\n", + "\n", + " - The model corresponding to the maximum log likelihood solution in parameter space.\n", + " - The corresponding maximum log likelihood `Ellipse` and `FitEllipse` objects." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "instance = result.max_log_likelihood_instance\n", + "\n", + "print(\"Max Log Likelihood Model:\")\n", + "print(instance)\n", + "\n", + "print(f\"First Ellipse Centre: {instance.ellipses[0].centre}\")\n", + "print(f\"First Ellipse Elliptical Components: {instance.ellipses[0].ell_comps}\")\n", + "print(f\"First Ellipse Major Axis: {instance.ellipses[0].major_axis}\")\n", + "print(f\"First Ellipse Axis Ratio: {instance.ellipses[0].axis_ratio}\")\n", + "print(f\"First Ellipse Angle: {instance.ellipses[0].angle}\")\n", + "\n", + "for i, ellipse in enumerate(result.max_log_likelihood_instance.ellipses):\n", + " print(f\"Ellipse {i} Minor Axis: {ellipse.minor_axis}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The maximum log likelihood fit is also available via the result, which can visualize the fit." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.plot_array(array=dataset.data, title=\"Data\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The result contains the full posterior information of our non-linear search, including all parameter samples, \n", + "log likelihood values and tools to compute the errors on the model. \n", + "\n", + "There are built in visualization tools for plotting this.\n", + "\n", + "The plot is labeled with short hand parameter names (e.g. `sersic_index` is mapped to the short hand \n", + "parameter `n`). These mappings ate specified in the `config/notation.yaml` file and can be customized by users.\n", + "\n", + "The superscripts of labels correspond to the name each component was given in the model (e.g. for the `Isothermal`\n", + "mass its name `mass` defined when making the `Model` above is used)." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt_af.corner_cornerpy(samples=result.samples)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Multiple Ellipses__" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "number_of_ellipses = 10\n", + "\n", + "major_axis_list = np.linspace(0.3, mask_radius * 0.9, number_of_ellipses)\n", + "\n", + "total_ellipses = len(major_axis_list)\n", + "\n", + "result_list = []\n", + "\n", + "for i in range(len(major_axis_list)):\n", + " ellipse = af.Model(ag.Ellipse)\n", + "\n", + " ellipse.centre.centre_0 = result.instance.ellipses[0].centre[0]\n", + " ellipse.centre.centre_1 = result.instance.ellipses[0].centre[1]\n", + "\n", + " ellipse.ell_comps.ell_comps_0 = af.UniformPrior(lower_limit=-0.6, upper_limit=0.6)\n", + " ellipse.ell_comps.ell_comps_1 = af.UniformPrior(lower_limit=-0.6, upper_limit=0.6)\n", + "\n", + " ellipse.major_axis = major_axis_list[i]\n", + "\n", + " model = af.Collection(ellipses=[ellipse])\n", + "\n", + " search = af.DynestyStatic(\n", + " path_prefix=Path(\"ellipse\"),\n", + " name=f\"fit_{i}\",\n", + " unique_tag=dataset_name,\n", + " sample=\"rwalk\",\n", + " n_live=50,\n", + " number_of_cores=4,\n", + " iterations_per_quick_update=10000,\n", + " )\n", + "\n", + " analysis = ag.AnalysisEllipse(dataset=dataset, use_jax=False)\n", + "\n", + " result = search.fit(model=model, analysis=analysis)\n", + "\n", + " result_list.append(result)\n" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Final Fit__\n", + "\n", + "A final fit is performed combining all ellipses." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "ellipses = [result.instance.ellipses[0] for result in result_list]\n", + "\n", + "model = af.Collection(ellipses=ellipses)\n", + "\n", + "model.dummy_0 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1)\n", + "\n", + "search = af.Drawer(\n", + " path_prefix=Path(\"ellipse\"),\n", + " name=f\"fit_all\",\n", + " unique_tag=dataset_name,\n", + " total_draws=1,\n", + ")\n", + "\n", + "result = search.fit(model=model, analysis=analysis)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Masking__" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset = dataset.apply_mask(mask=mask)\n", + "\n", + "\n", + "number_of_ellipses = 10\n", + "\n", + "major_axis_list = np.linspace(0.3, mask_radius * 0.9, number_of_ellipses)\n", + "\n", + "total_ellipses = len(major_axis_list)\n", + "\n", + "result_list = []\n", + "\n", + "for i in range(len(major_axis_list)):\n", + " ellipse = af.Model(ag.Ellipse)\n", + "\n", + " ellipse.centre.centre_0 = result.instance.ellipses[0].centre[0]\n", + " ellipse.centre.centre_1 = result.instance.ellipses[0].centre[1]\n", + "\n", + " ellipse.ell_comps.ell_comps_0 = af.UniformPrior(lower_limit=-0.6, upper_limit=0.6)\n", + " ellipse.ell_comps.ell_comps_1 = af.UniformPrior(lower_limit=-0.6, upper_limit=0.6)\n", + "\n", + " ellipse.major_axis = major_axis_list[i]\n", + "\n", + " model = af.Collection(ellipses=[ellipse])\n", + "\n", + " search = af.DynestyStatic(\n", + " path_prefix=Path(\"ellipse_mask\"),\n", + " name=f\"fit_{i}\",\n", + " unique_tag=dataset_name,\n", + " sample=\"rwalk\",\n", + " n_live=50,\n", + " number_of_cores=4,\n", + " iterations_per_quick_update=10000,\n", + " )\n", + "\n", + " analysis = ag.AnalysisEllipse(dataset=dataset, use_jax=False)\n", + "\n", + " result = search.fit(model=model, analysis=analysis)\n", + "\n", + " result_list.append(result)\n", + "\n", + "ellipses = [result.instance.ellipses[0] for result in result_list]\n", + "\n", + "model = af.Collection(ellipses=ellipses)\n", + "\n", + "model.dummy_0 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1)\n", + "\n", + "search = af.Drawer(\n", + " path_prefix=Path(\"ellipse_mask\"),\n", + " name=f\"fit_all\",\n", + " unique_tag=dataset_name,\n", + " total_draws=1,\n", + ")\n", + "\n", + "result = search.fit(model=model, analysis=analysis)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This script gives a concise overview of the ellipse fitting modeling API, fitting one the simplest models possible.\n", + "So, what next? \n", + "\n", + "__Data Preparation__\n", + "\n", + "If you are looking to fit your own CCD imaging data of a galaxy, checkout \n", + "the `autogalaxy_workspace/*/data_preparation/imaging/start_here.ipynb` script for an overview of how data should be \n", + "prepared before being modeled.\n", + "\n", + "__HowToGalaxy__\n", + "\n", + "This example script above explains ellipse fitting, but there are many other ways to model a galaxy, using\n", + "light profiles which represent its surface brightness. \n", + "\n", + "This is explained in the **HowToGalaxy** Jupyter notebook lectures, found at https://github.com/PyAutoLabs/HowToGalaxy. \n", + "\n", + "I recommend that you check them out if you are interested in more details!" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [], + "outputs": [], + "execution_count": null + } + ], + "metadata": { + "anaconda-cloud": {}, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.1" + } + }, + "nbformat": 4, + "nbformat_minor": 4 } \ No newline at end of file diff --git a/notebooks/guides/advanced/over_sampling.ipynb b/notebooks/guides/advanced/over_sampling.ipynb index ed3a9729..b8521951 100644 --- a/notebooks/guides/advanced/over_sampling.ipynb +++ b/notebooks/guides/advanced/over_sampling.ipynb @@ -563,7 +563,7 @@ " import sys\n", "\n", " subprocess.run(\n", - " [sys.executable, \"scripts/howtogalaxy/simulators/sersic.py\"],\n", + " [sys.executable, \"scripts/imaging/simulator_sersic.py\"],\n", " check=True,\n", " )\n", "\n", diff --git a/notebooks/guides/modeling/customize.ipynb b/notebooks/guides/modeling/customize.ipynb index 628a53c6..4b8225e9 100644 --- a/notebooks/guides/modeling/customize.ipynb +++ b/notebooks/guides/modeling/customize.ipynb @@ -76,7 +76,7 @@ " import sys\n", "\n", " subprocess.run(\n", - " [sys.executable, \"scripts/howtogalaxy/simulators/sersic.py\"],\n", + " [sys.executable, \"scripts/imaging/simulator_sersic.py\"],\n", " check=True,\n", " )\n", "\n", diff --git a/notebooks/guides/plot/examples/searches.ipynb b/notebooks/guides/plot/examples/searches.ipynb index ce761b15..f4c30baf 100644 --- a/notebooks/guides/plot/examples/searches.ipynb +++ b/notebooks/guides/plot/examples/searches.ipynb @@ -68,7 +68,7 @@ " import sys\n", "\n", " subprocess.run(\n", - " [sys.executable, \"scripts/howtogalaxy/simulators/sersic.py\"],\n", + " [sys.executable, \"scripts/imaging/simulator_sersic.py\"],\n", " check=True,\n", " )\n", "\n", diff --git a/notebooks/howtogalaxy/README.rst b/notebooks/howtogalaxy/README.rst deleted file mode 100644 index dad49ba8..00000000 --- a/notebooks/howtogalaxy/README.rst +++ /dev/null @@ -1,125 +0,0 @@ -The ``HowToGalaxy`` folder contains **HowToGalaxy** lectures, which teach a new user how to model a galaxy. - -Folders -------- - -- ``chapter_1_introduction``: An introduction to galaxy morphology and structure using **PyAutoGalaxy**. -- ``chapter_2_modeling``: How to model galaxies, including a primer on Bayesian non-linear analysis. -- ``chapter_3_search_chaining``: How to fit complex models using non-linear search chaining. -- ``chapter_4_pixelizations``: How to perform pixelized reconstructions of a galaxy. -- ``chapter_optional``: Optional tutorials. - -Full Explanation ----------------- - -Welcome to **HowToGalaxy** - The **PyAutoGalaxy** tutorial! - -HOW TO TACKLE HowToGalaxy ------------------------ - -The **HowToGalaxy** lecture series current sits at 4 chapters, and each will take a day or so to go through -properly. You probably want to be modeling galaxies faster than that! Furthermore, the concepts -in the later chapters are pretty challenging, and familiarity and modeling is desirable before -you tackle them. - -Therefore, we recommend that you complete chapters 1 & 2 and then apply what you've learnt to the modeling of simulated -and real galaxy data, using the scripts found in the 'autogalaxy_workspace'. Once you're happy -with the results and confident with your use of **PyAutoGalaxy**, you can then begin to cover the advanced functionality -covered in chapters 3 & 4. - -JUYPTER NOTEBOOKS ------------------ - -All tutorials are supplied as Jupyter Notebooks, which come with a '.ipynb' suffix. For those new to Python, Jupyter -Notebooks are a different way to write, view and use Python code. Compared to the traditional Python scripts, -they allow: - -- Small blocks of code to be viewed and run at a time. -- Images and visualization from a code to be displayed directly underneath it. -- Text script to appear between the blocks of code. - -This makes them an ideal way for us to present the **HowToGalaxy** lecture series, therefore I recommend you get yourself -a Jupyter notebook viewer (https://jupyter.org/) if you have not done so already. - -If you *really* want to use Python scripts, all tutorials are supplied a ```` python files in the 'scripts' folder of -each chapter. - -For actual **PyAutoGalaxy** use, I recommend you use Python scripts. Therefore, as you go through the lecture series -you will notice that we will transition you to Python scripts in the third chapter. - -VISUALIZATION -------------- - -Before beginning the **HowToGalaxy** lecture series, in chapter 1 you should do 'tutorial_0_visualization'. This will -take you through how **PyAutoGalaxy** interfaces with matplotlib to perform visualization and will get you setup such that -images and figures display correctly in your Jupyter notebooks. - -CODE STYLE AND FORMATTING -------------------------- - -When you begin the notebooks, you may notice the style and formatting of our Python code looks different to what you -are used to. For example, it is common for brackets to be placed on their own line at the end of function calls, -the inputs of a function or class may be listed over many separate lines and the code in general takes up a lot more -space then you are used to. - -This is intentional, because we believe it makes the cleanest, most readable code possible. In fact - lots of people do, -which is why we use an auto-formatter to produce the code in a standardized format. If you're interested in the style -and would like to adapt it to your own code, check out the Python auto-code formatter 'black'. - -https://github.com/python/black - -OVERVIEW OF CHAPTER 1 (Beginner) --------------------------------- - -**Galaxy Structure with PyAutoGalaxy** - -In chapter 1, we'll learn about galaxy structure and **PyAutoGalaxy**. At the end, you'll be able to: - -1) Create uniform grid's of (x,y) Cartesian coordinates. -2) Combine these grid's with light profiles to make images. -3) Combine these light profiles to make galaxies. -4) Simulate telescope CCD imaging data of a galaxy. -5) Fit imaging data with model images generated via galaxy objects. - -OVERVIEW OF CHAPTER 2 (Beginner) --------------------------------- - -**Bayesian Inference and Non-linear Searches** - -In chapter 2, we'll cover Bayesian inference and model-fitting via a non-linear search. We will use these tools to -fit CCD imaging data of a galaxy with a model. At the end, you'll understand: - -1) The concept of a non-linear search and non-linear parameter space. -2) How to fit a model to galaxy CCD imaging via a non-linear search. -3) The trade-off between realism and complexity when choosing a model. -4) Why an incorrect model may be inferred and how to prevent this from happening. -5) The challenges that are involved in inferred a robust model in a computationally reasonable run-time. - -**Once completed, you'll be ready to model your own galaxies with PyAutoGalaxy!** - -OVERVIEW OF CHAPTER 3 (Intermediate) ------------------------------------- - -**Automated Modeling with non-linear search chaining** - -In chapter 3, we'll learn how to chain multiple non-linear searches together to build automated modeling pipelines -which can: - -1) Break-down the fitting of a model using multiple non-linear searches and prior passing. -2) Use a custom pipeline to fit data containing multiple galaxy where each galaxy is fitted one at a time. -3) Fit the global structure of a galaxy, followed by faint morphological features like a bar. - -OVERVIEW OF CHAPTER 4 (Intermediate) ------------------------------------- - -**Using an inverison to perform a pixelized morphology reconstructions** - -In chapter 4, we'll learn how to reconstruct morphology features of a galaxy using a pixel-grid, ensuring that we can -fit an accurate model to sources with complex and irregular morphologies. You'll learn how to: - -1) Pixelize a galaxy reconstruction into pixels. -2) Perform a linear inversion using this pixelization to reconstruct the galaxy's light. -3) Apply a smoothness prior on the galaxy reconstruction, called regularization. -4) Apply smoothing within a Bayesian framework to objectively quantify the reconstruction's complexity. -5) Use alternative pixelizations, for example a Voronoi mesh. -6) Use these features to fit a model via non-linear searches. \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_1_introduction/README.rst b/notebooks/howtogalaxy/chapter_1_introduction/README.rst deleted file mode 100644 index 2be28012..00000000 --- a/notebooks/howtogalaxy/chapter_1_introduction/README.rst +++ /dev/null @@ -1,24 +0,0 @@ -In chapter 1, we introduce you to strong gravitational lensing and the core **PyAutoGalaxy** API. - -**Colab** links to every tutorial are included. - -Files ------ - -`Tutorial 0: Visualization `_ -- Setting up **PyAutoGalaxy**'s visualization library. - -`Tutorial 1: Grids And Galaxies `_ -- How grids of (y,x) coordinates are used to create images of galaxies that ultimately quantify their morphology. - -`Tutorial 2: Data `_ -- Simulating and inspecting telescope imaging data of a galaxy, for example from the Hubble Space Telescope. - -`Tutorial 3: Fitting `_ -- How to fit imaging data of a galaxy and quantify whether a fit is good or bad. - -`Tutorial 4: Methods `_ -- An overview of the different methods used to fit galaxies with. - -`Tutorial 5: Summary `_ -- A summary of the chapter. \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_1_introduction/tutorial_0_visualization.ipynb b/notebooks/howtogalaxy/chapter_1_introduction/tutorial_0_visualization.ipynb deleted file mode 100644 index 9ef9e327..00000000 --- a/notebooks/howtogalaxy/chapter_1_introduction/tutorial_0_visualization.ipynb +++ /dev/null @@ -1,285 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tutorial 0: Visualization\n", - "=========================\n", - "\n", - "In this tutorial, we quickly cover visualization in **PyAutoGalaxy** and make sure images display clearly in your\n", - "Jupyter notebook and on your computer screen.\n", - "\n", - "__Contents__\n", - "\n", - "**Directories:** Set the working directory so PyAutoGalaxy can find configs, data and output folders.\n", - "**Dataset:** Load an example imaging dataset of a galaxy.\n", - "**Plot Customization:** Customize matplotlib options like title, figure size and colormap.\n", - "**Subplots:** Plot all components of a dataset simultaneously using subplots.\n", - "**Visuals:** Add visual overlays like masks and grids to figures.\n", - "**Wrap Up:** Summary of visualization in PyAutoGalaxy." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autoconf import setup_notebook; setup_notebook()" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "If the printed working directory does not match the workspace path on your computer, you can manually set it\n", - "as follows (the example below shows the path I would use on my laptop. The code is commented out so you do not\n", - "use this path in this tutorial!" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "# workspace_path = \"/Users/Jammy/Code/PyAuto/autogalaxy_workspace\"\n", - "# #%cd $workspace_path\n", - "# print(f\"Working Directory has been set to `{workspace_path}`\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset__\n", - "\n", - "The `dataset_path` specifies where the dataset is located, which is the\n", - "directory `autogalaxy_workspace/dataset/imaging/simple__sersic`.\n", - "\n", - "There are many example simulated images of galaxies in this directory that will be used throughout the\n", - "**HowToGalaxy** lectures." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_path = Path(\"dataset\", \"imaging\", \"simple__sersic\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset Auto-Simulation__\n", - "\n", - "If the dataset does not already exist on your system, it will be created by running the corresponding\n", - "simulator script. This ensures that all example scripts can be run without manually simulating data first." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "if not dataset_path.exists():\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.run(\n", - " [sys.executable, \"scripts/howtogalaxy/simulators/sersic.py\"],\n", - " check=True,\n", - " )\n" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We now load this dataset from .fits files and create an instance of an `imaging` object." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset = ag.Imaging.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " pixel_scales=0.1,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can plot the data as follows:" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(array=dataset.data, title=\"Data\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Plot Customization__\n", - "\n", - "Does the figure display correctly on your computer screen?\n", - "\n", - "If not, you can customize common matplotlib options by passing them directly to `plot_array`:\n", - "\n", - " - `title=`: Set the figure title.\n", - " - `figsize=`: Control the figure size as a `(width, height)` tuple.\n", - " - `colormap=`: Set the matplotlib colormap name (e.g. `\"jet\"`, `\"gray\"`).\n", - " - `xlabel=`, `ylabel=`: Override the default axis labels." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(\n", - " array=dataset.data,\n", - " title=\"Data\",\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Many matplotlib options can be customized, but for now we're only concerned with making sure figures display clear in\n", - "your Jupyter Notebooks. Nevertheless, a comprehensive API reference guide of all available plot arguments can\n", - "be found in the `autogalaxy_workspace/*/guides/plot` package. You should check this out once you are more familiar with\n", - "**PyAutoGalaxy**.\n", - "\n", - "Ideally, we would not specify a `figsize` every time we plot an image. Fortunately, default values can be fully\n", - "customized via the config files.\n", - "\n", - "Checkout the `mat_wrap.yaml` file in `autogalaxy_workspace/config/visualize/mat_wrap`.\n", - "\n", - "All default matplotlib values are here. There are a lot of entries, so lets focus on whats important for displaying\n", - "figures:\n", - "\n", - " - mat_wrap.yaml -> Figure -> figure: -> figsize\n", - " - mat_wrap.yaml -> YLabel -> figure: -> fontsize\n", - " - mat_wrap.yaml -> XLabel -> figure: -> fontsize\n", - " - mat_wrap.yaml -> TickParams -> figure: -> labelsize\n", - " - mat_wrap.yaml -> YTicks -> figure: -> labelsize\n", - " - mat_wrap.yaml -> XTicks -> figure: -> labelsize\n", - "\n", - "Don't worry about all the other files or options listed for now, as they'll make a lot more sense once you are familiar\n", - "with **PyAutoGalaxy**.\n", - "\n", - "If you had to change any of the above settings to get the figures to display clearly, you should update their values\n", - "in the corresponding config files above (you will need to reset your Jupyter notebook server for these changes to\n", - "take effect, so make sure you have the right values using the `figsize` argument in the cell above beforehand!).\n", - "\n", - "__Subplots__\n", - "\n", - "In addition to plotting individual figures, **PyAutoGalaxy** can also plot subplots showing all components of a\n", - "dataset simultaneously.\n", - "\n", - "Lets plot a subplot of our `Imaging` data:" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_imaging_dataset(dataset=dataset)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Visuals__\n", - "\n", - "Visuals can be added to any figure by passing them as keyword arguments directly to `plot_array`.\n", - "\n", - "For example, we can plot a mask on the image above by passing `mask=mask`.\n", - "\n", - "The `visuals` example illustrates every overlay argument, for example `mask=`, `grid=`, `positions=`, `lines=`, etc." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "mask = ag.Mask2D.circular_annular(\n", - " shape_native=dataset.shape_native,\n", - " pixel_scales=dataset.pixel_scales,\n", - " inner_radius=0.3,\n", - " outer_radius=3.0,\n", - ")\n", - "\n", - "aplt.plot_array(array=dataset.data, title=\"Data\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Wrap Up__\n", - "\n", - "Throughout lectures you'll see lots more visuals that are plotted on figures and subplots.\n", - "\n", - "Great! Hopefully, visualization in **PyAutoGalaxy** is displaying nicely for us to get on with the **HowToGalaxy**\n", - "lecture series." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_1_introduction/tutorial_1_grids_and_galaxies.ipynb b/notebooks/howtogalaxy/chapter_1_introduction/tutorial_1_grids_and_galaxies.ipynb deleted file mode 100644 index f2d2126f..00000000 --- a/notebooks/howtogalaxy/chapter_1_introduction/tutorial_1_grids_and_galaxies.ipynb +++ /dev/null @@ -1,1085 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "HowToGalaxy: Introduction\n", - "=========================\n", - "\n", - "Nearly a century ago, Edwin Hubble famously classified galaxies into three distinct groups:\n", - "ellipticals, spirals and irregulars. He produced a diagram of these galaxies, called the Hubble Tuning Fork, which\n", - "is shown below and still discussed by astronomers in the modern day:\n", - "\n", - "![HubbleTuning](https://github.com/Jammy2211/autogalaxy_workspace/blob/main/scripts/howtogalaxy/chapter_1_introduction/HubbleTuningFork.jpg)\n", - "\n", - "To make his diagram, Hubble looked at images of each galaxy in his sample, and subjectively judged by eye how\n", - "to classify it. Today, Astronomers use computer software, statistical algorithms and image processing techniques to\n", - "perform this task in a more quantifiable and objective way.\n", - "\n", - "The **HowToGalaxy** series of tutorials will teach you how to perform this analysis yourself, using the open-source\n", - "software package **PyAutoGalaxy**. By the end of the **HowToGalaxy** series, you'll be able to take an image of a\n", - "galaxy and study its morphology and structure using the same techniques that professional astronomers use today.\n", - "\n", - "Tutorial 1: Grids And Galaxies\n", - "==============================\n", - "\n", - "In this tutorial, we will introduce the fundamental concepts and quantities used to study galaxy morphology.\n", - "These concepts will enable us to create images of galaxies and analyze how their light is distributed across space.\n", - "Additionally, we will explore how adjusting various properties of galaxies can alter their appearance. For instance,\n", - "we can change the size of a galaxy, rotate it, or modify its brightness.\n", - "\n", - "To create these images, we first need to define 2D grids of \\((y, x)\\) coordinates. We will shift and rotate these\n", - "grids to manipulate the appearance of the galaxy in the generated images. The grid will serve as the input for light\n", - "profiles, which are analytic functions that describe the distribution of a galaxy's light. By evaluating these light\n", - "profiles on the grid, we can effectively generate images that represent the structure and characteristics of galaxies.\n", - "\n", - "__Contents__\n", - "\n", - "**Grids:** Create a uniform grid of (y,x) coordinates and show how it can be used to measure the light of a galaxy.\n", - "**Geometry:** How to shift and rotate a grid, and convert it to elliptical coordinates.\n", - "**Light Profiles:** Using light profiles, analytic functions that describe how a galaxy's light is distributed.\n", - "**Galaxies:** Creating galaxies containing light profiles and computing the image of a galaxy.\n", - "**One Dimension Projection:** Create projected 2D radial grids for 1D profile calculations.\n", - "**Unit Conversion:** Converting angular distances to physical distances using cosmology.\n", - "**Wrap Up:** Summary of the key concepts covered in this tutorial.\n", - "\n", - "The imports below are required to run the howtogalaxy tutorials in a Jupiter notebook. They also import the\n", - "`autogalaxy` package and the `autogalaxy.plot` module which are used throughout the tutorials." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autoconf import setup_notebook; setup_notebook()\n", - "\n", - "import matplotlib.pyplot as plt\n", - "import numpy as np\n", - "\n", - "import autogalaxy as ag\n", - "import autogalaxy.plot as aplt" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Grids__\n", - "\n", - "A `Grid2D` is a set of two-dimensional $(y,x)$ coordinates that represent points in space where we evaluate the \n", - "light emitted by a galaxy.\n", - "\n", - "Each coordinate on the grid is referred to as a 'pixel'. This is because we use the grid to create the image of a\n", - "galaxy at each of these coordinates, meaning that each coordinate maps to the centre of each pixel in this image.\n", - "\n", - "Grids are defined in units of 'arc-seconds' (\"). An arc-second is a unit of angular measurement used by astronomers to \n", - "describe the apparent size of objects in the sky.\n", - "\n", - "The `pixel_scales` parameter sets how many arc-seconds each pixel represents. For example, if `pixel_scales=0.1`, \n", - "then each pixel covers 0.1\" of the sky.\n", - "\n", - "We create a uniform 2D grid of 101 x 101 pixels with a pixel scale of 0.1\", corresponding to an area \n", - "of 10.1\" x 10.1\", spanning from -5.05\" to 5.05\" in both the y and x directions." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "grid = ag.Grid2D.uniform(\n", - " shape_native=(\n", - " 101,\n", - " 101,\n", - " ), # The dimensions of the grid, which here is 101 x 101 pixels.\n", - " pixel_scales=0.1, # The conversion factor between pixel units and arc-seconds.\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can visualize this grid as a uniform grid of dots, each representing a coordinate where the light is measured." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_grid(grid=grid, title=\"Uniform Grid of Coordinates\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Each coordinate in the grid corresponds to an arc-second position. Below, we print a few of these coordinates to see \n", - "the values." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(\"(y,x) pixel 0:\")\n", - "print(grid.native[0, 0]) # The coordinate of the first pixel.\n", - "print(\"(y,x) pixel 1:\")\n", - "print(grid.native[0, 1]) # The coordinate of the second pixel.\n", - "print(\"(y,x) pixel 2:\")\n", - "print(grid.native[0, 2]) # The coordinate of the third pixel.\n", - "print(\"(y,x) pixel 100:\")\n", - "print(grid.native[1, 0]) # The coordinate of the 100th pixel.\n", - "print(\"...\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Grids have two internal representations, `native` and `slim`:\n", - "\n", - "- `native`: A 2D array with shape [total_y_pixels, total_x_pixels, 2], where the 2 corresponds to the (y,x) coordinates.\n", - "- `slim`: A 1D array with shape [total_y_pixels * total_x_pixels, 2], where the coordinates are 'flattened' into a single list.\n", - "\n", - "These formats are useful for different calculations and plotting. Here, we show the same coordinate using both formats." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(\"(y,x) pixel 0 (accessed via native):\")\n", - "print(grid.native[0, 0])\n", - "print(\"(y,x) pixel 0 (accessed via slim 1D):\")\n", - "print(grid.slim[0])" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can also check the shapes of the `Grid2D` object in both `native` and `slim` formats. For this grid, \n", - "the `native` shape is (101, 101, 2) and the `slim` shape is (10201, 2)." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(grid.native.shape)\n", - "print(grid.slim.shape)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "For the HowToGalaxy tutorials, you don't need to fully understand why grids have both native and slim representations. \n", - "Just note that both are used for calculations and plotting.\n", - "\n", - "*Exercise*: Try creating grids with different `shape_native` and `pixel_scales` using the `ag.Grid2D.uniform()` function \n", - "above. Observe how the grid coordinates change when you adjust `shape_native` and `pixel_scales`.\n", - "\n", - "__Geometry__\n", - "\n", - "The above grid is centered on the origin (0.0\", 0.0\"). Sometimes, we need to shift the grid to be centered on a \n", - "specific point, like the center of a galaxy.\n", - "\n", - "We can shift the grid to a new center, (y_c, x_c), by subtracting this center from each coordinate." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "centre = (0.3, 0.5) # Shifting the grid to be centered at y=1.0\", x=2.0\".\n", - "\n", - "grid_shifted = grid\n", - "grid_shifted[:, 0] = grid_shifted[:, 0] - centre[0] # Shift in y-direction.\n", - "grid_shifted[:, 1] = grid_shifted[:, 1] - centre[1] # Shift in x-direction.\n", - "\n", - "print(\"(y,x) pixel 0 After Shift:\")\n", - "print(grid_shifted.native[0, 0]) # The coordinate of the first pixel after shifting." - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The grid is now centered around (0.3\", 0.5\"). We can plot the shifted grid to see this change.\n", - "\n", - "*Exercise*: Try shifting the grid to a different center, for example (0.0\", 0.0\") or (2.0\", 3.0\"). Observe how the\n", - "center of the grid changes when you adjust the `centre` variable." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_grid(grid=grid_shifted, title=\"Grid Centered Around (0.3, 0.5)\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Next, we can rotate the grid by an angle `phi` (in degrees). The rotation is counter-clockwise from the positive x-axis.\n", - "\n", - "To rotate the grid:\n", - "\n", - "1. Calculate the distance `radius` of each coordinate from the origin using $r = \\sqrt{y^2 + x^2}$.\n", - "2. Determine the angle `theta` counter clockwise from the positive x-axis using $\\theta = \\arctan(y / x)$.\n", - "3. Adjust `theta` by the rotation angle and convert back to Cartesian coordinates via $y = r \\sin(\\theta)$ and $x = r \\cos(\\theta)$." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "angle_degrees = 60.0\n", - "\n", - "y = grid_shifted[:, 0]\n", - "x = grid_shifted[:, 1]\n", - "\n", - "radius = np.sqrt(y**2 + x**2)\n", - "theta = np.arctan2(y, x) - np.radians(angle_degrees)\n", - "\n", - "grid_rotated = grid_shifted\n", - "grid_rotated[:, 0] = radius * np.sin(theta)\n", - "grid_rotated[:, 1] = radius * np.cos(theta)\n", - "\n", - "print(\"(y,x) pixel 0 After Rotation:\")\n", - "print(grid_rotated.native[0, 0]) # The coordinate of the first pixel after rotation." - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The grid has now been rotated 60 degrees counter-clockwise. We can plot it to see the change.\n", - "\n", - "*Exercise*: Try rotating the grid by a different angle, for example 30 degrees or 90 degrees. Observe how the grid\n", - "changes when you adjust the `angle_degrees` variable." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_grid(grid=grid_rotated, title=\"Grid Rotated 60 Degrees\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Next, we convert the rotated grid to elliptical coordinates using:\n", - "\n", - "$\\eta = \\sqrt{(x_r)^2 + (y_r)^2/q^2}$\n", - "\n", - "Where `q` is the axis-ratio of the ellipse and `(x_r, y_r)` are the rotated coordinates. \n", - "\n", - "Elliptical coordinates are a system used to describe positions in relation to an ellipse rather than a circle. They \n", - "are particularly useful in astronomy when dealing with objects like galaxies, which often have elliptical shapes \n", - "due to their inclination or intrinsic shape.\n", - "\n", - "*Exercise*: Try converting the grid to elliptical coordinates using a different axis-ratio, for example 0.3 or 0.8.\n", - "What happens to the grid when you adjust the `axis_ratio` variable?" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "axis_ratio = 0.5\n", - "eta = np.sqrt((grid_rotated[:, 0]) ** 2 + (grid_rotated[:, 1]) ** 2 / axis_ratio**2)\n", - "\n", - "print(\"First Ten Elliptical Coordinates:\")\n", - "print(eta[:10])" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Above, the angle $\\phi$ (in degrees) was used to rotate the grid, and the axis-ratio $q$ was used to convert the grid \n", - "to elliptical coordinates.\n", - "\n", - "From now on, we'll describe ellipticity using \"elliptical components\" $\\epsilon_{1}$ and $\\epsilon_{2}$, calculated \n", - "from $\\phi$ and $q$:\n", - "\n", - "$\\epsilon_{1} = \\frac{1 - q}{1 + q} \\sin(2\\phi)$ \n", - "$\\epsilon_{2} = \\frac{1 - q}{1 + q} \\cos(2\\phi)$\n", - "\n", - "We'll refer to these as `ell_comps` in the code for brevity.\n", - "\n", - "Future tutorials will explain why $\\epsilon_{1}$ and $\\epsilon_{2}$ are preferred over $q$ and $\\phi$.\n", - "\n", - "*Exercise*: Try computing the elliptical components from the axis-ratio and angle above. What happens to the elliptical\n", - "components when you adjust the `axis_ratio` and `angle_degrees` variables?" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "fac = (1 - axis_ratio) / (1 + axis_ratio)\n", - "epsilon_y = fac * np.sin(2 * np.radians(angle_degrees))\n", - "epsilon_x = fac * np.cos(2 * np.radians(angle_degrees))\n", - "\n", - "ell_comps = (epsilon_y, epsilon_x)\n", - "\n", - "print(\"Elliptical Components:\")\n", - "print(ell_comps)\n" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Light Profiles__\n", - "\n", - "Galaxies are collections of stars, gas, dust, and other astronomical objects that emit light. Astronomers study this \n", - "light to understand various properties of galaxies.\n", - "\n", - "To model the light of a galaxy, we use light profiles, which are mathematical functions that describe how a galaxy's \n", - "light is distributed across space. By applying these light profiles to 2D grids of $(y, x)$ coordinates, we can \n", - "create images that represent a galaxy's luminous emission.\n", - "\n", - "A commonly used light profile is the `Sersic` profile, which is widely adopted in astronomy for representing galaxy \n", - "light. The `Sersic` profile is defined by the equation:\n", - "\n", - "$I_{\\rm Ser} (\\eta_{\\rm l}) = I \\exp \\left\\{ -k \\left[ \\left( \\frac{\\eta}{R} \\right)^{\\frac{1}{n}} - 1 \\right] \\right\\}$\n", - "\n", - "In this equation:\n", - "\n", - " - $\\eta$ represents the elliptical coordinates of the profile in arc-seconds (refer to earlier sections for elliptical coordinates).\n", - " - $I$ is the intensity normalization of the profile, given in arbitrary units, which controls the overall brightness of the Sersic profile.\n", - " - $R$ is the effective radius in arc-seconds, which determines the size of the profile.\n", - " - $n$ is the Sersic index, which defines how 'steep' the profile is, influencing the concentration of light.\n", - " - $k$ is a constant that ensures half the light of the profile lies within the radius $R$, where $k = 2n - \\frac{1}{3}$.\n", - "\n", - "We can evaluate this function using values for $(\\eta, I, R, n)$ to calculate the intensity of the profile at \n", - "a particular elliptical coordinate." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "elliptical_coordinate = (\n", - " 0.5 # The elliptical coordinate where we compute the intensity, in arc-seconds.\n", - ")\n", - "intensity = 1.0 # Intensity normalization of the profile in arbitrary units.\n", - "effective_radius = 2.0 # Effective radius of the profile in arc-seconds.\n", - "sersic_index = 1.0 # Sersic index of the profile.\n", - "k = 2 * sersic_index - (\n", - " 1.0 / 3.0\n", - ") # Calculating the constant k, note that this is an approximation.\n", - "\n", - "# Calculate the intensity of the Sersic profile at a specific elliptical coordinate.\n", - "sersic_value = np.exp(\n", - " -k * ((elliptical_coordinate / effective_radius) ** (1.0 / sersic_index) - 1.0)\n", - ")\n", - "\n", - "print(\"Intensity of Sersic Light Profile at Elliptical Coordinate 0.5:\")\n", - "print(sersic_value)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The calculation above gives the intensity of the Sersic profile at an elliptical coordinate of 0.5.\n", - "\n", - "To create a complete image of the Sersic profile, we can evaluate the intensity at every point in our grid of \n", - "elliptical coordinates." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "sersic_image = np.exp(-k * ((eta / effective_radius) ** (1.0 / sersic_index) - 1.0))" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "When we plot the resulting image, we can see how the properties of the grid affect its appearance:\n", - "\n", - " - The peak intensity is at the position (0.3\", 0.5\"), where we shifted the grid.\n", - " - The image is elongated along a 60\u00b0 counter-clockwise angle, corresponding to the rotation of the grid.\n", - " - The image has an elliptical shape, consistent with the axis ratio of 0.5.\n", - "\n", - "This demonstrates how the geometry of the grid directly influences the appearance of the light profile.\n", - "\n", - "*Exercise*: Try changing the values of `centre`, `ell_comps`, `effective_radius`, and `sersic_index` above. \n", - "Observe how these adjustments change the Sersic profile image." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(\n", - " array=ag.Array2D(\n", - " values=sersic_image, mask=grid.mask\n", - " ), # The `Array2D` object is discussed below.\n", - " title=\"Sersic Image\",\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Instead of manually handling these transformations, we can use `LightProfile` objects from the `light_profile` \n", - "module (`lp`) for faster and more efficient calculations.\n", - "\n", - "Below, we define a `Sersic` light profile using the `Sersic` object. We can print the profile to display its parameters." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "sersic_light_profile = ag.lp.Sersic(\n", - " centre=(0.0, 0.0),\n", - " ell_comps=(0.0, 0.1),\n", - " intensity=1.0,\n", - " effective_radius=2.0,\n", - " sersic_index=1.0,\n", - ")\n", - "\n", - "print(sersic_light_profile)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "With this `Sersic` light profile, we can create an image by passing a grid to its `image_2d_from` method.\n", - "\n", - "The calculation will internally handle all the coordinate transformations and intensity evaluations we performed \n", - "manually earlier, making it much simpler.\n", - "\n", - "The `Sersic` profile we created just above is different from the one we used to manually compute the image,\n", - "so the image will look different. However, the process is the same." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "image = sersic_light_profile.image_2d_from(grid=grid)\n", - "\n", - "aplt.plot_array(array=image, title=\"Sersic Image via Light Profile\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `image` is returned as an `Array2D` object. Similar to a `Grid2D`, it has two forms:\n", - "\n", - " - `native`: A 2D array with shape [total_y_image_pixels, total_x_image_pixels].\n", - " - `slim`: A 1D array that flattens this data into shape [total_y_image_pixels * total_x_image_pixels].\n", - "\n", - "The `native` form is often used for visualizations, while the `slim` form can be useful for certain calculations." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(\"Intensity of pixel 0:\")\n", - "print(image.native[0, 0])\n", - "print(\"Intensity of pixel 1:\")\n", - "print(image.slim[1])" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can visualize the light profile's image.\n", - "\n", - "We provide it with the light profile and the grid, which are used to create and plot the image." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(\n", - " array=sersic_light_profile.image_2d_from(grid=grid), title=\"Image via LightProfile\"\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__One Dimension Projection__\n", - "\n", - "We often want to calculative 1D quantities of a light profile, for example to plot how its light changes as\n", - "a function of radius.\n", - "\n", - "To do this, we must still input a 2D grid into the `image_2d_from` method, therefore we create a project 2D \n", - "radial grid as follows which has shape [Number_of_1d_coordinates, 2] and where all [:,0] entries are the same.\n", - "\n", - "A simple example of such a grid is as follows with 4 1D coordinates is:" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "grid_2d_projected = ag.Grid2DIrregular(\n", - " [\n", - " [1.000000e-06, 1.000000e-06],\n", - " [1.000000e-06, 1.000001e00],\n", - " [1.000000e-06, 2.000001e00],\n", - " [1.000000e-06, 3.000001e00],\n", - " ]\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "As in this example, we often already have a 2D grid we are using to calculate images of a light profile\n", - "and it would be convenient to simply create `grid_2d_projected` from that.\n", - "\n", - "For example, we may want the project grid which traces it major axis in uniform radial steps.\n", - "\n", - "This is easily computed using the `grid_2d_radial_project_from` function and passing the `centre` and `angle`\n", - "of a light profile we can make it align with the light profile itself.\n", - "\n", - "Note how in this example the two galaxy bulges are not rotationally aligned but we aligned the projected\n", - "grid with the first galaxy. The centres are aligned, but if they were not that would cause similar\n", - "issues." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "grid_2d_projected = grid.grid_2d_radial_projected_from(\n", - " centre=sersic_light_profile.centre, angle=sersic_light_profile.angle()\n", - ")\n", - "\n", - "image_1d = sersic_light_profile.image_2d_from(grid=grid_2d_projected)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can now plot the 1D radial profile of the light profile. This profile shows how the intensity of the light \n", - "changes as a function of distance from the profile's center. This is a more informative way to visualize the light p\n", - "rofile's distribution.\n", - "\n", - "When we plot 1D quantities, we do not use built-in plotting functions as in 2D, but instead use standard\n", - "matplotlib functionality.\n", - "\n", - "The reason is partly that 1D plotting is simple, but also because 1D plots have many different decisions\n", - "about what is plotted and how they are computed, meaning its better to give the user full control.\n", - "\n", - "**Exercise**: Try plotting the 1D radial profile of Sersic profiles with different effective radii and Sersic indices.\n", - "Does the 1D representation show more clearly how the light distribution changes with these parameters?" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "plt.plot(grid_2d_projected[:, 1], image_1d)\n", - "plt.xlabel(\"Radius (arcseconds)\")\n", - "plt.ylabel(\"Luminosity\")\n", - "plt.show()\n", - "plt.close()" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Since galaxy light distributions often cover a wide range of values, they are typically better visualized on a log10 \n", - "scale. This approach helps highlight details in the faint outskirts of a light profile.\n", - "\n", - "The `MatPlot2D` object has a `use_log10` option that applies this transformation automatically. Below, you can see \n", - "that the image plotted in log10 space reveals more details." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(\n", - " array=sersic_light_profile.image_2d_from(grid=grid),\n", - " title=\"Sersic Image\",\n", - " use_log10=True,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Galaxies__\n", - "\n", - "Now, let's introduce `Galaxy` objects, which are key components in **PyAutoGalaxy**.\n", - "\n", - "A light profile represents a single feature of a galaxy, such as its bulge or disk. To model a complete galaxy, \n", - "we combine multiple `LightProfiles` into a `Galaxy` object. This allows us to create images that include different \n", - "components of a galaxy.\n", - "\n", - "In addition to light profiles, a `Galaxy` has a `redshift`, which indicates how far away it is from Earth. The redshift \n", - "is essential for performing unit conversions using cosmological calculations, such as converting arc-seconds into \n", - "kiloparsecs. (A kiloparsec is a distance unit in astronomy, equal to about 3.26 million light-years.)\n", - "\n", - "Let's start by creating a galaxy with two `Sersic` light profiles, which notationally we will consider to represent\n", - "a bulge and disk component of the galaxy, the two most important structures seen in galaxies which drive the\n", - "Hubble tuning fork classification shown at the beginning of this tutorial." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "bulge = ag.lp.Sersic(\n", - " centre=(0.0, 0.0),\n", - " ell_comps=(0.0, 0.111111),\n", - " intensity=1.0,\n", - " effective_radius=1.0,\n", - " sersic_index=2.5,\n", - ")\n", - "\n", - "disk = ag.lp.Sersic(\n", - " centre=(0.0, 0.0),\n", - " ell_comps=(0.0, 0.3),\n", - " intensity=0.3,\n", - " effective_radius=3.0,\n", - " sersic_index=1.0,\n", - ")\n", - "\n", - "galaxy = ag.Galaxy(redshift=0.5, bulge=bulge, disk=disk)\n", - "\n", - "print(galaxy)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can pass a 2D grid to a light profile to compute its image using the `image_2d_from` method. \n", - "\n", - "The same approach works for a `Galaxy` object:" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "image = galaxy.image_2d_from(grid=grid)\n", - "\n", - "print(\"Intensity of `Grid2D` pixel 0:\")\n", - "print(image.native[0, 0])\n", - "print(\"Intensity of `Grid2D` pixel 1:\")\n", - "print(image.native[0, 1])\n", - "print(\"Intensity of `Grid2D` pixel 2:\")\n", - "print(image.native[0, 2])\n", - "print(\"...\")\n", - "\n", - "aplt.plot_array(array=image, title=\"Bulge+Disk Image via Galaxy\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can plot the galaxy's image, just like how we did for a light profile." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(array=galaxy.image_2d_from(grid=grid), title=\"Galaxy Bulge+Disk Image\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The bulge dominates the center of the image, and is pretty much the only luminous emission we see can see on a linear\n", - "scale. The disk's emission is present, but it is much fainter and spread over a larger area.\n", - "\n", - "We can confirm this using the `subplot_of_light_profiles` method, which plots each individual light profile separately." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_galaxy_light_profiles(galaxy=galaxy, grid=grid)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Because galaxy light distributions often follow a log10 pattern, plotting in log10 space helps reveal details in the \n", - "outskirts of the light profile, in this case the emission of the disk.\n", - "\n", - "This is especially helpful to separate the bulge and disk profiles, which have different intensities and sizes." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(\n", - " array=galaxy.image_2d_from(grid=grid),\n", - " title=\"Galaxy Bulge+Disk Image\",\n", - " use_log10=True,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Using the tools above, we can visualize each light profile's contribution in 1D.\n", - "\n", - "1D plots show the intensity of the light profile as a function of distance from the profile\u2019s center. The bulge\n", - "and disk profiles in this example share the same `centre`, meaning that plotting them together on the same 1D plot \n", - "shows how they vary relative to one another. \n", - "\n", - "If the `centre` of the profiles were different, when you make the 1D plot you would need to decide if you should the\n", - "profiles offset from one another or plot them both from zero." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "grid_2d_projected = grid.grid_2d_radial_projected_from(\n", - " centre=galaxy.bulge.centre, angle=galaxy.bulge.angle()\n", - ")\n", - "bulge_image_1d = galaxy.bulge.image_2d_from(grid=grid_2d_projected)\n", - "\n", - "grid_2d_projected = grid.grid_2d_radial_projected_from(\n", - " centre=galaxy.disk.centre, angle=galaxy.disk.angle()\n", - ")\n", - "disk_image_1d = galaxy.disk.image_2d_from(grid=grid_2d_projected)\n", - "\n", - "plt.plot(grid_2d_projected[:, 1], bulge_image_1d, label=\"Bulge\")\n", - "plt.plot(grid_2d_projected[:, 1], disk_image_1d, label=\"Disk\")\n", - "plt.xlabel(\"Radius (arcseconds)\")\n", - "plt.ylabel(\"Luminosity\")\n", - "plt.legend()\n", - "plt.show()\n", - "plt.close()" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can group multiple galaxies at the same redshift into a `Galaxies` object, which is created from a list of \n", - "individual galaxies.\n", - "\n", - "We create an additional galaxy and combine it with the original galaxy into a `Galaxies` object. This could\n", - "represent two galaxies merging or interacting with each other, which is commonly seen in studies of galaxy evolution." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "extra_galaxy = ag.Galaxy(\n", - " redshift=0.5,\n", - " bulge=ag.lp.Sersic(\n", - " centre=(0.2, 0.3),\n", - " ell_comps=(0.0, 0.111111),\n", - " intensity=1.0,\n", - " effective_radius=1.0,\n", - " sersic_index=2.5,\n", - " ),\n", - ")\n", - "\n", - "galaxies = ag.Galaxies(galaxies=[galaxy, extra_galaxy])" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `Galaxies` object has similar methods as those for light profiles and individual galaxies.\n", - "\n", - "For example, `image_2d_from` sums the images of all the galaxies." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "image = galaxies.image_2d_from(grid=grid)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can plot the combined image using a `Galaxies`, just like with other plotters." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(array=galaxies.image_2d_from(grid=grid), title=\"Image\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "A subplot of each individual galaxy image can also be created." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_galaxies(galaxies=galaxies, grid=grid)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Because galaxy light distributions often follow a log10 pattern, plotting in log10 space helps reveal details in the \n", - "outskirts of the light profile.\n", - "\n", - "This is especially helpful when visualizing how multiple galaxies overlap." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(array=galaxies.image_2d_from(grid=grid), title=\"Image\", use_log10=True)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Unit Conversion__\n", - "\n", - "Earlier, we mentioned that a galaxy\u2019s `redshift` allows us to convert between arcseconds and kiloparsecs.\n", - "\n", - "A redshift measures how much a galaxy's light is stretched by the Universe's expansion. A higher redshift means the \n", - "galaxy is further away, and its light has been stretched more. By knowing a galaxy\u2019s redshift, we can convert angular \n", - "distances (like arcseconds) to physical distances (like kiloparsecs).\n", - "\n", - "To perform this conversion, we use a cosmological model that describes the Universe's expansion. Below, we use \n", - "the `Planck15` cosmology, which is based on observations from the Planck satellite." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "cosmology = ag.cosmo.Planck15()\n", - "\n", - "kpc_per_arcsec = cosmology.kpc_per_arcsec_from(redshift=galaxy.redshift)\n", - "\n", - "print(\"Kiloparsecs per Arcsecond:\")\n", - "print(kpc_per_arcsec)\n" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This `kpc_per_arcsec` can be used as a conversion factor between arcseconds and kiloparsecs when plotting images of\n", - "galaxies.\n", - "\n", - "We compute this value and plot the image in converted units of kiloparsecs.\n", - "\n", - "This passes the plotting modules `Units` object a `ticks_convert_factor` and manually specified the new units of the\n", - "plot ticks." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(array=galaxy.image_2d_from(grid=grid), title=\"Image\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Wrap Up__\n", - "\n", - "You've learnt the basic quantities used to study galaxy morphology. \n", - "\n", - "Lets summarise what we've learnt:\n", - "\n", - "- **Grids**: We've learnt that a grid is a set of 2D coordinates that represent the positions where we measure the\n", - "light of a galaxy. \n", - "\n", - "- **Geometry**: We've learnt how to shift, rotate, and convert grids to elliptical coordinates.\n", - "\n", - "- **Light Profiles**: We've learnt that light profiles are mathematical functions that describe how a galaxy's light is\n", - "distributed in space. We've used the `Sersic` profile to create images of galaxies.\n", - "\n", - "- **Galaxies**: We've learnt that galaxies are collections of light profiles that represent a galaxy's light. We've\n", - "created galaxies with multiple light profiles and visualized their images.\n", - "\n", - "- **Unit Conversion**: We've learnt that by assuming redshifts for galaxies we can convert their quantities from\n", - "arcseconds to kiloparsecs.\n", - "\n", - "__Advanced Topics__\n", - "\n", - "The following advanced topics are not important for a new user learning the software for the first time. However,\n", - "once you are an expert user, the following guides and concepts are important for doing accurate galaxy morphology\n", - "analysis, and thus may be things you want to commit to memory as future references.\n", - "\n", - "__Other Unit Conversion__\n", - "\n", - "Above, we used a redshift to convert between arcseconds and kiloparsecs. This is just one example of a unit conversion\n", - "that can be performed using a galaxy's redshift.\n", - "\n", - "There are many other unit conversions that can be performed, such as converting the units of a galaxy's image from\n", - "to what Astronomers call an AB magnitude system, which is a system used to measure the brightness of galaxies.\n", - "\n", - "The `autogalaxy_workspace/*/guides/units` module contains many examples of unit conversions and how to use them,\n", - "but they will not be covered in the *HowToGalaxy* tutorials.\n", - "\n", - "__Over Sampling__\n", - "\n", - "Over sampling is a numerical technique where the images of light profiles and galaxies are evaluated \n", - "on a higher resolution grid than the image data to ensure the calculation is accurate. \n", - "\n", - "For a new user, the details of over-sampling are not important, therefore just be aware that all calculations use an\n", - "adaptive over sampling scheme which high accuracy across all use cases.\n", - "\n", - "Once you are more experienced, you should read up on over-sampling in more detail via \n", - "the `autogalaxy_workspace/*/guides/over_sampling.ipynb` notebook." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_1_introduction/tutorial_2_data.ipynb b/notebooks/howtogalaxy/chapter_1_introduction/tutorial_2_data.ipynb deleted file mode 100644 index 81b2de21..00000000 --- a/notebooks/howtogalaxy/chapter_1_introduction/tutorial_2_data.ipynb +++ /dev/null @@ -1,630 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tutorial 2: Data\n", - "================\n", - "\n", - "In the previous tutorial, we used light profiles to create images of galaxies. However, those images don't accurately\n", - "represent what we would observe through a telescope.\n", - "\n", - "Real telescope images, like those taken with the Charge Coupled Device (CCD) imaging detectors on the Hubble Space\n", - "Telescope (HST), include several factors that affect what we see:\n", - "\n", - "**Telescope Optics:** The optical components of the telescope can blur the light, influencing the image's sharpness.\n", - "\n", - "**Exposure Time:** The time the detector collects light, affecting the clarity of the image. Longer exposure times\n", - "gather more light, improving the signal-to-noise ratio and creating a clearer image.\n", - "\n", - "**Background Sky:** Light from a background sky, such as distant stars or zodiacal light, adds noise to the image.\n", - "\n", - "In this tutorial, we'll simulate a galaxy image by applying these real-world effects to the light profiles and images\n", - "we created earlier.\n", - "\n", - "__Contents__\n", - "\n", - "**Initial Setup:** Create a 2D grid and define galaxy light profiles for simulation.\n", - "**Optics Blurring:** Simulate how the telescope optics blur the galaxy's light using PSF convolution.\n", - "**Poisson Noise:** Add Poisson noise to the image, simulating CCD photon-to-electron randomness.\n", - "**Background Sky:** Add background sky light that introduces noise across the entire image.\n", - "**Simulator:** Use the SimulatorImaging object to simulate imaging data with all effects combined.\n", - "**Output:** Save the simulated data to .fits files for use in future tutorials.\n", - "**Wrap Up:** Summary of how CCD imaging data is simulated." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autoconf import setup_notebook; setup_notebook()\n", - "\n", - "import numpy as np\n", - "from pathlib import Path\n", - "import autogalaxy as ag\n", - "import autogalaxy.plot as aplt" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Initial Setup__\n", - "\n", - "To create our simulated galaxy image, we first need a 2D grid. This grid will represent the coordinate space over \n", - "which we will simulate the galaxy's light distribution." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "grid = ag.Grid2D.uniform(\n", - " shape_native=(\n", - " 101,\n", - " 101,\n", - " ), # The dimensions of the grid, which here is 101 x 101 pixels.\n", - " pixel_scales=0.1, # The conversion factor between pixel units and arc-seconds.\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Next, we define the properties of our galaxy. In this tutorial, we\u2019ll represent the galaxy with a bulge using a \n", - "Sersic light profile.\n", - "\n", - "In the previous tutorial, the units of `intensity` were arbitrary. However, for this tutorial, where we simulate \n", - "realistic imaging data, the intensity must have specific units. We\u2019ll use units of electrons per second per pixel \n", - "($e- pix^-1 s^-1$), which is standard for CCD imaging data." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "galaxy = ag.Galaxy(\n", - " redshift=0.5,\n", - " bulge=ag.lp.Sersic(\n", - " centre=(0.0, 0.0),\n", - " ell_comps=(0.0, 0.111111),\n", - " intensity=1.0, # in units of e- pix^-1 s^-1\n", - " effective_radius=1.0,\n", - " sersic_index=2.5,\n", - " ),\n", - ")\n", - "\n", - "galaxies = ag.Galaxies(galaxies=[galaxy])" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "To visualize the galaxy\u2019s image, which we will use as the starting point for the simulations, we use the following code:" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(\n", - " array=galaxies.image_2d_from(grid=grid), title=\"Galaxy Image Before Simulating\"\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Optics Blurring__\n", - "\n", - "All images captured using CCDs (like those on the Hubble Space Telescope) experience some level of blurring \n", - "due to the optics of the telescope. This blurring occurs because the optical system spreads out the light from each \n", - "point source (e.g., a star or a part of a galaxy).\n", - "\n", - "The Point Spread Function (PSF) describes how the telescope blurs the image. It can be thought of as a 2D representation \n", - "of how a single point of light would appear in the image, spread out by the optics. In practice, the PSF is a 2D \n", - "convolution kernel that we apply to the image to simulate this blurring effect." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "psf = ag.Convolver.from_gaussian(\n", - " shape_native=(11, 11), # The size of the PSF kernel, represented as an 11x11 grid.\n", - " sigma=0.1, # Controls the width of the Gaussian PSF, which determines the level of blurring.\n", - " pixel_scales=grid.pixel_scales, # Maintains consistency with the scale of the image grid.\n", - " normalize=True, # Normalizes the PSF kernel so that its values sum to 1.\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can visualize the PSF to better understand how it will blur the galaxy's image. The PSF is essentially a small \n", - "image that represents the spreading out of light from a single point source. This kernel will be used to blur the \n", - "entire galaxy image when we perform the convolution." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(array=psf.kernel, title=\"PSF 2D Kernel\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The PSF is often more informative when plotted on a log10 scale. This approach allows us to clearly observe values \n", - "in its tail, which are much smaller than the central peak yet critical for many scientific analyses. The tail \n", - "values may significantly affect the spread and detail captured in the data." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(array=psf.kernel, title=\"PSF 2D Kernel\", use_log10=True)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Next, we'll manually perform a 2D convolution of the PSF with the image of the galaxy. This convolution simulates the \n", - "blurring that occurs when the telescope optics spread out the galaxy's light.\n", - "\n", - "1. **Padding the Image**: Before convolution, we add padding (extra space with zero values) around the edges of the \n", - " image. This prevents unwanted edge effects when we perform the convolution, ensuring that the image's edges don't \n", - " become artificially altered by the process.\n", - "\n", - "2. **Convolution**: Using the `Convolver` object's `convolve` method, we apply the 2D PSF convolution to the padded \n", - " image. This step combines the PSF with the galaxy's light, simulating how the telescope spreads out the light.\n", - "\n", - "3. **Trimming the Image**: After convolution, we trim the padded areas back to their original size, obtaining a \n", - " convolved (blurred) image that matches the dimensions of the initial galaxy image." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "image = galaxies.image_2d_from(grid=grid) # The original unblurred image of the galaxy.\n", - "padded_image = galaxies.padded_image_2d_from(\n", - " grid=grid,\n", - " psf_shape_2d=psf.kernel.shape_native, # Adding padding based on the PSF size.\n", - ")\n", - "convolved_image = psf.convolved_image_from(\n", - " image=padded_image, blurring_image=None\n", - ") # Applying the PSF convolution.\n", - "blurred_image = convolved_image.trimmed_after_convolution_from(\n", - " kernel_shape=psf.kernel.shape_native\n", - ") # Trimming back to the original size." - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can now plot the original and the blurred images side by side. This allows us to clearly see how the PSF \n", - "convolution affects the appearance of the galaxy, making the image appear softer and less sharp." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(array=image, title=\"Galaxy Image Before PSF\")\n", - "aplt.plot_array(array=blurred_image, title=\"Galaxy Image After PSF\")\n" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Poisson Noise__\n", - "\n", - "In addition to the blurring caused by telescope optics, we also need to consider Poisson noise when simulating imaging \n", - "data.\n", - "\n", - "When a telescope captures an image of a galaxy, photons from the galaxy are collected by the telescope's mirror and \n", - "directed onto a CCD (Charge-Coupled Device). The CCD is made up of a silicon lattice (or another material) that \n", - "converts incoming photons into electrons. These electrons are then gathered into discrete squares, which form the \n", - "pixels of the final image.\n", - "\n", - "The process of converting photons into electrons is inherently random, following a Poisson distribution. This randomness \n", - "means that the number of electrons in each pixel can vary, even if the same number of photons hits the CCD. Therefore, \n", - "the electron count per pixel becomes a Poisson random variable. For our simulation, this means that the recorded \n", - "number of photons in each pixel will differ slightly from the true number due to this randomness.\n", - "\n", - "To replicate this effect in our simulation, we can add Poisson noise to the galaxy image using NumPy\u2019s random module, \n", - "which generates values from a Poisson distribution.\n", - "\n", - "It's important to note that the blurring caused by the telescope optics occurs before the photons reach the CCD. \n", - "Therefore, we need to add the Poisson noise after blurring the galaxy image.\n", - "\n", - "We also need to consider the units of our image data. Let\u2019s assume that the galaxy image is measured in units of \n", - "electrons per second ($e^- s^{-1}$), which is standard for CCD imaging data. To simulate the number of electrons \n", - "actually detected in each pixel, we multiply the image by the observation\u2019s exposure time. This conversion changes t\n", - "he units to the total number of electrons collected per pixel over the entire exposure time.\n", - "\n", - "Once the image is converted, we add Poisson noise, simulating the randomness in the photon-to-electron conversion \n", - "process. After adding the noise, we convert the image back to units of electrons per second for analysis, as \n", - "this is the preferred unit for astronomers when studying their data." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "exposure_time = 300.0 # Units of seconds\n", - "blurred_image_counts = (\n", - " blurred_image * exposure_time\n", - ") # Convert to total electrons detected over the exposure time.\n", - "blurred_image_with_poisson_noise = (\n", - " np.random.poisson(blurred_image_counts, blurred_image_counts.shape) / exposure_time\n", - ") # Add Poisson noise and convert back to electrons per second." - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Here is what the blurred image with Poisson noise looks like." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(\n", - " array=ag.Array2D(values=blurred_image_with_poisson_noise, mask=grid.mask),\n", - " title=\"Image With Poisson Noise\",\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "It is challenging to see the Poisson noise directly in the image above, as it is often subtle. To make the noise more \n", - "visible, we can subtract the blurred image without Poisson noise from the one with noise.\n", - "\n", - "This subtraction yields the \"Poisson noise realization\" which highlights the variation in each pixel due to the Poisson \n", - "distribution of photons hitting the CCD. It represents the noise values that were added to each pixel. We call\n", - "it the realization because it is one possible outcome of the Poisson process, and the noise will be different each time\n", - "we simulate the image." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "poisson_noise_realization = blurred_image_with_poisson_noise - blurred_image\n", - "\n", - "aplt.plot_array(\n", - " array=ag.Array2D(values=poisson_noise_realization, mask=grid.mask),\n", - " title=\"Poisson Noise Realization\",\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Background Sky__\n", - "\n", - "The final effect we will consider when simulating imaging data is the background sky.\n", - "\n", - "In addition to light from the galaxy, the telescope also picks up light from the sky. This background sky light is \n", - "primarily due to two sources: zodiacal light, which is light scattered by interplanetary dust in the solar system, \n", - "and the unresolved emission from distant stars and galaxies.\n", - "\n", - "For our simulation, we'll assume that the background sky has a uniform brightness across the image, measured at \n", - "0.1 electrons per second per pixel. The background sky is added to the image before applying the PSF convolution \n", - "and adding Poisson noise. This is important because it means that the background contributes additional noise to the \n", - "image.\n", - "\n", - "The background sky introduces noise throughout the entire image, including areas where the galaxy is not present. \n", - "This is why CCD images often appear noisy, especially in regions far from where the galaxy signal is detected. \n", - "The sky noise can make it more challenging to observe faint details of the galaxy.\n", - "\n", - "To simulate this, we add a constant background sky to the galaxy image and then apply Poisson noise to create the \n", - "final simulated image as it would appear through a telescope." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "background_sky_level = 0.1\n", - "\n", - "# Add background sky to the blurred galaxy image.\n", - "blurred_image_with_sky = blurred_image + background_sky_level\n", - "blurred_image_with_sky_counts = blurred_image_with_sky * exposure_time\n", - "\n", - "# Apply Poisson noise to the image with the background sky.\n", - "blurred_image_with_sky_poisson_noise = (\n", - " np.random.poisson(\n", - " blurred_image_with_sky_counts, blurred_image_with_sky_counts.shape\n", - " )\n", - " / exposure_time\n", - ")\n", - "\n", - "# Visualize the image with background sky and Poisson noise.\n", - "aplt.plot_array(\n", - " array=ag.Array2D(values=blurred_image_with_sky_poisson_noise, mask=grid.mask),\n", - " title=\"Image With Background Sky\",\n", - ")\n", - "\n", - "# Create a noise map showing the differences between the blurred image with and without noise.\n", - "poisson_noise_realization = (\n", - " blurred_image_with_sky_poisson_noise - blurred_image_with_sky\n", - ")\n", - "\n", - "aplt.plot_array(\n", - " array=ag.Array2D(values=poisson_noise_realization, mask=grid.mask),\n", - " title=\"Poisson Noise Realization\",\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Simulator__\n", - "\n", - "The `SimulatorImaging` object lets us create simulated imaging data while including the effects of PSF blurring, \n", - "Poisson noise, and background sky all at once:" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "simulator = ag.SimulatorImaging(\n", - " exposure_time=300.0,\n", - " psf=psf,\n", - " background_sky_level=0.1,\n", - " add_poisson_noise_to_data=True,\n", - ")\n", - "\n", - "dataset = simulator.via_galaxies_from(galaxies=galaxies, grid=grid)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "By plotting the `data` from the dataset, we can see that it matches the image we simulated earlier. It includes \n", - "the effects of PSF blurring, Poisson noise, and noise from the background sky. This image is a realistic \n", - "approximation of what a telescope like the Hubble Space Telescope would capture." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(array=dataset.data, title=\"Simulated Imaging Data\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The dataset also includes the `psf` (Point Spread Function) used to blur the galaxy image.\n", - "\n", - "For actual telescope data, the PSF is determined during data processing and is provided along with the observations. \n", - "It's crucial for accurately deconvolving the PSF from the galaxy image, allowing us to recover the true properties \n", - "of the galaxy. We'll explore this further in the next tutorial." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(array=dataset.psf.kernel, title=\"Simulated PSF\", use_log10=True)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The dataset includes a `noise_map`, which represents the Root Mean Square (RMS) standard deviation of the noise \n", - "estimated for each pixel in the image. Higher noise values mean that the measurements in those pixels are \n", - "less certain, so those pixels are given less weight when analyzing the data.\n", - "\n", - "This `noise_map` is different from the Poisson noise arrays we plotted earlier. The Poisson noise arrays show the \n", - "actual noise added to the image due to the random nature of photon-to-electron conversion on the CCD, as calculated \n", - "using the numpy random module. These noise values are theoretical and cannot be directly measured in real telescope data.\n", - "\n", - "In contrast, the `noise_map` is our best estimate of the noise present in the image, derived from the data itself \n", - "and used in the fitting process." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(array=dataset.noise_map, title=\"Simulated Noise Map\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `signal-to-noise_map` shows the ratio of the signal in each pixel to the noise level in that pixel. It is \n", - "calculated by dividing the `data` by the `noise_map`.\n", - "\n", - "This ratio helps us understand how much of the observed signal is reliable compared to the noise, allowing us to \n", - "see where we can trust the detected signal from the galaxy and where the noise is more significant.\n", - "\n", - "In general, a signal-to-noise ratio greater than 3 indicates that the signal is likely real and not overwhelmed by \n", - "noise. For our datasets, the signal-to-noise ratio peaks at ~70, meaning we can trust the signal detected in the\n", - "image." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(array=dataset.signal_to_noise_map, title=\"Signal-To-Noise Map\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `Imaging` object can display all of these components together, making it a powerful tool for visualizing \n", - "simulated imaging data.\n", - "\n", - "It also shows the Data and PSF on a logarithmic (log10) scale, which helps highlight the faint details in these \n", - "components.\n", - "\n", - "The \"Over Sampling\" plots on the bottom of the figures display advanced features that can be ignored for now." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_imaging_dataset(dataset=dataset)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Output__\n", - "\n", - "We will now save these simulated data to `.fits` files, the standard format used by astronomers for storing images.\n", - "Most imaging data from telescopes like the Hubble Space Telescope (HST) are stored in this format.\n", - "\n", - "The `dataset_path` specifies where the data will be saved, in this case, in the directory \n", - "`autogalaxy_workspace/dataset/imaging/howtogalaxy/`, which contains many example images distributed with \n", - "the `autogalaxy_workspace`.\n", - "\n", - "The files are named `data.fits`, `noise_map.fits`, and `psf.fits`, and will be used in the next tutorial." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_path = Path(\"dataset\", \"imaging\", \"howtogalaxy\")\n", - "print(\"Dataset Path: \", dataset_path)\n", - "\n", - "aplt.fits_imaging(\n", - " dataset=dataset,\n", - " data_path=dataset_path / \"data.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " overwrite=True,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Wrap Up__\n", - "\n", - "In this tutorial, you learned how CCD imaging data of a galaxy is collected using real telescopes like the \n", - "Hubble Space Telescope, and how to simulate this data using the `SimulatorImaging` object.\n", - "\n", - "Let's summarise what we've covered:\n", - "\n", - "- **Optics Blurring**: The optics of a telescope blur the light from galaxies, reducing the clarity and sharpness of \n", - "the images.\n", - "\n", - "- **Poisson Noise**: The process of converting photons to electrons on a CCD introduces Poisson noise, which is random \n", - "variability in the number of electrons collected in each pixel.\n", - "\n", - "- **Background Sky**: Light from the sky is captured along with light from the galaxy, adding a layer of noise across \n", - "the entire image.\n", - "\n", - "- **Simulator**: The `SimulatorImaging` object enables us to simulate realistic imaging data by including all of \n", - "these effects together and contains the `data`, `psf`, and `noise_map` components.\n", - "\n", - "- **Output**: We saved the simulated data to `.fits` files, the standard format used by astronomers for storing images." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_1_introduction/tutorial_3_fitting.ipynb b/notebooks/howtogalaxy/chapter_1_introduction/tutorial_3_fitting.ipynb deleted file mode 100644 index 09d630fe..00000000 --- a/notebooks/howtogalaxy/chapter_1_introduction/tutorial_3_fitting.ipynb +++ /dev/null @@ -1,1086 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tutorial 3: Fitting\n", - "===================\n", - "\n", - "In previous tutorials, we used light profiles to create simulated images of galaxies and visualized how these images\n", - "would appear when captured by a CCD detector on a telescope like the Hubble Space Telescope.\n", - "\n", - "However, this simulation process is the reverse of what astronomers typically do when analyzing real data. Usually,\n", - "astronomers start with an observation\u2014an actual image of a galaxy\u2014and aim to infer detailed information about the\n", - "galaxy\u2019s properties, such as its shape, structure, formation, and evolutionary history.\n", - "\n", - "To achieve this, we must fit the observed image data with a model, identifying the combination of light profiles that\n", - "best matches the galaxy's appearance in the image. In this tutorial, we'll illustrate this process using the imaging\n", - "data simulated in the previous tutorial. Our goal is to demonstrate how we can recover the parameters of the light\n", - "profiles that we used to create the original simulation, as a proof of concept for the fitting procedure.\n", - "\n", - "The process of fitting data introduces essential statistical concepts like the `model`, `residual_map`, `chi-squared`,\n", - "`likelihood`, and `noise_map`. These terms are crucial for understanding how fitting works, not only in astronomy but\n", - "also in any scientific field that involves data modeling. This tutorial will provide a detailed introduction to these\n", - "concepts and show how they are applied in practice to analyze astronomical data.\n", - "\n", - "__Contents__\n", - "\n", - "**Dataset:** Load the imaging dataset previously simulated, consisting of the image, noise map, and PSF.\n", - "**Mask:** Apply a mask to the data, excluding regions with low signal-to-noise ratios from the analysis.\n", - "**Masked Grid:** Create a masked grid containing only coordinates of unmasked pixels.\n", - "**Fitting:** Fit the data with a galaxy model, computing the model image, residuals, chi-squared, and log likelihood.\n", - "**Incorrect Fit:** Demonstrate how small deviations from true parameters impact fit quality.\n", - "**Model Fitting:** Perform a basic model fit, adjusting parameters to improve the fit.\n", - "**Wrap Up:** Summary of the fitting process and key statistical concepts." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "import numpy as np\n", - "from pathlib import Path\n", - "import autogalaxy as ag\n", - "import autogalaxy.plot as aplt" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset__\n", - "\n", - "We begin by loading the imaging dataset that we will use for fitting in this tutorial. This dataset is identical to the \n", - "one we simulated in the previous tutorial, representing how a galaxy would appear if captured by a CCD camera.\n", - "\n", - "In the previous tutorial, we saved this dataset as .fits files in the `autogalaxy_workspace/dataset/imaging/howtogalaxy` \n", - "folder. The `.fits` format is commonly used in astronomy for storing image data along with metadata, making it a\n", - "standard for CCD imaging.\n", - "\n", - "The `dataset_path` below specifies where these files are located: `autogalaxy_workspace/dataset/imaging/howtogalaxy/`." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_path = Path(\"dataset\", \"imaging\", \"howtogalaxy\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset Auto-Simulation__\n", - "\n", - "If the dataset does not already exist on your system, it will be created by running the corresponding\n", - "simulator script. This ensures that all example scripts can be run without manually simulating data first." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "if not dataset_path.exists():\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.run(\n", - " [sys.executable, \"scripts/imaging/simulator.py\"],\n", - " check=True,\n", - " )\n", - "\n", - "\n", - "dataset = ag.Imaging.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " pixel_scales=0.1,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `Imaging` object contains three key components: `data`, `noise_map`, and `psf`:\n", - "\n", - "- `data`: The actual image of the galaxy, which we will analyze.\n", - "\n", - "- `noise_map`: A map indicating the uncertainty or noise level in each pixel of the image, reflecting how much the \n", - " observed signal in each pixel might fluctuate due to instrumental or background noise.\n", - " \n", - "- `psf`: The Point Spread Function, which describes how a point source of light is spread out in the image by the \n", - " telescope's optics. It characterizes the blurring effect introduced by the instrument.\n", - "\n", - "Let's print some values from these components and plot a summary of the dataset to refresh our understanding of the \n", - "imaging data." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(\"Value of first pixel in imaging data:\")\n", - "print(dataset.data.native[0, 0])\n", - "print(\"Value of first pixel in noise map:\")\n", - "print(dataset.noise_map.native[0, 0])\n", - "print(\"Value of first pixel in PSF:\")\n", - "print(dataset.psf.kernel.native[0, 0])\n", - "\n", - "aplt.subplot_imaging_dataset(dataset=dataset)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Mask__\n", - "\n", - "The signal-to-noise map of the image highlights areas where the signal (light from the galaxy) is detected above the \n", - "background noise. Values above 3.0 indicate regions where the galaxy's light is detected with a signal-to-noise ratio\n", - "of at least 3, while values below 3.0 are dominated by noise, where the galaxy's light is not clearly distinguishable.\n", - "\n", - "To ensure the fitting process focuses only on meaningful data, we typically mask out regions with low signal-to-noise \n", - "ratios, removing areas dominated by noise from the analysis. This allows the fitting process to concentrate on the \n", - "regions where the galaxy is clearly detected.\n", - "\n", - "Here, we create a `Mask2D` to exclude certain regions of the image from the analysis. The mask defines which parts of \n", - "the image will be used during the fitting process.\n", - "\n", - "For our simulated image, a circular 3\" mask centered at the center of the image is appropriate, since the simulated \n", - "galaxy was positioned at the center." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "mask = ag.Mask2D.circular(\n", - " shape_native=dataset.shape_native,\n", - " pixel_scales=dataset.pixel_scales,\n", - " radius=3.0, # The circular mask's radius in arc-seconds\n", - " centre=(0.0, 0.0), # center of the image which is also the center of the galaxy\n", - ")\n", - "\n", - "print(mask) # 1 = True, meaning the pixel is masked. Edge pixels are indeed masked.\n", - "print(mask[48:53, 48:53]) # Central pixels are `False` and therefore unmasked." - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can visualize the mask over the galaxy image using an `Imaging`, which helps us adjust the mask as needed. \n", - "This is useful to ensure that the mask appropriately covers the galaxy's light and does not exclude important regions.\n", - "\n", - "To overlay objects like a mask onto a figure, we use the `Visuals2D` object. This tool allows us to add custom \n", - "visuals to any plot, providing flexibility in creating tailored visual representations." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(array=dataset.data, title=\"Imaging Data With Mask\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Once we are satisfied with the mask, we apply it to the imaging data using the `apply_mask()` method. This ensures \n", - "that only the unmasked regions are considered during the analysis." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset = dataset.apply_mask(mask=mask)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "When we plot the masked imaging data again, the mask is now automatically included in the plot, even though we did \n", - "not explicitly pass it using the `Visuals2D` object. The plot also zooms into the unmasked area, showing only the \n", - "region where we will focus our analysis. This is particularly helpful when working with large images, as it centers \n", - "the view on the regions where the galaxy's signal is detected." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(array=dataset.data, title=\"Masked Imaging Data\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The mask is now stored as an additional attribute of the `Imaging` object, meaning it remains attached to the \n", - "dataset. This makes it readily available when we pass the dataset to a `FitImaging` object for the fitting process." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(\"Mask2D:\")\n", - "print(dataset.mask)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In earlier tutorials, we discussed how grids and arrays have `native` and `slim` representations:\n", - "\n", - "- `native`: Represents the original 2D shape of the data, maintaining the full pixel array of the image.\n", - "- `slim`: Represents a 1D array containing only the values from unmasked pixels, allowing for more efficient \n", - " processing when working with large images.\n", - "\n", - "After applying the mask, the `native` and `slim` representations change as follows:\n", - "\n", - "- `native`: The 2D array keeps its original shape, [total_y_pixels, total_x_pixels], but masked pixels (those where \n", - " the mask is True) are set to 0.0.\n", - "- `slim`: This now only contains the unmasked pixel values, reducing the array size \n", - " from [total_y_pixels * total_x_pixels] to just the number of unmasked pixels.\n", - "\n", - "Let's verify this by checking the shape of the data in its `slim` representation." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(\"Number of unmasked pixels:\")\n", - "print(dataset.data.native.shape)\n", - "print(\n", - " dataset.data.slim.shape\n", - ") # This should be lower than the total number of pixels, e.g., 100 x 100 = 10,000" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `mask` object also has a `pixels_in_mask` attribute, which gives the number of unmasked pixels. This should \n", - "match the size of the `slim` data structure." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(dataset.data.mask.pixels_in_mask)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can use the `slim` attribute to print the first unmasked values from the image and noise map:" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(\"First unmasked image value:\")\n", - "print(dataset.data.slim[0])\n", - "print(\"First unmasked noise map value:\")\n", - "print(dataset.noise_map.slim[0])" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Additionally, we can verify that the `native` data structure has zeros at the edges where the mask is applied and \n", - "retains non-zero values in the central unmasked regions." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(\"Example masked pixel in the image's native representation at its edge:\")\n", - "print(dataset.data.native[0, 0])\n", - "print(\"Example unmasked pixel in the image's native representation at its center:\")\n", - "centre = tuple(s // 2 for s in dataset.data.shape_native)\n", - "print(dataset.data.native[centre])" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Masked Grid__\n", - "\n", - "In tutorial 1, we emphasized that the `Grid2D` object is crucial for evaluating a galaxy's light profile. This grid \n", - "contains (y, x) coordinates for each pixel in the image and is used to map out the positions where the galaxy's \n", - "light is calculated.\n", - "\n", - "From a `Mask2D`, we derive a `masked_grid`, which consists only of the coordinates of unmasked pixels. This ensures \n", - "that light profile calculations focus exclusively on regions where the galaxy's light is detected, saving computational \n", - "time and improving efficiency.\n", - "\n", - "Below, we plot the masked grid:" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "masked_grid = mask.derive_grid.unmasked\n", - "\n", - "aplt.plot_grid(grid=masked_grid, title=\"Masked Grid2D\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "By plotting this masked grid over the galaxy image, we can see that the grid aligns with the unmasked pixels of the \n", - "image.\n", - "\n", - "This alignment **is crucial** for accurate fitting because it ensures that when we evaluate a galaxy's light profile, \n", - "the calculations occur only at positions where we have real data from." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(array=dataset.data, title=\"Image Data With 2D Grid Overlaid\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Fitting__\n", - "\n", - "Now that our data is masked, we are ready to proceed with the fitting process.\n", - "\n", - "Fitting the data is done using the `Galaxy` and `Galaxies objects that we introduced in tutorial 2. We will start by \n", - "setting up a `Galaxies`` object, using the same galaxy configuration that we previously used to simulate the \n", - "imaging data. This setup will give us what is known as a 'perfect' fit, as the simulated and fitted models are identical." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "galaxy = ag.Galaxy(\n", - " redshift=0.5,\n", - " bulge=ag.lp.Sersic(\n", - " centre=(0.0, 0.0),\n", - " ell_comps=(0.0, 0.111111),\n", - " intensity=1.0, # in units of e- pix^-1 s^-1\n", - " effective_radius=1.0,\n", - " sersic_index=2.5,\n", - " ),\n", - ")\n", - "\n", - "galaxies = ag.Galaxies(galaxies=[galaxy])" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Next, let's plot the image of the galaxies. This should look familiar, as it is the same image we saw in \n", - "previous tutorials. The difference now is that we use the dataset's `grid`, which corresponds to the `masked_grid` \n", - "we defined earlier. This means that the galaxy image is only evaluated in the unmasked region, skipping calculations \n", - "in masked regions." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(\n", - " array=galaxies.image_2d_from(grid=dataset.grid), title=\"Galaxy Image To Be Fitted\"\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now, we proceed to fit the image by passing both the `Imaging` and `Galaxies` objects to a `FitImaging` object. \n", - "This object will compute key quantities that describe the fit\u2019s quality:\n", - "\n", - "`image`: Creates an image of the galaxies using their image_2d_from() method.\n", - "`model_data`: Convolves the galaxy image with the data's PSF to account for the effects of telescope optics.\n", - "`residual_map`: The difference between the model data and observed data.\n", - "`normalized_residual_map`: Residuals divided by noise values, giving units of noise.\n", - "`chi_squared_map`: Squares the normalized residuals.\n", - "`chi_squared` and `log_likelihood`: Sums the chi-squared values to compute chi_squared, and converts this into \n", - "a log_likelihood, which measures how well the model fits the data (higher values indicate a better fit).\n", - "\n", - "Let's create the fit and inspect each of these attributes:" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "fit = ag.FitImaging(dataset=dataset, galaxies=galaxies)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `model_data` represents the galaxy's image after accounting for effects like PSF convolution.\n", - "\n", - "An important technical note is that when we mask data, we discussed above how the image of the galaxy is not evaluated\n", - "outside the mask and is set to zero. This is a problem for PSF convolution, as the PSF blurs light from these regions\n", - "outside the mask but at its edge into the mask. They must be correctly evaluated to ensure the model image accurately\n", - "represents the image data.\n", - "\n", - "The `FitImaging` object handles this internally, but evaluating the model image in the additional regions outside the mask\n", - "that are close enough to the mask edge to be blurred into the mask. " - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(\"First model image pixel:\")\n", - "print(fit.model_data.slim[0])\n", - "aplt.plot_array(array=fit.model_data, title=\"Model Image\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Even before computing other fit quantities, we can normally assess if the fit is going to be good by visually comparing\n", - "the `data` and `model_data` and assessing if they look similar.\n", - "\n", - "In this example, the galaxies used to fit the data are the same as the galaxies used to simulate it, so the two\n", - "look very similar (the only difference is the noise in the image)." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(array=dataset.data, title=\"Data\")\n", - "aplt.plot_array(array=fit.model_data, title=\"Model Image\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `residual_map` is the different between the observed image and model image, showing where in the image the fit is\n", - "good (e.g. low residuals) and where it is bad (e.g. high residuals).\n", - "\n", - "The expression for the residual map is simply:\n", - "\n", - "\\[ \\text{residual} = \\text{data} - \\text{model\\_data} \\]\n", - "\n", - "The residual-map is plotted below, noting that all values are very close to zero because the fit is near perfect.\n", - "The only non-zero residuals are due to noise in the image." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "residual_map = dataset.data - fit.model_data\n", - "print(\"First residual-map pixel:\")\n", - "print(residual_map.slim[0])\n", - "\n", - "print(\"First residual-map pixel via fit:\")\n", - "print(fit.residual_map.slim[0])\n", - "\n", - "aplt.plot_array(array=fit.residual_map, title=\"Residual Map\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Are these residuals indicative of a good fit to the data? Without considering the noise in the data, it's difficult \n", - "to ascertain. That is, its hard to ascenrtain if a residual value is large or small because this depends on the\n", - "amount of noise in that pixel.\n", - "\n", - "The `normalized_residual_map` divides the residual-map by the noise-map, giving the residual in units of the noise.\n", - "Its expression is:\n", - "\n", - "\\[ \\text{normalized\\_residual} = \\frac{\\text{residual\\_map}}{\\text{noise\\_map}} = \\frac{\\text{data} - \\text{model\\_data}}{\\text{noise\\_map}} \\]\n", - "\n", - "If you're familiar with the concept of standard deviations (sigma) in statistics, the normalized residual map represents \n", - "how many standard deviations the residual is from zero. For instance, a normalized residual of 2.0 (corresponding \n", - "to a 95% confidence interval) means that the probability of the model underestimating the data by that amount is only 5%." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "normalized_residual_map = residual_map / dataset.noise_map\n", - "\n", - "print(\"First normalized residual-map pixel:\")\n", - "print(normalized_residual_map.slim[0])\n", - "\n", - "print(\"First normalized residual-map pixel via fit:\")\n", - "print(fit.normalized_residual_map.slim[0])\n", - "\n", - "aplt.plot_array(array=fit.normalized_residual_map, title=\"Normalized Residual Map\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Next, we define the `chi_squared_map`, which is obtained by squaring the `normalized_residual_map` and serves as a \n", - "measure of goodness of fit.\n", - "\n", - "The chi-squared map is calculated as:\n", - "\n", - "\\[ \\chi^2 = \\left(\\frac{\\text{data} - \\text{model\\_data}}{\\text{noise\\_map}}\\right)^2 \\]\n", - "\n", - "Squaring the normalized residual map ensures all values are positive. For instance, both a normalized residual of -0.2 \n", - "and 0.2 would square to 0.04, indicating the same quality of fit in terms of `chi_squared`.\n", - "\n", - "As seen from the normalized residual map, it's evident that the model provides a good fit to the data, in this\n", - "case because the chi-squared values are close to zero." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "chi_squared_map = (normalized_residual_map) ** 2\n", - "print(\"First chi-squared pixel:\")\n", - "print(chi_squared_map.slim[0])\n", - "\n", - "print(\"First chi-squared pixel via fit:\")\n", - "print(fit.chi_squared_map.slim[0])\n", - "\n", - "aplt.plot_array(array=fit.chi_squared_map, title=\"Chi-Squared Map\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now, we consolidate all the information in our `chi_squared_map` into a single measure of goodness-of-fit \n", - "called `chi_squared`. \n", - "\n", - "It is defined as the sum of all values in the `chi_squared_map` and is computed as:\n", - "\n", - "\\[ \\chi^2 = \\sum \\left(\\frac{\\text{data} - \\text{model\\_data}}{\\text{noise\\_map}}\\right)^2 \\]\n", - "\n", - "This summing process highlights why ensuring all values in the chi-squared map are positive is crucial. If we \n", - "didn't square the values (making them positive), positive and negative residuals would cancel each other out, \n", - "leading to an inaccurate assessment of the model's fit to the data.\n", - "\n", - "The lower the `chi_squared`, the fewer residuals exist between the model's fit and the data, indicating a better \n", - "overall fit!" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "chi_squared = np.sum(chi_squared_map)\n", - "print(\"Chi-squared = \", chi_squared)\n", - "print(\"Chi-squared via fit = \", fit.chi_squared)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The reduced chi-squared is the `chi_squared` value divided by the number of data points (e.g., the number of pixels\n", - "in the mask). \n", - "\n", - "This quantity offers an intuitive measure of the goodness-of-fit, as it normalizes the `chi_squared` value by the\n", - "number of data points. That is, a reduced chi-squared of 1.0 indicates that the model provides a good fit to the data,\n", - "because every data point is fitted with a chi-squared value of 1.0.\n", - "\n", - "A reduced chi-squared value significantly greater than 1.0 indicates that the model is not a good fit to the data,\n", - "whereas a value significantly less than 1.0 suggests that the model is overfitting the data." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "reduced_chi_squared = chi_squared / dataset.mask.pixels_in_mask\n", - "print(\"Reduced Chi-squared = \", reduced_chi_squared)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Another quantity that contributes to our final assessment of the goodness-of-fit is the `noise_normalization`.\n", - "\n", - "The `noise_normalization` is computed as the logarithm of the sum of squared noise values in our data: \n", - "\n", - "\\[\n", - "\\text{{noise\\_normalization}} = \\sum \\log(2 \\pi \\text{{noise\\_map}}^2)\n", - "\\]\n", - "\n", - "This quantity is fixed because the noise-map remains constant throughout the fitting process. Despite this, \n", - "including the `noise_normalization` is considered good practice due to its statistical significance.\n", - "\n", - "Understanding the exact meaning of `noise_normalization` isn't critical for our primary goal of successfully \n", - "fitting a model to a dataset. Essentially, it provides a measure of how well the noise properties of our data align \n", - "with a Gaussian distribution." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "noise_normalization = np.sum(np.log(2 * np.pi * dataset.noise_map**2))\n", - "print(\"Noise Normalization = \", noise_normalization)\n", - "print(\"Noise Normalization via fit = \", fit.noise_normalization)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "From the `chi_squared` and `noise_normalization`, we can define a final goodness-of-fit measure known as \n", - "the `log_likelihood`. \n", - "\n", - "This measure is calculated by taking the sum of the `chi_squared` and `noise_normalization`, and then multiplying the \n", - "result by -0.5:\n", - "\n", - "\\[ \\text{log\\_likelihood} = -0.5 \\times \\left( \\chi^2 + \\text{noise\\_normalization} \\right) \\]\n", - "\n", - "Don't worry about why we multiply by -0.5; it's a standard practice in statistics to ensure the log likelihood is\n", - "defined correctly." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "log_likelihood = -0.5 * (chi_squared + noise_normalization)\n", - "print(\"Log Likelihood = \", log_likelihood)\n", - "print(\"Log Likelihood via fit = \", fit.log_likelihood)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In the previous discussion, we noted that a lower \\(\\chi^2\\) value indicates a better fit of the model to the \n", - "observed data. \n", - "\n", - "When we calculate the log likelihood, we take the \\(\\chi^2\\) value and multiply it by -0.5. This means that a \n", - "higher log likelihood corresponds to a better model fit. Our goal when fitting models to data is to maximize the \n", - "log likelihood.\n", - "\n", - "The **reduced \\(\\chi^2\\)** value provides an intuitive measure of goodness-of-fit. Values close to 1.0 suggest a \n", - "good fit, while values below or above 1.0 indicate potential underfitting or overfitting of the data, respectively. \n", - "In contrast, the log likelihood values can be less intuitive. For instance, a log likelihood value printed above \n", - "might be around 5300.\n", - "\n", - "However, log likelihoods become more meaningful when we compare them. For example, if we have two models, one with \n", - "a log likelihood of 5300 and the other with 5310 we can conclude that the first model fits the data better \n", - "because it has a higher log likelihood by 10.0. \n", - "\n", - "In fact, the difference in log likelihood between models can often be associated with a probability indicating how \n", - "much better one model fits the data compared to another. This can be expressed in terms of standard deviations (sigma). \n", - "\n", - "As a rule of thumb:\n", - "\n", - "- A difference in log likelihood of **2.5** suggests that one model is preferred at the **2.0 sigma** level.\n", - "- A difference in log likelihood of **5.0** indicates a preference at the **3.0 sigma** level.\n", - "- A difference in log likelihood of **10.0** suggests a preference at the **5.0 sigma** level.\n", - "\n", - "All these metrics can be visualized together using the `FitImaging` object, which offers a comprehensive \n", - "overview of the fit quality." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "fit = ag.FitImaging(dataset=dataset, galaxies=galaxies)\n", - "\n", - "aplt.subplot_fit_imaging(fit=fit)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "If you're familiar with model-fitting, you've likely encountered terms like 'residuals', 'chi-squared', \n", - "and 'log_likelihood' before. \n", - "\n", - "These metrics are standard ways to quantify the quality of a model fit. They are applicable not only to 1D data but \n", - "also to more complex data structures like 2D images, 3D data cubes, or any other multidimensional datasets.\n", - "\n", - "__Incorrect Fit___\n", - "\n", - "In the previous section, we successfully created and fitted a galaxy model to the image data, resulting in an \n", - "excellent fit. The residual map and chi-squared map showed no significant discrepancies, indicating that the \n", - "galaxy's light was accurately captured by our model. This optimal solution translates to one of the highest log \n", - "likelihood values possible, reflecting a good match between the model and the observed data.\n", - "\n", - "Now, let's modify our galaxy model to create a fit that is close to the correct solution but slightly off. \n", - "Specifically, we will slightly offset the center of the galaxy by half a pixel (0.05\") in both the x and y directions. \n", - "This change will allow us to observe how even small deviations from the true parameters can impact the quality of the fit." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "galaxy = ag.Galaxy(\n", - " redshift=0.5,\n", - " bulge=ag.lp.Sersic(\n", - " centre=(0.05, 0.05), # This is different from the previous center.\n", - " ell_comps=(0.0, 0.111111),\n", - " intensity=1.0,\n", - " effective_radius=1.0,\n", - " sersic_index=2.5,\n", - " ),\n", - ")\n", - "\n", - "galaxies = ag.Galaxies(galaxies=[galaxy])" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "After implementing this slight adjustment, we can now plot the fit. In doing so, we observe that residuals have \n", - "emerged at the center of the galaxy, which indicates a mismatch between our model and the data. Consequently, \n", - "this discrepancy results in increased chi-squared values, which in turn affects our log likelihood." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "fit_bad = ag.FitImaging(dataset=dataset, galaxies=galaxies)\n", - "\n", - "aplt.subplot_fit_imaging(fit=fit_bad)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Next, we can compare the log likelihood of our current model to the log likelihood value we computed previously." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(\"Previous Likelihood:\")\n", - "print(fit.log_likelihood)\n", - "print(\"New Likelihood:\")\n", - "print(fit_bad.log_likelihood)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "As expected, we observe that the log likelihood has decreased! This decline confirms that our new model is indeed a \n", - "worse fit to the data compared to the original model.\n", - "\n", - "Now, let\u2019s change our galaxy model once more, this time setting it to a position that is far from the true parameters. \n", - "We will offset the galaxy's center significantly to see how this extreme deviation affects the fit quality." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "galaxy = ag.Galaxy(\n", - " redshift=0.5,\n", - " bulge=ag.lp.Sersic(\n", - " centre=(\n", - " 0.65,\n", - " 0.65,\n", - " ), # This position is significantly different from the previous one.\n", - " ell_comps=(0.0, 0.111111),\n", - " intensity=1.0,\n", - " effective_radius=1.0,\n", - " sersic_index=2.5,\n", - " ),\n", - ")\n", - "\n", - "galaxies = ag.Galaxies(galaxies=[galaxy])\n", - "\n", - "fit_very_bad = ag.FitImaging(dataset=dataset, galaxies=galaxies)\n", - "\n", - "aplt.subplot_fit_imaging(fit=fit_very_bad)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "It is now evident that this model provides a terrible fit to the data. The galaxies do not resemble a plausible \n", - "representation of our simulated galaxy dataset, which we already anticipated given that we generated the data ourselves!\n", - "\n", - "As expected, the log likelihood has dropped dramatically with this poorly fitting model." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(\"Previous Likelihoods:\")\n", - "print(fit.log_likelihood)\n", - "print(fit_bad.log_likelihood)\n", - "print(\"New Likelihood:\")\n", - "print(fit_very_bad.log_likelihood)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Model Fitting__\n", - "\n", - "In the previous sections, we used the true model to fit the data, which resulted in a high log likelihood and minimal \n", - "residuals. We also demonstrated how even small deviations from the true parameters can significantly degrade the fit \n", - "quality, reducing the log likelihood.\n", - "\n", - "In practice, however, we don't know the \"true\" model. For example, we might have an image of a galaxy observed with \n", - "the Hubble Space Telescope, but the values for parameters like its `effective_radius`, `sersic_index`, and others are \n", - "unknown. The process of determining the best-fit model is called model fitting, and it is the main topic of \n", - "Chapter 2 of *HowToGalaxy*.\n", - "\n", - "To conclude this section, let's perform a basic, hands-on model fit to develop some intuition about how we can find \n", - "the best-fit model. We'll start by loading a simple dataset that was simulated using a `Sersic` profile, where the \n", - "true parameters of this profile are unknown." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_name = \"simple\"\n", - "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name\n", - "\n", - "dataset = ag.Imaging.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " pixel_scales=0.1,\n", - ")\n", - "\n", - "mask = ag.Mask2D.circular(\n", - " shape_native=dataset.shape_native,\n", - " pixel_scales=dataset.pixel_scales,\n", - " radius=3.0,\n", - ")\n", - "\n", - "dataset = dataset.apply_mask(mask=mask)\n", - "\n", - "aplt.subplot_imaging_dataset(dataset=dataset)\n" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now, you'll try to determine the best-fit model for this image, corresponding to the parameters used to simulate the \n", - "dataset.\n", - "\n", - "We'll use the simplest possible approach: try different combinations of light profile parameters and adjust them \n", - "based on how well each model fits the data. You\u2019ll quickly find that certain parameters produce a much better fit \n", - "than others. For example, determining the correct values of the `centre` should not take too long.\n", - "\n", - "Pay attention to the `log_likelihood` and the `residual_map` as you adjust the parameters. These will guide you in \n", - "determining if your model is providing a good fit to the data. Aim to increase the log likelihood and reduce the \n", - "residuals.\n", - "\n", - "Keep experimenting with different values for a while, seeing how small you can make the residuals and how high you \n", - "can push the log likelihood. Eventually, you\u2019ll likely reach a point where further improvements become difficult, \n", - "even after trying many different parameter values. This is a good point to stop and reflect on your first experience \n", - "with model fitting, and then to scroll to the next cell to see a discussion of the exercise." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "galaxy = ag.Galaxy(\n", - " redshift=0.5,\n", - " bulge=ag.lp.Sersic(\n", - " centre=(1.0, 10), # These are the parameters\n", - " ell_comps=(0.5, 0.9), # you need to adjust\n", - " intensity=1.0, # to try and improve\n", - " effective_radius=1.0, # the model's fit\n", - " sersic_index=1.0, # to the data!\n", - " ),\n", - ")\n", - "galaxies = ag.Galaxies(galaxies=[galaxy])\n", - "\n", - "fit = ag.FitImaging(dataset=dataset, galaxies=galaxies)\n", - "\n", - "aplt.subplot_fit_imaging(fit=fit)\n", - "\n", - "print(\"Log Likelihood:\")\n", - "print(fit.log_likelihood)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Manually guessing model parameters repeatedly is a very inefficient and slow way to find the best fit. If the model \n", - "were more complex\u2014say, if the `galaxy` had additional light profile components beyond just its `bulge` (like a \n", - "second `Sersic` profile representing a `disk`)\u2014the model would become so intricate that this manual approach \n", - "would be practically impossible. This is definitely not how model fitting is done in practice.\n", - "\n", - "However, this exercise has given you a basic intuition for how model fitting works. The statistical inference tools \n", - "that are actually used for model fitting will be introduced in Chapter 2. Interestingly, these tools are not entirely \n", - "different from the approach you just tried. Essentially, they also involve iteratively testing models until those \n", - "with high log likelihoods are found. The key difference is that a computer can perform this process thousands of \n", - "times, and it does so in a much more efficient and strategic way.\n", - "\n", - "__Wrap Up__\n", - "\n", - "In this tutorial, you have learned how to fit a galaxy model to imaging data, a fundamental process in astronomy\n", - "and statistical inference. \n", - "\n", - "Let's summarise what we have covered:\n", - "\n", - "- **Dataset**: We loaded the imaging dataset that we previously simulated, consisting of the galaxy image, noise map,\n", - " and PSF.\n", - " \n", - "- **Mask**: We applied a circular mask to the data, excluding regions with low signal-to-noise ratios from the analysis.\n", - "\n", - "- **Masked Grid**: We created a masked grid, which contains only the coordinates of unmasked pixels, to evaluate the\n", - " galaxy's light profile.\n", - " \n", - "- **Fitting**: We fitted the data with a galaxy model, computing key quantities like the model image, residuals,\n", - " chi-squared, and log likelihood to assess the quality of the fit.\n", - " \n", - "- **Bad Fits**: We demonstrated how even small deviations from the true parameters can significantly impact the fit\n", - " quality, leading to decreased log likelihood values.\n", - " \n", - "- **Model Fitting**: We performed a basic model fit on a simple dataset, adjusting the model parameters to improve the\n", - " fit quality." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_1_introduction/tutorial_4_methods.ipynb b/notebooks/howtogalaxy/chapter_1_introduction/tutorial_4_methods.ipynb deleted file mode 100644 index edbdb30e..00000000 --- a/notebooks/howtogalaxy/chapter_1_introduction/tutorial_4_methods.ipynb +++ /dev/null @@ -1,33 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_1_introduction/tutorial_5_summary.ipynb b/notebooks/howtogalaxy/chapter_1_introduction/tutorial_5_summary.ipynb deleted file mode 100644 index d82ec3ed..00000000 --- a/notebooks/howtogalaxy/chapter_1_introduction/tutorial_5_summary.ipynb +++ /dev/null @@ -1,279 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tutorial 9: Summary\n", - "===================\n", - "\n", - "In this chapter, we have learnt that:\n", - "\n", - " 1) **PyAutoGalaxy** uses Cartesian `Grid2D`'s of $(y,x)$ coordinates to evaluate galaxy luminous emission.\n", - " 2) These grids are combined with light profiles to compute images and other quantities.\n", - " 3) Profiles are grouped together to make galaxies.\n", - " 4) Collections of galaxies (at the same redshift) can be made..\n", - " 5) The Universe's cosmology can be input into this `Galaxies` to convert its units to kiloparsecs.\n", - " 6) The galaxies's image can be used to simulate galaxy `Imaging` like it was observed with a real telescope.\n", - " 7) This data can be fitted, so to as quantify how well a model galaxy system represents the observed image.\n", - "\n", - "In this summary, we'll go over all the different Python objects introduced throughout this chapter and consider how\n", - "they come together as one.\n", - "\n", - "__Contents__\n", - "\n", - "**Initial Setup:** Create profiles, galaxies and a Galaxies object for illustration.\n", - "**Object Composition:** How Galaxies, Galaxy and Profile objects compose together.\n", - "**Visualization:** Customize and visualize any aspect of galaxies using the plotting API.\n", - "**Code Design:** Discussion of PyAutoGalaxy's object-oriented design philosophy.\n", - "**Source Code:** Links to the source code repositories for PyAutoFit, PyAutoArray and PyAutoGalaxy.\n", - "**Wrap Up:** Summary of chapter 1 and preview of the modeling chapter." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autoconf import setup_notebook; setup_notebook()\n", - "\n", - "from pathlib import Path\n", - "import autogalaxy as ag\n", - "import autogalaxy.plot as aplt" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Initial Setup__\n", - "\n", - "Below, we do all the steps we have learned this chapter, making profiles, galaxies, etc. \n", - "\n", - "Note that we use two galaxies, the first of which has a bulge and disk." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "grid = ag.Grid2D.uniform(shape_native=(100, 100), pixel_scales=0.05)\n", - "\n", - "galaxy_0 = ag.Galaxy(\n", - " redshift=0.5,\n", - " bulge=ag.lp.Sersic(\n", - " centre=(0.0, 0.0),\n", - " ell_comps=(0.0, 0.111111),\n", - " intensity=1.0,\n", - " effective_radius=1.0,\n", - " sersic_index=2.5,\n", - " ),\n", - " disk=ag.lp.Exponential(\n", - " centre=(0.0, 0.0),\n", - " ell_comps=(0.0, 0.111111),\n", - " intensity=1.0,\n", - " effective_radius=1.0,\n", - " ),\n", - ")\n", - "\n", - "galaxy_1 = ag.Galaxy(\n", - " redshift=0.5,\n", - " bulge=ag.lp.Sersic(\n", - " centre=(1.0, 1.0),\n", - " ell_comps=(0.0, 0.111111),\n", - " intensity=1.0,\n", - " effective_radius=1.0,\n", - " sersic_index=2.5,\n", - " ),\n", - ")\n", - "\n", - "galaxies = ag.Galaxies(galaxies=[galaxy_0, galaxy_1])" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Object Composition__\n", - "\n", - "Lets now consider how all of the objects we've covered throughout this chapter (`LightProfile`'s, `MassProfile`'s,\n", - "`Galaxy`'s, `Galaxies`'s) come together.\n", - "\n", - "The `Galaxies` contain the `Galaxy`'s which contains the `Profile`'s:" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(galaxies[0])\n", - "print()\n", - "print(galaxies[0].bulge)\n", - "print()\n", - "print(galaxies[0].disk)\n", - "print()\n", - "print(galaxies[1].bulge)\n", - "print()" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Once we have defined the galaxies, we can plot any quantity introduced throughout this chapter for a specific component, \n", - "a single galaxy, or multiple galaxies as needed.\n", - "\n", - "For example, if we want to plot the image of the first galaxy's bulge and disk, we can do this in a variety of \n", - "different ways." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(array=galaxies.image_2d_from(grid=grid), title=\"Image\")\n", - "\n", - "aplt.plot_array(array=galaxies[0].image_2d_from(grid=grid), title=\"Image\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Understanding how these objects decompose into the different components of a galaxy is important for general \n", - "**PyAutoGalaxy** use.\n", - "\n", - "As the galaxy systems that we analyse become more complex, it is useful to know how to decompose their light \n", - "profiles, galaxies and galaxies to extract different pieces of information about the galaxy. \n", - "\n", - "For example, we made our galaxy above with two light profiles, a `bulge` and `disk`. We can plot the image of \n", - "each component individually, now that we know how to break-up the different components of the galaxies." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(array=galaxies[0].bulge.image_2d_from(grid=grid), title=\"Bulge Image\")\n", - "\n", - "aplt.plot_array(array=galaxies[0].disk.image_2d_from(grid=grid), title=\"Disk Image\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Visualization__\n", - "\n", - "Furthermore, using the `MatPLot2D` and `Visuals2D` objects we can visualize any aspect we're interested \n", - "in and fully customize the figure. \n", - "\n", - "Before beginning chapter 2 of **HowToGalaxy**, you should checkout the package `autogalaxy_workspace/plot`. \n", - "This provides a full API reference of every plotting option in **PyAutoGalaxy**, allowing you to create your own \n", - "fully customized figures of galaxies with minimal effort!" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(array=galaxies[0].bulge.image_2d_from(grid=grid), title=\"Bulge Image\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "And, we're done, not just with the tutorial, but the chapter!\n", - "\n", - "__Code Design__\n", - "\n", - "To end, I want to quickly talk about the **PyAutoGalaxy** code-design and structure, which was really the main topic of\n", - "this tutoriag.\n", - "\n", - "Throughout this chapter, we never talk about anything like it was code. We didn`t refer to 'variables', 'parameters`' \n", - "'functions' or 'dictionaries', did we? Instead, we talked about 'galaxies'. We discussed \n", - "the objects that we, as scientists, think about when we consider a galaxy system.\n", - "\n", - "Software that abstracts the underlying code in this way follows an `object-oriented design`, and it is our hope \n", - "with **PyAutoGalaxy** that we've made its interface (often called the API for short) very intuitive, whether you were\n", - "previous familiar with galaxy morphology or a complete newcomer!\n", - "\n", - "__Source Code__\n", - "\n", - "If you do enjoy code, variables, functions, and parameters, you may want to dig deeper into the **PyAutoGalaxy** source \n", - "code at some point in the future. Firstly, you should note that all of the code we discuss throughout the **HowToGalaxy** \n", - "lectures is not contained in just one project (e.g. the **PyAutoGalaxy** GitHub repository) but in fact three repositories:\n", - "\n", - "**PyAutoFit** - Everything required for modeling (the topic of chapter 2): https://github.com/rhayes777/PyAutoFit\n", - "\n", - "**PyAutoArray** - Handles all data structures and Astronomy dataset objects: https://github.com/Jammy2211/PyAutoArray\n", - "\n", - "**PyAutoGalaxy** - Contains the light profiles and galaxies: https://github.com/Jammy2211/PyAutoGalaxy\n", - "\n", - "Instructions on how to build these projects from source are provided here:\n", - "\n", - "https://pyautogalaxy.readthedocs.io/en/latest/installation/source.html\n", - "\n", - "We take a lot of pride in our source code, so I can promise you its well written, well documented and thoroughly \n", - "tested (check out the `test` directory if you're curious how to test code well!).\n", - "\n", - "__Wrap Up__\n", - "\n", - "You`ve learn a lot in this chapter, but what you have not learnt is how to 'model' a real galaxy.\n", - "\n", - "In the real world, we have no idea what the 'correct' combination of light profiles are that will give a good fit to \n", - "a galaxy. Modeling is the process of finding the model which provides a good fit and it is the topic of chapter 2 \n", - "of **HowToGalaxy**.\n", - "\n", - "Finally, if you enjoyed doing the **HowToGalaxy** tutorials please git us a star on the **PyAutoGalaxy** GitHub\n", - "repository: \n", - "\n", - " https://github.com/Jammy2211/PyAutoGalaxy\n", - "\n", - "Even the smallest bit of exposure via a GitHub star can help our project grow!" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_2_modeling/README.rst b/notebooks/howtogalaxy/chapter_2_modeling/README.rst deleted file mode 100644 index 1584168d..00000000 --- a/notebooks/howtogalaxy/chapter_2_modeling/README.rst +++ /dev/null @@ -1,30 +0,0 @@ -In chapter 2, we'll take you through how to model galaxies using a non-linear search. - -**Colab** links to every tutorial are included. - -Files ------ - -`Tutorial 1: Non-linear Search `_ -- How a non-linear search is used to fit a model and the concepts of a parameter space and priors. - -`Tutorial 2: Practicalities `_ -- Practicalities of performing model-fitting, like how to inspect the results on your hard-disk. - -`Tutorial 3: Realism and Complexity `_ -- Finding a balance between realism and complexity when composing and fitting a model. - -`Tutorial 4: Dealing with Failure `_ -- What to do when PyAutoGalaxy finds an inaccurate model. - -`Tutorial 5: Linear Profiles `_ -- Light profiles which capture complex morphologies in a reduced number of non-linear parameters. - -`Tutorial 6: Masking `_ -- How to mask your data to improve the model. - -`Tutorial 7: Results `_ -- Overview of the results available after successfully fitting a model. - -`Tutorial 8: Need for Speed `_ -- How to fit complex models whilst balancing efficiency and run-time. diff --git a/notebooks/howtogalaxy/chapter_2_modeling/tutorial_1_non_linear_search.ipynb b/notebooks/howtogalaxy/chapter_2_modeling/tutorial_1_non_linear_search.ipynb deleted file mode 100644 index 693ae367..00000000 --- a/notebooks/howtogalaxy/chapter_2_modeling/tutorial_1_non_linear_search.ipynb +++ /dev/null @@ -1,991 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tutorial 1: Non-linear Search\n", - "=============================\n", - "\n", - "The starting point for most scientific analysis conducted by an Astronomer is that they have observations of a galaxy\n", - "using a telescope like the Hubble Space Telescope, and seek to learn about the galaxy and the Universe from these\n", - "observations. With **PyAutoGalaxy**, we seek to learn about the galaxy's structure and morphology, asking questions like\n", - "how big is the galaxy, is it disky or bulgy, and how is its light distributed?\n", - "\n", - "To answer these questions, we must therefore fit the dataset with a model of the galaxy, where the model defines the\n", - "light profile that make up the galaxy we fit. Our goal is the find the combination of light profile parameters that\n", - "best-fit the data, such that the model represents the galaxy, and therefore the Universe, as well as possible.\n", - "\n", - "This process is called model-fitting, or \"modeling\" for short, and we actually did our first example of this in the\n", - "previous chapter. If you recall, in the fitting tutorial we set up a fit to a simulated dataset where the parameter\n", - "used to simulate the data were unknown, and you guessed the values of the parameters that best fit the data. You\n", - "iteratively improved the model-fit by guessing new parameters, over and over, finding solutions which produced\n", - "higher `log_likelihood` values.\n", - "\n", - "However, this approach was not optimal: it was manual and slow, we had no certainty that we had found the best\n", - "(e.g. maximum likelihood) solution and for more complex models, with more parameters and light profiles, it would\n", - "have been completely unfeasible.\n", - "\n", - "In this chapter, we perform modeling as a scientist would actually do it, and introduce the statistical inference\n", - "techniques that will ultimately allow us to fit complex models made of many light profiles to real galaxy data,\n", - "and begin learning about real galaxies in the Universe.\n", - "\n", - "This first tutorial introduces a number of key statistical concepts that are fundamental to understanding how\n", - "model-fitting works, both for **PyAutoGalaxy** and in general.\n", - "\n", - "__Overview__\n", - "\n", - "In this tutorial, we will use a non-linear search to fit a single Serisc light profile to simulated imaging of a\n", - "galaxy. We will:\n", - "\n", - "- Introduce concept like a \"parameter space\", \"likelihood surface\" and \"priors\", and relate them to how a non-linear\n", - " search works.\n", - "\n", - "- Introduce the `Analysis` class, which defines the `log_likelihood_function` that quantifies the goodness of fit of a\n", - " model instance to the data.\n", - "\n", - "- Fit datasets with different non-linear searches, including a maximum likelihood estimator (MLE),\n", - " Markok Chain Monte Carlo (MCMC) and nested sampling.\n", - "\n", - "__Contents__\n", - "\n", - "This tutorial is split into the following sections:\n", - "\n", - "- **Parameter Space**: Introduce the concept of a \"parameter space\" and how it relates to model-fitting.\n", - "- **Non-Linear Search**: Introduce the concept of a \"non-linear search\" and how it fits models to data.\n", - "- **Search Types**: Introduce the maximum likelihood estimator (MLE), Markov Chain Monte Carlo (MCMC) and nested sampling search algorithms used in this tutorial.\n", - "- **Deeper Background**: Provide links to resources that more thoroughly describe the statistical principles that underpin non-linear searches.\n", - "- **Data**: Load and plot the galaxy dataset we'll fit.\n", - "- **Model**: Introduce the galaxy model we'll fit to the data.\n", - "- **Priors**: Introduce priors and how they are used to define the parameter space and guide the non-linear search.\n", - "- **Analysis**: Introduce the `Analysis` class, which contains the `log_likelihood_function` used to fit the model to the data.\n", - "- **Searches**: An overview of the searches used in this tutorial.\n", - "- **Maximum Likelihood Estimation (MLE)**: Perform a model-fit using the MLE search.\n", - "- **Markov Chain Monte Carlo (MCMC)**: Perform a model-fit using the MCMC search.\n", - "- **Nested Sampling**: Perform a model-fit using the nested sampling search.\n", - "- **Result**: The result of the model-fit, including the maximum likelihood model.\n", - "- **Samples**: The samples of the non-linear search, used to compute parameter estimates and uncertainties.\n", - "- **Customizing Searches**: How to customize the settings of the non-linear search.\n", - "- **Wrap Up**: A summary of the concepts introduced in this tutorial.\n", - "\n", - "__Parameter Space__\n", - "\n", - "In mathematics, a function is defined by its parameters, which map inputs to outputs. For example, consider the simple function:\n", - "\n", - "\\[\n", - "f(x) = x^2\n", - "\\]\n", - "\n", - "Here, \\(x\\) is the input parameter, and \\(f(x)\\) returns the output \\(x^2\\). This relationship defines\n", - "the \"parameter space\" of the function, which in this case forms a parabola.\n", - "\n", - "Functions can also have multiple parameters, such as:\n", - "\n", - "\\[\n", - "f(x, y, z) = x + y^2 - z^3\n", - "\\]\n", - "\n", - "This defines a parameter space in three dimensions, representing the relationships between \\(x\\), \\(y\\), \\(z\\),\n", - "and the output \\(f(x, y, z)\\).\n", - "\n", - "This concept of parameter space is closely related to how we approach model-fitting. For instance, in chapter 1, w\n", - "e created instances of a `Galaxy` object with\n", - "parameters like \\( (\\text{`centre_0`}, \\text{`centre_1`}, \\text{`ell_comps_0`}, \\text{`ell_comps_1`}, \\text{`intensity`}, \\text{`effective_radius`}, \\text{`sersic_index`}) \\).\n", - "These parameters were used to fit data and compute a log likelihood.\n", - "\n", - "We can think of this process as analogous to the function \\(f(\\( (\\text{`centre_0`}, \\text{`centre_1`}, \\text{`ell_comps_0`}, \\text{`ell_comps_1`}, \\text{`intensity`}, \\text{`effective_radius`}, \\text{`sersic_index`}) \\))\\),\n", - "where the output is the log likelihood. This function, which maps parameter values to a log likelihood, is known\n", - "as the \"likelihood function\" in statistical inference. To be explicit, we\u2019ll refer to it as the `log_likelihood_function`\n", - "since it deals with the log of the likelihood function.\n", - "\n", - "By framing the likelihood this way, we can think of our model as having its own parameter space\u2014a multidimensional\n", - "surface defined by all possible values of the\n", - "parameters \\( (\\text{`centre_0`}, \\text{`centre_1`}, \\text{`ell_comps_0`}, \\text{`ell_comps_1`}, \\text{`intensity`}, \\text{`effective_radius`}, \\text{`sersic_index`}) \\).\n", - "This surface, known as the \"likelihood surface,\" represents how the log likelihood changes across different parameter\n", - "values. During model-fitting, our goal is to find the peak of this surface, where the fit to the data is optimal.\n", - "\n", - "This parameter space is \"non-linear,\" meaning the relationship between the model parameters and the log likelihood is\n", - "not a simple linear one. Because of this non-linearity, we cannot predict the log likelihood from a given set of model\n", - "parameters without actually performing a fit to the data, as we did in tutorial 1.\n", - "\n", - "__Non-Linear Search__\n", - "\n", - "Now that we understand our problem in terms of a non-linear parameter space with a likelihood surface, we can\n", - "introduce the method used to fit the model to the data \u2014- the \"non-linear search\".\n", - "\n", - "Previously, our approach involved manually guessing models until finding one with a good fit and high log likelihood.\n", - "Surprisingly, this random guessing forms the basis of how model-fitting using a non-linear search actually works!\n", - "\n", - "A non-linear search involves systematically guessing many models while tracking their log likelihoods. As the\n", - "algorithm progresses, it tends to favor models with parameter combinations that have previously yielded higher\n", - "log likelihoods. This iterative refinement helps to efficiently explore the vast parameter space.\n", - "\n", - "There are two key differences between guessing random models and using a non-linear search:\n", - "\n", - "- **Computational Efficiency**: The non-linear search can evaluate the log likelihood of a model parameter\n", - " combinations in milliseconds and therefore many thousands of models in minutes. This computational speed enables\n", - " it to thoroughly sample potential solutions, which would be impractical for a human.\n", - "\n", - "- **Effective Sampling**: The search algorithm maintains a robust memory of previously guessed models and their log\n", - " likelihoods. This allows it to sample potential solutions more thoroughly and converge on the highest\n", - " likelihood solutions more efficiently, which is again impractical for a human.\n", - "\n", - "Think of the non-linear search as systematically exploring parameter space to pinpoint regions with the highest log\n", - "likelihood values. Its primary goal is to identify and converge on the parameter values that best describe the data.\n", - "\n", - "__Search Types__\n", - "\n", - "There are different types of non-linear searches, each of which explores parameter space in a unique way.\n", - "In this example, we will use three types of searches, which broadly represent the various approaches to non-linear\n", - "searches used in statistical inference.\n", - "\n", - "These are:\n", - "\n", - "- **Maximum Likelihood Estimation (MLE)**: This method aims to find the model that maximizes the likelihood function.\n", - " It does so by testing nearby models and adjusting parameters in the direction that increases the likelihood.\n", - "\n", - "- **Markov Chain Monte Carlo (MCMC)**: This approach uses a group of \"walkers\" that explore parameter space randomly.\n", - " The likelihood at each walker's position influences the probability of the walker moving to a new position.\n", - "\n", - "- **Nested Sampling**: This technique samples points from the parameter space iteratively. Lower likelihood points\n", - " are replaced by higher likelihood ones, gradually concentrating the samples in regions of high likelihood.\n", - "\n", - "We will provide more details on each of these searches below.\n", - "\n", - "__Deeper Background__\n", - "\n", - "**The descriptions of how searches work in this example are simplfied and phoenomenological and do not give a full\n", - "description of how they work at a deep statistical level. The goal is to provide you with an intuition for how to use\n", - "them and when different searches are appropriate for different problems. Later tutorials will provide a more formal\n", - "description of how these searches work.**\n", - "\n", - "If you're interested in learning more about these principles, you can explore resources such as:\n", - "\n", - "- [Markov Chain Monte Carlo (MCMC)](https://en.wikipedia.org/wiki/Markov_chain_Monte_Carlo)\n", - "- [Introduction to MCMC Sampling](https://twiecki.io/blog/2015/11/10/mcmc-sampling/)\n", - "- [Nested Sampling](https://www.imperial.ac.uk/media/imperial-college/research-centres-and-groups/astrophysics/public/icic/data-analysis-workshop/2016/NestedSampling_JRP.pdf)\n", - "- [A Zero-Math Introduction to MCMC Methods](https://towardsdatascience.com/a-zero-math-introduction-to-markov-chain-monte-carlo-methods-dcba889e0c50)" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autoconf import setup_notebook; setup_notebook()\n", - "\n", - "from pathlib import Path\n", - "import autogalaxy as ag\n", - "import autogalaxy.plot as aplt" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__PyAutoFit__\n", - "\n", - "Modeling uses the probabilistic programming language\n", - "[PyAutoFit](https://github.com/rhayes777/PyAutoFit), an open-source project that allows complex model\n", - "fitting techniques to be straightforwardly integrated into scientific modeling software. \n", - "\n", - "**PyAutoFit** is actually a spin-off project of **PyAutoGalaxy**. whereby we found that the statistic techniques and\n", - "methods we applied to model galaxies could be used in a more general setting to many different scientific \n", - "problems. Check it out if you are interested in developing your own software to perform advanced model-fitting!\n", - "\n", - "We import this library separately from **PyAutoGalaxy**." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "import autofit as af" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Initial Setup__\n", - "\n", - "Let's first load the `Imaging` dataset, which we will use to fit a model with a non-linear search.\n", - "\n", - "The galaxy in this image was generated using a `Sersic` light profile, which we'll also use in our model fitting \n", - "in this tutorial. This means the model we are going to fit is identical to the one used to simulate the data, \n", - "allowing us to assess the fitting process under controlled conditions.\n", - "\n", - "The dataset, as well as all subsequent datasets used in future tutorials, is stored in \n", - "the `autogalaxy_workspace/dataset/imaging` folder. " - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_name = \"simple__sersic\"\n", - "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset Auto-Simulation__\n", - "\n", - "If the dataset does not already exist on your system, it will be created by running the corresponding\n", - "simulator script. This ensures that all example scripts can be run without manually simulating data first." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "if not dataset_path.exists():\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.run(\n", - " [sys.executable, \"scripts/howtogalaxy/simulators/sersic.py\"],\n", - " check=True,\n", - " )\n", - "\n", - "\n", - "dataset = ag.Imaging.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " pixel_scales=0.1,\n", - ")\n", - "\n", - "aplt.subplot_imaging_dataset(dataset=dataset)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Mask__\n", - "\n", - "The fit requires a mask, which we define as a 3.0\" circle." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "mask = ag.Mask2D.circular(\n", - " shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=3.0\n", - ")\n", - "\n", - "dataset = dataset.apply_mask(mask=mask)\n", - "\n", - "aplt.subplot_imaging_dataset(dataset=dataset)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Model__\n", - "\n", - "To compose a model, we will set up a `Galaxy` using a `af.Model`. Instead of manually specifying every parameter \n", - "for the galaxy's light profiles (as we did before), we will now define the galaxy using only the class of each \n", - "profile. Using a `Model` object tells **PyAutoGalaxy** that the parameters of the profiles should be fitted for \n", - "during the non-linear search.\n", - "\n", - "In this case, we'll model the galaxy with an elliptical Sersic light profile, which is linear and represents \n", - "its bulge component (the same profile used to simulate the galaxy). " - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "galaxy_model = af.Model(ag.Galaxy, redshift=0.5, bulge=ag.lp_linear.Sersic)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We now input the model component into a `Collection` object, which groups all the model components used to fit the data.\n", - "\n", - "As with profiles, we give galaxies descriptive names like `bulge`, or `disk`. Since this model has only one \n", - "galaxy, we'll simply refer to it as `galaxy` throughout the tutorials.\n", - "\n", - "It may seem odd that we define two `Collections`, with the `Collection` in the outer loop only having a `galaxies`\n", - "attribute. In future tutorials, we'll see that we can add additional model-components to a model other than just\n", - "galaxies, and the API below therefore makes it simple to extend the model to include these components." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "model = af.Collection(galaxies=af.Collection(galaxy=galaxy_model))" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `info` attribute shows the model in a readable format.\n", - "\n", - "[The `info` below may not display optimally on your computer screen, for example the whitespace between parameter\n", - "names on the left and parameter priors on the right may lead them to appear across multiple lines. This is a\n", - "common issue in Jupyter notebooks.\n", - "\n", - "The`info_whitespace_length` parameter in the file `config/generag.yaml` in the [output] section can be changed to \n", - "increase or decrease the amount of whitespace (The Jupyter notebook kernel will need to be reset for this change to \n", - "appear in a notebook).]" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(model.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Priors__\n", - "\n", - "When we examine the `.info` of our model, we notice that each parameter (like `centre`, `effective_radius`, \n", - "and `sersic_index` in our `Sersic` model) is associated with priors, such as `UniformPrior`. Priors define the \n", - "range of permissible values that each parameter can assume during the model fitting process, for example a uniform\n", - "prior means that a parameter is equally likely to be any value within the given range, but cannot be outside of it.\n", - "\n", - "The priors displayed above use default values defined in the `config/priors` directory. These default values have\n", - "been chosen to be broad, and contain the breadth of plausible solutions one should expect when fitting light\n", - "profiles to a real galaxy.\n", - "\n", - "For instance, consider the `centre` parameter of our `Sersic` light profile. In theory, it could take on any value from \n", - "negative to positive infinity. However, imaging datasets are typically reduced such that the galaxy centre is close \n", - "to (0.0\", 0.0\"). Therefore, a `GaussianPrior` with `mean=0.0` and `sigma=0.1` is a good description of where the\n", - "galaxy `centre` is. \n", - "\n", - "If the galaxy had a different centre in the dataset, we would change the mean of the prior to reflect this.\n", - "However, in general, we advise all galaxy images are reduced such that the galaxy is at (0.0\", 0.0\").\n", - "\n", - "Priors serve two primary purposes:\n", - "\n", - "**Defining Valid Parameter Space:** Priors specify the range of parameter values that constitute valid solutions. \n", - "This ensures that our model explores only those solutions that are consistent with our observed data and physical \n", - "constraints.\n", - "\n", - "**Incorporating Prior Knowledge:** Priors also encapsulate our prior beliefs or expectations about the model \n", - "parameters. For instance, if we have previously fitted a similar model to another dataset and obtained certain \n", - "parameter values, we can incorporate this knowledge into our priors for a new dataset. This approach guides the \n", - "model fitting process towards parameter values that are more probable based on our prior understanding.\n", - "\n", - "Below, we manually specify the priors on all parameter in our `Sersic` model. The custom priors below are\n", - "close to the default priors drawn via configfiles, with the main purpose of the code below to show you how to\n", - "customize priors yourself." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "model.galaxies.galaxy.bulge.centre.centre_0 = af.GaussianPrior(mean=0.0, sigma=0.1)\n", - "model.galaxies.galaxy.bulge.centre.centre_1 = af.GaussianPrior(mean=0.0, sigma=0.1)\n", - "model.galaxies.galaxy.bulge.ell_comps.ell_comps_0 = af.UniformPrior(\n", - " lower_limit=-1.0, upper_limit=1.0\n", - ")\n", - "model.galaxies.galaxy.bulge.ell_comps.ell_comps_1 = af.UniformPrior(\n", - " lower_limit=-1.0, upper_limit=1.0\n", - ")\n", - "model.galaxies.galaxy.bulge.effective_radius = af.UniformPrior(\n", - " lower_limit=0.0, upper_limit=10.0\n", - ")\n", - "model.galaxies.galaxy.bulge.sersic_index = af.UniformPrior(\n", - " lower_limit=0.8, upper_limit=8.0\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "By reprinting the `model.info`, we can see that the priors have been updated to the values we specified." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(model.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Analysis__\n", - "\n", - "The `AnalysisImaging` object defines how an instance of a model, consisting of a set of parameters values for the \n", - "light profiles, is fitted to the `Imaging` dataset.\n", - "\n", - "The fit is performed using the analysis class's `log_likelihood_function`, which in model-fitting is a commonly used \n", - "term to describe a function that given a model and data, fits the model to the data to return a value of log \n", - "likelihood. \n", - "\n", - "In the fitting tutorial in chapter 1 you essentially performed this likelihood function yourself by hand, when you \n", - "entered different models to a `FitImaging` object and used its `log_likelihood` property to quantify how well it \n", - "fitted the data.\n", - "\n", - "A detailed step-by-step visual guide of the likelihood function is provided \n", - "at `autogalaxy_workspace/*/imaging/log_likelihood_function/parametric.ipynb`." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "analysis = ag.AnalysisImaging(dataset=dataset, use_jax=True)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Searches__\n", - "\n", - "To perform a non-linear search, we create an instance of a `NonLinearSearch` object. **PyAutoFit** offers many options \n", - "for this. A detailed description of each search method and guidance on when to use them can be found in \n", - "the [search cookbook](https://pyautofit.readthedocs.io/en/latest/cookbooks/search.html).\n", - "\n", - "In this tutorial, we\u2019ll focus on three searches that represent different approaches to model fitting:\n", - "\n", - "1. **Maximum Likelihood Estimation (MLE)** using the `LBFGS` non-linear search.\n", - "2. **Markov Chain Monte Carlo (MCMC)** using the `Emcee` non-linear search.\n", - "3. **Nested Sampling** using the `Nautilus` non-linear search.\n", - "\n", - "In this example, non-linear search results are stored in memory rather and not written to hard disk because the fits \n", - "are fast and can therefore be easily regenerated. The next tutorial will perform fits which write results to the\n", - "hard-disk and discuss the outputs that are generated.\n", - "\n", - "__Maximum Likelihood Estimation (MLE)__\n", - "\n", - "Maximum likelihood estimation (MLE) is the most straightforward type of non-linear search. Here is a simplified \n", - "overview of how it works:\n", - "\n", - "1. Starts at a point in parameter space with a set of initial values for the model parameters.\n", - "2. Calculates the likelihood of the model at this starting point.\n", - "3. Evaluates the likelihood at nearby points to estimate the gradient, determining the direction in which to move \"up\" in parameter space.\n", - "4. Moves to a new point where, based on the gradient, the likelihood is higher.\n", - "\n", - "This process repeats until the search finds a point where the likelihood can no longer be improved, indicating that \n", - "the maximum likelihood has been reached.\n", - "\n", - "The `LBFGS` search is an example of an MLE algorithm that follows this iterative procedure. Let\u2019s see how it \n", - "performs on our Sersic model.\n", - "\n", - "In the example below, we do not specify a starting point for the MLE, so it begins at the center of the prior \n", - "range for each parameter." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "search = af.LBFGS()" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "To begin the model-fit via the non-linear search, we pass it our model and analysis and begin the fit.\n", - "\n", - "The fit will take a minute or so to run." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(\n", - " \"\"\"\n", - " The non-linear search has begun running.\n", - " This Jupyter notebook cell with progress once the search has completed - this could take a few minutes!\n", - " \"\"\"\n", - ")\n", - "\n", - "model = af.Collection(galaxies=af.Collection(galaxy=galaxy_model))\n", - "\n", - "result = search.fit(model=model, analysis=analysis)\n", - "\n", - "print(\"The search has finished run - you may now continue the notebook.\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Upon completion the non-linear search returns a `Result` object, which contains information about the model-fit.\n", - "\n", - "The `info` attribute shows the result in a readable format.\n", - "\n", - "[Above, we discussed that the `info_whitespace_length` parameter in the config files could b changed to make \n", - "the `model.info` attribute display optimally on your computer. This attribute also controls the whitespace of the\n", - "`result.info` attribute.]" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(result.info)\n" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "One thing the result contains we'll use now is the `FitImaging` object that corresponds to the set of model\n", - "parameters that gave the maximum log likelihood solution. \n", - "\n", - "We plot this object to inspect how good our fit was." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_fit_imaging(fit=result.max_log_likelihood_fit)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The fit quality is poor, with significant residuals, meaning that the MLE failed to identify the correct model.\n", - "\n", - "This happened because the starting point of the search was a poor match to the data, placing it far from the true \n", - "solution in parameter space. As a result, after moving \"up\" the likelihood gradient several times, the search \n", - "settled into a \"local maximum\", a local peak in the likelihood surface that was better than models that surrounded\n", - "it in parameter space, but far from the global maximum solution.\n", - "\n", - "To achieve a better fit with MLE, the search needs to begin in a region of parameter space where the log likelihood \n", - "is higher. This process is known as \"initialization\" and it involves providing the search with an \n", - "appropriate \"starting point\" in parameter space.\n", - "\n", - "The code below shows how we can customize the starting point of the search to provide a better initialization,\n", - "with the values input below close to the true values used to simulate the dataset." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "initializer = af.InitializerParamStartPoints(\n", - " {\n", - " model.galaxies.galaxy.bulge.centre.centre_0: 0.0,\n", - " model.galaxies.galaxy.bulge.centre.centre_1: 0.0,\n", - " model.galaxies.galaxy.bulge.ell_comps.ell_comps_0: 0.1,\n", - " model.galaxies.galaxy.bulge.ell_comps.ell_comps_1: 0.05,\n", - " model.galaxies.galaxy.bulge.effective_radius: 1.0,\n", - " model.galaxies.galaxy.bulge.sersic_index: 3.7,\n", - " }\n", - ")\n", - "\n", - "search = af.LBFGS(initializer=initializer)\n", - "\n", - "print(\n", - " \"\"\"\n", - " The non-linear search has begun running.\n", - " This Jupyter notebook cell with progress once the search has completed - this could take a few minutes!\n", - " \"\"\"\n", - ")\n", - "\n", - "model = af.Collection(galaxies=af.Collection(galaxy=galaxy_model))\n", - "\n", - "result = search.fit(model=model, analysis=analysis)\n", - "\n", - "print(\"The search has finished run - you may now continue the notebook.\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "By printing `result.info` and looking at the maximum log likelihood model, we can confirm the search provided a\n", - "good model fit with a much higher likelihood than the incorrect model above." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(result.info)\n", - "\n", - "aplt.subplot_fit_imaging(fit=result.max_log_likelihood_fit)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The MLE is a great starting point for model-fitting because it\u2019s fast, conceptually simple, and often yields \n", - "accurate results. It is especially effective if you can provide a good initialization, allowing it to find the \n", - "best-fit solution quickly.\n", - "\n", - "However, MLE has its limitations. As seen above, it can get \"stuck\" in a local maximum, particularly if the \n", - "starting point is poorly chosen. In complex model-fitting problems, providing a suitable starting point can be \n", - "challenging. While MLE performed well in the example with just three parameters, it struggles with models that have \n", - "many parameters, as the complexity of the likelihood surface makes simply moving \"up\" the gradient less effective.\n", - "\n", - "The MLE also does not provide any information on the errors on the parameters, which is a significant limitation.\n", - "The next two types of searches \"map out\" the likelihood surface, such that they not only infer the maximum likelihood\n", - "solution but also quantify the errors on the parameters.\n", - "\n", - "__Markov Chain Monte Carlo (MCMC)__\n", - "\n", - "Markov Chain Monte Carlo (MCMC) is a more powerful method for model-fitting, though it is also more computationally \n", - "intensive and conceptually complex. Here\u2019s a simplified overview:\n", - "\n", - "1. Place a set of \"walkers\" in parameter space, each with random parameter values.\n", - "2. Calculate the likelihood of each walker's position.\n", - "3. Move the walkers to new positions, guided by the likelihood of their current positions. Walkers in high-likelihood \n", - "regions encourage those in lower regions to move closer to them.\n", - "\n", - "This process repeats, with the walkers converging on the highest-likelihood regions of parameter space.\n", - "\n", - "Unlike MLE, MCMC thoroughly explores parameter space. While MLE moves a single point up the likelihood gradient, \n", - "MCMC uses many walkers to explore high-likelihood regions, making it more effective at finding the global maximum, \n", - "though slower. \n", - "\n", - "There is a reduced risk of getting stuck in local maxima, because whilst some walkers may temporarily get stuck in \n", - "one, other walkers will explore other regions of parameter space and encourage the walker in local maxima to move \n", - "away from it and head towards higher likelihood regions.\n", - "\n", - "In the example below, we use the `Emcee` MCMC search to fit the galaxy. The search starts with walkers \n", - "initialized in a \"ball\" around the center of the model\u2019s priors, similar to the MLE search that failed earlier." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "search = af.Emcee(\n", - " nwalkers=20, # The number of walkers we'll use to sample parameter space.\n", - " nsteps=300, # The number of steps each walker takes, after which 20 * 300 = 6000 steps the non-linear search ends.\n", - ")\n", - "\n", - "print(\n", - " \"\"\"\n", - " The non-linear search has begun running.\n", - " This Jupyter notebook cell with progress once the search has completed - this could take a few minutes!\n", - " \"\"\"\n", - ")\n", - "\n", - "model = af.Collection(galaxies=af.Collection(galaxy=galaxy_model))\n", - "\n", - "result = search.fit(model=model, analysis=analysis)\n", - "\n", - "print(\"The search has finished run - you may now continue the notebook.\")\n", - "\n", - "print(result.info)\n", - "\n", - "aplt.subplot_fit_imaging(fit=result.max_log_likelihood_fit)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The MCMC search succeeded, finding the same high-likelihood model that the MLE search with a good starting point \n", - "identified, even without a good initialization. Its use of multiple walkers exploring parameter space allowed it to \n", - "avoid the local maxima that had trapped the MLE search.\n", - "\n", - "A major advantage of MCMC is that it provides estimates of parameter uncertainties by \"mapping out\" the likelihood \n", - "surface, unlike MLE, which only finds the maximum likelihood solution. These error estimates are accessible in \n", - "the `result.info` string and through the `result.samples` object, which is explained fully in tutorial 5.\n", - "\n", - "While a good starting point wasn't necessary for this simple model, it becomes essential for efficiently mapping the \n", - "likelihood surface in more complex models with many parameters. The code below shows an MCMC fit using a good starting \n", - "point, with two key differences from the MLE initialization:\n", - "\n", - "1. Instead of single starting values, we provide bounds for each parameter. MCMC initializes each walker in a \n", - "small \"ball\" in parameter space, requiring a defined range for each parameter from which values are randomly drawn.\n", - "\n", - "2. We do not specify a starting point for the `ell_comps` parameters, allowing its initial values to be drawn from its \n", - "priors. This illustrates that with MCMC, it\u2019s not necessary to know a good starting point for every parameter." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "initializer = af.InitializerParamBounds(\n", - " {\n", - " model.galaxies.galaxy.bulge.centre.centre_0: (-0.01, 0.01),\n", - " model.galaxies.galaxy.bulge.centre.centre_1: (-0.01, 0.01),\n", - " model.galaxies.galaxy.bulge.effective_radius: (0.0, 2.0),\n", - " model.galaxies.galaxy.bulge.sersic_index: (3.0, 5.0),\n", - " }\n", - ")\n", - "\n", - "search = af.Emcee(\n", - " nwalkers=20, # The number of walkers we'll use to sample parameter space.\n", - " nsteps=300, # The number of steps each walker takes, after which 10 * 200 = 2000 steps the non-linear search ends.\n", - " initializer=initializer,\n", - ")\n", - "\n", - "print(\n", - " \"\"\"\n", - " The non-linear search has begun running.\n", - " This Jupyter notebook cell with progress once the search has completed - this could take a few minutes!\n", - " \"\"\"\n", - ")\n", - "\n", - "model = af.Collection(galaxies=af.Collection(galaxy=galaxy_model))\n", - "\n", - "result = search.fit(model=model, analysis=analysis)\n", - "\n", - "print(\"The search has finished run - you may now continue the notebook.\")\n", - "\n", - "print(result.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "MCMC is a powerful tool for model-fitting, providing accurate parameter estimates and uncertainties. For simple models\n", - "without a starting point, MCMC can still find the correct solution, and if a good starting point is provided, it can\n", - "efficiently scale to more complex models with more parameters.\n", - "\n", - "The main limitation of MCMC is that one has to supply the number of steps the walkers take (`nsteps`). If this value \n", - "is too low, the walkers may not explore the likelihood surface sufficiently. It can be challenging to know the right \n", - "number of steps, especially if models of different complexity are being fitted or if datasets of varying quality are \n", - "used. One often ends up having to perform \"trial and error\" to verify a sufficient number of steps are used.\n", - "\n", - "MCMC can perform badly in parameter spaces with certain types of complexity, for example when there are\n", - "are many local maxima \"peaks\" that many of the walkers can become stuck walking around them, so many walkers that\n", - "when they communicate with one another they cannot gain enough information to move away from the local peak.\n", - "\n", - "__Nested Sampling__\n", - "\n", - "**Nested Sampling** is an advanced method for model-fitting that excels in handling complex models with intricate \n", - "parameter spaces. Here\u2019s a simplified overview of its process:\n", - "\n", - "1. Start with a set of \"live points\" in parameter space, each initialized with random parameter values drawn from their respective priors.\n", - "\n", - "2. Compute the log likelihood for each live point.\n", - "\n", - "3. Draw a new point based on the likelihood of the current live points, favoring regions of higher likelihood.\n", - "\n", - "4. If the new point has a higher likelihood than any existing live point, it becomes a live point, and the lowest likelihood live point is discarded.\n", - "\n", - "This iterative process continues, gradually focusing the live points around higher likelihood regions of parameter \n", - "space until they converge on the highest likelihood solution.\n", - "\n", - "Like MCMC, Nested Sampling effectively maps out parameter space, providing accurate estimates of parameters and \n", - "their uncertainties." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "search = af.Nautilus(\n", - " n_live=100,\n", - " n_batch=50, # Explained in next tutorial\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "To begin the model-fit via the non-linear search, we pass it our model and analysis and begin the fit.\n", - "\n", - "The fit will take a minute or so to run." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(\n", - " \"\"\"\n", - " The non-linear search has begun running.\n", - " This Jupyter notebook cell with progress once the search has completed - this could take a few minutes!\n", - " \"\"\"\n", - ")\n", - "\n", - "model = af.Collection(galaxies=af.Collection(galaxy=galaxy_model))\n", - "\n", - "result = search.fit(model=model, analysis=analysis)\n", - "\n", - "print(\"The search has finished run - you may now continue the notebook.\")\n", - "\n", - "print(result.info)\n", - "\n", - "aplt.subplot_fit_imaging(fit=result.max_log_likelihood_fit)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The **Nested Sampling** search was successful, identifying the same high-likelihood model as the MLE and MCMC searches. \n", - "One of the main benefits of Nested Sampling is its ability to provide accurate parameter estimates and uncertainties, \n", - "similar to MCMC. Additionally, it features a built-in stopping criterion, which eliminates the need for users to \n", - "specify the number of steps the search should take. \n", - "\n", - "This method also excels in handling complex parameter spaces, particularly those with multiple local peaks. This is because\n", - "the live points will identify each peak and converge around them, but then begin to be discarded from a peak if higher\n", - "likelihood points are found elsewhere in parameter space. In MCMC, the walkers can get stuck indefinitely in\n", - "peaks, causing the method to stall.\n", - "\n", - "Another significant advantage is that Nested Sampling estimates an important statistical quantity \n", - "known as \"evidence.\" This value quantifies how well the model fits the data while considering the model's complexity,\n", - "making it essential for Bayesian model comparison, which will be covered in later tutorials. \n", - "\n", - "Nested sampling cannot use a starting point, as it always samples parameter space from scratch by drawing live points\n", - "from the priors. This is both good and bad, depending on if you have access to a good starting point or not. If you do\n", - "not, your MCMC / MLE fit will likely struggle with initialization compared to Nested Sampling. Conversely, if you do \n", - "possess a robust starting point, it can significantly enhance the performance of MCMC, allowing it to begin closer to \n", - "the highest likelihood regions of parameter space. This proximity can lead to faster convergence and more reliable results.\n", - "\n", - "However, Nested Sampling does have limitations; it often scales poorly with increased model complexity. For example, \n", - "once a model has around 50 or more parameters, Nested Sampling can become very slow, whereas MCMC remains efficient \n", - "even in such complex parameter spaces.\n", - "\n", - "__What is The Best Search To Use?__\n", - "\n", - "In statistical inference, the choice of the best search method depends on several factors specific to the problem, \n", - "such as model complexity, availability of a starting point, and whether error estimates are required. More subtle \n", - "aspects, like the shape of the likelihood surface and the presence of multiple peaks, also play a role, as you'll \n", - "explore in later tutorials.\n", - "\n", - "When approaching a model-fitting problem, it's usually advisable to try various search methods to find the most \n", - "effective one. \n", - "\n", - "However, after extensive testing within **PyAutoGalaxy**, a clear recommendation has emerged for fitting galaxy light \n", - "profiles and studying galaxy structure. The nested sampling method `Nautilus` consistently proves to be the most \n", - "effective. It requires fewer iterations than MCMC or MLE methods (when no starting point is used), provides robust \n", - "sampling even for complex models, includes a built-in stopping criterion, and delivers reliable error estimates.\n", - "\n", - "All examples in the `autogalaxy_workspace` use the `Nautilus` search, and future tutorials in the **HowToGalaxy** \n", - "series will also use it. We strongly recommend using `Nautilus` from now on.\n", - "\n", - "That said, MLE and MCMC searches can still be effective, and you're encouraged to experiment with them. If you have \n", - "a reliable starting point, MLE and MCMC can outperform `Nautilus` in terms of speed and efficiency. Additionally, \n", - "for very complex models (e.g., with 30+ parameters), `Nautilus` may struggle, while MCMC might handle the complexity \n", - "better\u2014though you're unlikely to encounter that level of complexity soon.\n", - "\n", - "Lastly, MLE, MCMC, and nested sampling are just three categories of non-linear searches, each containing different \n", - "algorithms with their own strengths and weaknesses. Exploring these options could help you find the best approach \n", - "for your model-fitting needs. For more details on each method, see the [search cookbook](https://pyautofit.readthedocs.io/en/latest/cookbooks/search.html).\n", - "\n", - "__Wrap Up__\n", - "\n", - "This tutorial has laid the foundation with several fundamental concepts in model fitting and statistical inference:\n", - "\n", - "1. **Parameter Space**: This refers to the range of possible values that each parameter in a model can take. It \n", - "defines the dimensions over which the likelihood of different parameter values is evaluated.\n", - "\n", - "2. **Likelihood Function**: This function receives as input a model with specific parameter values and returns a\n", - "log likelihood value that quantifies how well the model describes the data.\n", - "\n", - "3. **Likelihood Surface**: This surface represents how the likelihood of the model varies across the parameter space. \n", - "It helps in identifying the best-fit parameters that maximize the likelihood of the model given the data.\n", - "\n", - "4. **Non-linear Search**: This is an optimization technique used to explore the parameter space and find the \n", - "combination of parameter values that best describe the data. It iteratively adjusts the parameters to maximize the \n", - "likelihood. Many different search algorithms exist, each with their own strengths and weaknesses, and this tutorial\n", - "used the MLE, MCMC, and nested sampling searches.\n", - "\n", - "5. **Priors**: Priors are probabilities assigned to different values of parameters before considering the data. \n", - "They encapsulate our prior knowledge or assumptions about the parameter values. Priors can constrain the parameter \n", - "space, making the search more efficient and realistic.\n", - "\n", - "6. **Model Fitting**: The process of adjusting model parameters to minimize the difference between model predictions \n", - "and observed data, quantified by the likelihood function.\n", - "\n", - "Understanding these concepts is crucial as they form the backbone of model fitting and parameter estimation in \n", - "scientific research and data analysis. In the next tutorials, these concepts will be further expanded upon to \n", - "deepen your understanding and provide more advanced techniques for model fitting and analysis." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_2_modeling/tutorial_2_practicalities.ipynb b/notebooks/howtogalaxy/chapter_2_modeling/tutorial_2_practicalities.ipynb deleted file mode 100644 index b34367e7..00000000 --- a/notebooks/howtogalaxy/chapter_2_modeling/tutorial_2_practicalities.ipynb +++ /dev/null @@ -1,570 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tutorial 2: Practicalities\n", - "==========================\n", - "\n", - "In the last tutorial, we introduced foundational statistical concepts essential for model-fitting, such as parameter\n", - "spaces, likelihoods, priors, and non-linear searches. Understanding these statistical concepts is crucial for\n", - "performing model fits effectively.\n", - "\n", - "However, achieving successful model-fitting also requires practical skills, including how to manage outputs,\n", - "review results, interpret model quality, and ensure run-times are efficient enough for your scientific needs.\n", - "\n", - "This tutorial will focus on these practical aspects of model-fitting, including:\n", - "\n", - "- How to save results to your hard disk.\n", - "- How to navigate the output folder and examine model-fit results to assess quality.\n", - "- How to estimate the run-time of a model-fit before initiating it, and change settings to make it faster or run the analysis in parallel.\n", - "\n", - "__Contents__\n", - "\n", - "This tutorial is split into the following sections:\n", - "\n", - " **PyAutoFit:** The parent package of PyAutoGalaxy, which handles practicalities of model-fitting.\n", - " **Initial Setup:** Load the dataset we'll fit a model to using a non-linear search.\n", - " **Mask:** Apply a mask to the dataset.\n", - " **Model:** Introduce the model we will fit to the data.\n", - " **Search:** Setup the non-linear search, Nautilus, used to fit the model to the data.\n", - " **Search Settings:** Discuss the settings of the non-linear search, including the number of live points.\n", - " **Number Of Cores:** Discuss how to use multiple cores to fit models faster in parallel.\n", - " **Parallel Script:** Running the model-fit in parallel if a bug occurs in a Jupiter notebook.\n", - " **Iterations Per Update:** How often the non-linear search outputs the current results to hard-disk.\n", - " **Analysis:** Create the Analysis object which contains the `log_likelihood_function` that the non-linear search calls.\n", - " **Model-Fit:** Fit the model to the data.\n", - " **Result:** Print the results of the model-fit to the terminal.\n", - " **Output Folder:** Inspect the output folder where results are stored.\n", - " **Unique Identifier:** Discussion of the unique identifier of the model-fit which names the folder in the output directory.\n", - " **Output Folder Contents:** What is output to the output folder (model results, visualization, etc.).\n", - " **Result:** Plot the best-fit model to the data." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autoconf import setup_notebook; setup_notebook()\n", - "\n", - "from pathlib import Path\n", - "import autogalaxy as ag\n", - "import autogalaxy.plot as aplt" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__PyAutoFit__\n", - "\n", - "Modeling uses the probabilistic programming language\n", - "[PyAutoFit](https://github.com/rhayes777/PyAutoFit), an open-source project that allows complex model\n", - "fitting techniques to be straightforwardly integrated into scientific modeling software. \n", - "\n", - "The majority of tools that make model-fitting practical are provided by PyAutoFit, for example it handles\n", - "all output of the non-linear search to hard-disk, the visualization of results and the estimation of run-times." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "import autofit as af" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Initial Setup__\n", - "\n", - "Lets first load the `Imaging` dataset we'll fit a model with using a non-linear search. \n", - "\n", - "This is the same dataset we fitted in the previous tutorial, and we'll repeat the same fit, as we simply want\n", - "to illustrate the practicalities of model-fitting in this tutorial." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_name = \"simple__sersic\"\n", - "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset Auto-Simulation__\n", - "\n", - "If the dataset does not already exist on your system, it will be created by running the corresponding\n", - "simulator script. This ensures that all example scripts can be run without manually simulating data first." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "if not dataset_path.exists():\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.run(\n", - " [sys.executable, \"scripts/howtogalaxy/simulators/sersic.py\"],\n", - " check=True,\n", - " )\n", - "\n", - "\n", - "dataset = ag.Imaging.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " pixel_scales=0.1,\n", - ")\n", - "\n", - "aplt.subplot_imaging_dataset(dataset=dataset)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Mask__\n", - "\n", - "We again apply a 3.0\" mask to the dataset." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "mask = ag.Mask2D.circular(\n", - " shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=3.0\n", - ")\n", - "\n", - "dataset = dataset.apply_mask(mask=mask)\n", - "\n", - "aplt.subplot_imaging_dataset(dataset=dataset)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Over Sampling__\n", - "\n", - "Over sampling is a numerical technique where the images of light profiles and galaxies are evaluated \n", - "on a higher resolution grid than the image data to ensure the calculation is accurate. \n", - "\n", - "For a new user, the details of over-sampling are not important, therefore just be aware that below we make it so that \n", - "all calculations use an adaptive over sampling scheme which ensures high accuracy and precision.\n", - "\n", - "Once you are more experienced, you should read up on over-sampling in more detail via \n", - "the `autogalaxy_workspace/*/guides/over_sampling.ipynb` notebook." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "over_sample_size = ag.util.over_sample.over_sample_size_via_radial_bins_from(\n", - " grid=dataset.grid,\n", - " sub_size_list=[8, 4, 1],\n", - " radial_list=[0.3, 0.6],\n", - " centre_list=[(0.0, 0.0)],\n", - ")\n", - "\n", - "dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size)\n" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Model__\n", - "\n", - "We compose the model using the same API as the previous tutorial.\n", - "\n", - "This model is the same as the previous tutorial, a single `Galaxy` with a `Sersic` light profile." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "galaxy_model = af.Model(ag.Galaxy, redshift=0.5, bulge=ag.lp_linear.Sersic)\n", - "\n", - "model = af.Collection(galaxies=af.Collection(galaxy=galaxy_model))\n", - "\n", - "print(model.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Search__\n", - "\n", - "To fit the model, we now create the non-linear search object, selecting the nested sampling algorithm, Nautilus, \n", - "as recommended in the previous tutorial.\n", - "\n", - "We set up the `Nautilus` object with these parameters:\n", - "\n", - "- **`path_prefix`**: specifies the output directory, here set to `autogalaxy_workspace/output/howtogalaxy/chapter_2`.\n", - " \n", - "- **`name`**: gives the search a descriptive name, which creates the full output path \n", - "as `autogalaxy_workspace/output/howtogalaxy/chapter_2/tutorial_2_practicalities`.\n", - "\n", - "- **`n_live`**: controls the number of live points Nautilus uses to sample parameter space.\n", - "\n", - "__Search Settings__\n", - "\n", - "Nautilus samples parameter space by placing \"live points\" representing different galaxy models. Each point has an \n", - "associated `log_likelihood` that reflects how well it fits the data. By mapping where high-likelihood solutions are \n", - "located, it can focus on searching those regions.\n", - "\n", - "The main setting to balance is the **number of live points**. More live points allow Nautilus to map parameter space \n", - "more thoroughly, increasing accuracy but also runtime. Fewer live points reduce run-time but may make the search less \n", - "reliable, possibly getting stuck in local maxima.\n", - "\n", - "The ideal number of live points depends on model complexity. More parameters generally require more live points, but \n", - "the default of 200 is sufficient for most galaxy models. Lower values can still yield reliable results, particularly \n", - "for simpler models. For this example (6 parameters), we reduce the live points to 80 to speed up runtime without \n", - "compromising accuracy.\n", - "\n", - "Tuning non-linear search settings (e.g., the number of live points) to match model complexity is essential. We aim \n", - "for enough live points to ensure accurate results (i.e., finding a global maximum) but not so many that runtime \n", - "is excessive.\n", - "\n", - "In practice, the optimal number of live points is often found through trial and error, guided by summary statistics \n", - "on how well the search is performing, which we\u2019ll cover below. For this single Sersic model with a linear light \n", - "profile, 80 live points is sufficient to achieve reliable results.\n", - "\n", - "__Iterations Per Update__\n", - "\n", - "Every N iterations, the non-linear search outputs the current results to the folder `autogalaxy_workspace/output`,\n", - "which includes producing visualization. \n", - "\n", - "Depending on how long it takes for the model to be fitted to the data (see discussion about run times below), \n", - "this can take up a large fraction of the run-time of the non-linear search.\n", - "\n", - "For this fit, the fit is very fast, thus we set a high value of `iterations_per_quick_update=10000` to ensure these updates\n", - "so not slow down the overall speed of the model-fit. \n", - "\n", - "**If the iteration per update is too low, the model-fit may be significantly slowed down by the time it takes to\n", - "output results and visualization frequently to hard-disk. If your fit is consistent displaying a log saying that it\n", - "is outputting results, try increasing this value to ensure the model-fit runs efficiently.**" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "search = af.Nautilus(\n", - " path_prefix=Path(\"howtogalaxy\", \"chapter_2\"),\n", - " name=\"tutorial_2_practicalities\",\n", - " unique_tag=dataset_name,\n", - " n_live=80,\n", - " n_batch=50, # GPU lens model fits are batched and run simultaneously, see VRAM section below.\n", - " iterations_per_quick_update=2500,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Analysis__\n", - "\n", - "We again create the `AnalysisImaging` object which contains the `log_likelihood_function` that the non-linear search\n", - "calls to fit the model to the data." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "analysis = ag.AnalysisImaging(dataset=dataset, use_jax=True)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Run Times__\n", - "\n", - "Another key practicality in model-fitting is the run-time of the model-fit. Modeling can be a computationally \n", - "expensive process, whereby the fitting of complex models to high resolution datasets run times may be of order hours, \n", - "days or weeks.\n", - "\n", - "Run times are dictated by two factors:\n", - "\n", - " - **The log likelihood evaluation time:** the time it takes for a single `instance` of the model to be fitted to \n", - " the dataset such that a log likelihood is returned.\n", - "\n", - " - **The number of iterations (e.g. log likelihood evaluations) performed by the non-linear search:** more complex lens\n", - " models require more iterations to converge to a solution (and as discussed above, settings like the number of live\n", - " points also control this).\n", - "\n", - "For this analysis, the log likelihood evaluation time is ~0.05 seconds, which is extremely fast for model fitting. \n", - "\n", - "The more advanced fitting techniques discussed at the end of chapter 1 (e.g. shapelets, multi Gaussian expansions, \n", - "pixelizations) have longer log likelihood evaluation times. However, on GPU, they can be so fast they may not produce\n", - "significantly longer overall run-time feasible.\n", - "\n", - "To estimate the expected overall run time of the model-fit we multiply the log likelihood evaluation time by an \n", - "estimate of the number of iterations the non-linear search will perform, which is around 50000 to 100000 for this model.\n", - "\n", - "Tests thus far reveal CPU run times are around 30 minutes, and GPU run times around 10 minutes.\n", - "\n", - "__Model-Fit__\n", - "\n", - "To begin the model-fit, we pass the model and analysis objects to the search, which performs a non-linear search to \n", - "identify models that best fit the data.\n", - "\n", - "Running model fits via non-linear search can take significant time. While the fit in this tutorial should \n", - "complete in a few minutes, more complex models may require longer run times. In Jupyter notebooks, this can be \n", - "limiting, as the notebook cell will only complete once the fit finishes, preventing you from advancing through the \n", - "tutorial or running additional code.\n", - "\n", - "To work around this, we recommend running tutorial scripts as standalone Python scripts, found \n", - "in `autogalaxy_workspace/scripts/howtogalaxy`. These scripts mirror the notebook tutorials but run independently of \n", - "Jupyter notebooks. For example, you can start a script with:\n", - "\n", - "`python3 scripts/howtogalaxy/chapter_2_modeling/tutorial_2_practicalities.py`\n", - "\n", - "Using scripts allows results to be saved to the hard drive in the `output` folder, enabling you to inspect results \n", - "immediately once the script completes. When rerun, the script loads results directly from disk, so any Jupyter \n", - "notebook cells will quickly load and display the complete model-fit results if they\u2019re already saved.\n", - "\n", - "This approach not only avoids the slowdowns associated with notebook cells during lengthy runs but is also essential \n", - "for using super-computers for fitting tasks, as they require separate Python scripts.\n", - "\n", - "For tasks like loading results, inspecting data, plotting, and interpreting results, Jupyter notebooks remain ideal." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(\n", - " \"The non-linear search has begun running - checkout the autogalaxy_workspace/output/\"\n", - " \" folder for live output of the results, images and model.\"\n", - " \" This Jupyter notebook cell with progress once search has completed - this could take some time!\"\n", - ")\n", - "\n", - "result = search.fit(model=model, analysis=analysis)\n", - "\n", - "print(\"Search has finished run - you may now continue the notebook.\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Result Info__\n", - "\n", - "A concise readable summary of the results is given by printing its `info` attribute.\n", - "\n", - "[Above, we discussed that the `info_whitespace_length` parameter in the config files could b changed to make \n", - "the `model.info` attribute display optimally on your computer. This attribute also controls the whitespace of the\n", - "`result.info` attribute.]" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(result.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Output Folder__\n", - "\n", - "Now checkout the `autogalaxy_workspace/output` folder.\n", - "\n", - "This is where the results of the search are written to hard-disk (in the `tutorial_2_practicalities` folder). \n", - "\n", - "Once completed images, results and information about the fit appear in this folder, meaning that you don't need \n", - "to keep running Python code to see the result.\n", - "\n", - "__Unique Identifier__\n", - "\n", - "In the output folder, you will note that results are in a folder which is a collection of random characters. This acts \n", - "as a `unique_identifier` of the model-fit, where this identifier is generated based on the model, search and dataset \n", - "that are used in the fit.\n", - " \n", - "An identical combination of model, search and dataset generates the same identifier, meaning that rerunning the\n", - "script will use the existing results to resume the model-fit. In contrast, if you change the model, search or dataset,\n", - "a new unique identifier will be generated, ensuring that the model-fit results are output into a separate folder. \n", - "\n", - "We additionally want the unique identifier to be specific to the dataset fitted, so that if we fit different datasets\n", - "with the same model and search results are output to a different folder. We achieved this for the fit above by passing \n", - "the `dataset_name` to the search's `unique_tag`.\n", - "\n", - "__Output Folder Contents__\n", - "\n", - "Now this is running you should checkout the `autogalaxy_workspace/output` folder. This is where the results of the \n", - "search are written to hard-disk (in the `start_here` folder), where all outputs are human readable (e.g. as .json,\n", - ".csv or text files).\n", - "\n", - "As the fit progresses, results are written to the `output` folder on the fly using the highest likelihood model found\n", - "by the non-linear search so far. This means you can inspect the results of the model-fit as it runs, without having to\n", - "wait for the non-linear search to terminate.\n", - " \n", - "The `output` folder includes:\n", - "\n", - " - `model.info`: Summarizes the model, its parameters and their priors discussed in the next tutorial.\n", - " \n", - " - `model.results`: Summarizes the highest likelihood model inferred so far including errors.\n", - " \n", - " - `images`: Visualization of the highest likelihood model-fit to the dataset, (e.g. a fit subplot showing the \n", - " galaxies, model data and residuals).\n", - " \n", - " - `files`: A folder containing .fits files of the dataset, the model as a human-readable .json file, \n", - " a `.csv` table of every non-linear search sample and other files containing information about the model-fit.\n", - " \n", - " - `search.summary`: A file providing summary statistics on the performance of the non-linear search.\n", - " \n", - " - `search_internal`: Internal files of the non-linear search (in this case Nautilus) used for resuming the fit and\n", - " visualizing the search.\n", - "\n", - "__Result__\n", - "\n", - "The `search.fit` method produces a `result` object, packed with useful information about the model fit that we\u2019ll \n", - "explore in detail in a later tutorial.\n", - "\n", - "One component of the `result` object we\u2019ll use now is the `FitImaging` object, which corresponds to the set of model \n", - "parameters that yielded the maximum log-likelihood solution. Plotting this object lets us visually inspect how well \n", - "the model fits the data.\n", - "\n", - "In this example, the fit to the data is excellent, with residuals near zero, as expected since the same model was \n", - "used both to simulate and fit the data." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_fit_imaging(fit=result.max_log_likelihood_fit)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The Probability Density Functions (PDF's) of the results can be plotted using Nautilus's in-built visualization \n", - "library, which is wrapped via the `corner_anesthetic` function.\n", - "\n", - "The PDF shows the 1D and 2D probabilities estimated for every parameter after the model-fit. The two dimensional \n", - "figures can show the degeneracies between different parameters, for example how increasing the intensity $I$ of the\n", - "source galaxy and decreasing its effective radius $R_{Eff}$ lead to similar likelihoods and probabilities.\n", - "\n", - "This PDF will be discussed more in the next tutorial.\n", - "\n", - "The plot is labeled with short hand parameter names (e.g. `sersic_index` is mapped to the short hand \n", - "parameter `n`). These mappings ate specified in the `config/notation.yaml` file and can be customized by users.\n", - "\n", - "The superscripts of labels correspond to the name each component was given in the model (e.g. for the `Isothermal`\n", - "mass its name `mass` defined when making the `Model` above is used)." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "# %%\n", - "'''\n", - "__Other Practicalities__\n", - "\n", - "The following are examples of other practicalities which I will document fully in this example script in the future,\n", - "but so far have no found the time:\n", - "\n", - "- `config`: The files in `autogalaxy_workspace/config` which control many aspects of how PyAutoGalaxy runs,\n", - " including visualization, the non-linear search settings.\n", - "\n", - "- `config/priors`: Folder containing the default priors on all model components.\n", - "\n", - "- `results`: How to load the results of a model-fit from the output folder to a Python script or Jupyter notebook.\n", - "\n", - "- `output.yaml`: What files are output to control file size.\n", - "\n", - "__Wrap Up__\n", - "\n", - "This tutorial has illustrated how to handle a number of practicalities that are key to performing model-fitting\n", - "effectively. These include:\n", - "\n", - "- How to save results to your hard disk.\n", - "\n", - "- How to navigate the output folder and examine model-fit results to assess quality.\n", - "\n", - "- How to estimate the run-time of a model-fit before initiating it, and change settings to make it faster or run the\n", - " analysis in parallel.\n", - "'''" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_2_modeling/tutorial_3_realism_and_complexity.ipynb b/notebooks/howtogalaxy/chapter_2_modeling/tutorial_3_realism_and_complexity.ipynb deleted file mode 100644 index bc89cf67..00000000 --- a/notebooks/howtogalaxy/chapter_2_modeling/tutorial_3_realism_and_complexity.ipynb +++ /dev/null @@ -1,423 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tutorial 3: Realism and Complexity\n", - "==================================\n", - "\n", - "In the previous two tutorials, we fitted a fairly basic model: the galaxy's light was a single bulge component.\n", - "In real observations we know that galaxies are observed to have multiple different morphological structures.\n", - "\n", - "In this tutorial, we'll use a more realistic model, which consists of the following light profiles:\n", - "\n", - " - An `Sersic` light profile for the galaxy's bulge [7 parameters].\n", - " - An `Exponential` light profile for the galaxy's disk [6 parameters]\n", - "\n", - "This model has 13 free parameters, meaning that the parameter space and likelihood function it defines has a\n", - "dimensionality of N=13. This is over double the number of parameters and dimensions of the models we fitted in the\n", - "previous tutorials and in future exercises, we will fit even more complex models with some 13+ parameters.\n", - "\n", - "Therefore, take note, as we make our model more realistic, we also make its parameter space more complex, this is\n", - "an important concept to keep in mind for the remainder of this chapter!\n", - "\n", - "__Contents__\n", - "\n", - "**Initial Setup:** Load the dataset and apply a mask.\n", - "**Model + Search + Analysis:** Compose a bulge+disk model and fit it using Nautilus.\n", - "**Result:** Inspect the result and compare to the true model.\n", - "**Global and Local Maxima:** Demonstrate how the search can infer incorrect local maxima solutions.\n", - "**Wrap Up:** Summary of the challenges of fitting more complex and realistic models." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autoconf import setup_notebook; setup_notebook()\n", - "\n", - "from pathlib import Path\n", - "import autogalaxy as ag\n", - "import autogalaxy.plot as aplt\n", - "import autofit as af" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Initial Setup__\n", - "\n", - "we'll use new galaxying data, where:\n", - "\n", - " - The galaxy's bulge is an `Sersic`.\n", - " - The galaxy's disk is an `Exponential`." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_name = \"simple\"\n", - "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset Auto-Simulation__\n", - "\n", - "If the dataset does not already exist on your system, it will be created by running the corresponding\n", - "simulator script. This ensures that all example scripts can be run without manually simulating data first." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "if not dataset_path.exists():\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.run(\n", - " [sys.executable, \"scripts/imaging/simulator.py\"],\n", - " check=True,\n", - " )\n", - "\n", - "\n", - "dataset = ag.Imaging.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " pixel_scales=0.1,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Mask__\n", - "\n", - "We'll create and use a 2.5\" `Mask2D`, which is slightly smaller than the masks we used in previous tutorials." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "mask = ag.Mask2D.circular(\n", - " shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=2.5\n", - ")\n", - "\n", - "dataset = dataset.apply_mask(mask=mask)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Over Sampling__\n", - "\n", - "Apply adaptive over sampling to ensure the calculation is accurate, you can read up on over-sampling in more detail via \n", - "the `autogalaxy_workspace/*/guides/over_sampling.ipynb` notebook." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "over_sample_size = ag.util.over_sample.over_sample_size_via_radial_bins_from(\n", - " grid=dataset.grid,\n", - " sub_size_list=[8, 4, 1],\n", - " radial_list=[0.3, 0.6],\n", - " centre_list=[(0.0, 0.0)],\n", - ")\n", - "\n", - "dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "When plotted, the galaxy's bulge and disk are clearly visible in the centre of the image." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_imaging_dataset(dataset=dataset)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Model + Search + Analysis__\n", - "\n", - "Now lets fit the dataset using a search." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "model = af.Collection(\n", - " galaxies=af.Collection(\n", - " galaxy=af.Model(\n", - " ag.Galaxy, redshift=0.5, bulge=ag.lp.Sersic, disk=ag.lp.Exponential\n", - " )\n", - " )\n", - ")\n", - "\n", - "search = af.Nautilus(\n", - " path_prefix=Path(\"howtogalaxy\", \"chapter_2\"),\n", - " name=\"tutorial_3_realism_and_complexity\",\n", - " unique_tag=dataset_name,\n", - " n_live=100,\n", - " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - ")\n", - "\n", - "analysis = ag.AnalysisImaging(dataset=dataset, use_jax=True)\n", - "\n", - "print(\n", - " \"The non-linear search has begun running - checkout the autogalaxy_workspace/output/howtogalaxy/chapter_2/tutorial_3_realism_and_complexity\"\n", - " \" folder for live output of the results, images and model.\"\n", - " \" This Jupyter notebook cell with progress once search has completed - this could take some time!\"\n", - ")\n", - "\n", - "result = search.fit(model=model, analysis=analysis)\n", - "\n", - "print(\"Search has finished run - you may now continue the notebook.\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Result__\n", - "\n", - "Inspection of the `info` summary of the result suggests the model has gone to reasonable values." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(result.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "And lets look at how well the model fits the imaging data, which as we are used to fits the data brilliantly!" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_fit_imaging(fit=result.max_log_likelihood_fit)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Global and Local Maxima__\n", - "\n", - "Up to now, all of our non-linear searches have been successes. They find a model that provides a visibly good fit\n", - "to the data, minimizing the residuals and inferring a high log likelihood value. \n", - "\n", - "These solutions are called 'global maxima', they correspond to the highest likelihood regions over all of parameter \n", - "space. There are no other models in parameter space that would give higher likelihoods, this is the model we want \n", - "to always infer!\n", - "\n", - "However, non-linear searches may not always successfully locate the global maxima models. They may instead infer \n", - "a 'local maxima', a solution which has a high log likelihood value relative to the models near it in parameter \n", - "space, but where the log likelihood is significantly below the global maxima solution located somewhere else in \n", - "parameter space. \n", - "\n", - "Why does a non-linear search infer these local maxima solutions? As discussed previously, the search guesses many \n", - "models over and over, guessing more models in regions of parameter space where previous guesses gave the highest \n", - "likelihood solutions. The search gradually 'converges' around any solution that gives a higher likelihood than the \n", - "models nearby it in parameter space. If the search is not thorough enough, it may converge around a solution that \n", - "appears to give a high likelihood (compared to the models around it) but, as discussed, is only a local maxima over \n", - "all of parameter space.\n", - "\n", - "Inferring such solutions is essentially a failure of our non-linear search and it is something we do not want to\n", - "happen! Lets infer a local maxima, by reducing the number of live points, `n_live`, nautilus uses to map out \n", - "parameter space. We are going to use so few that the initial search over parameter space has an extremely low \n", - "probability of getting close the global maxima, meaning it converges on a local maxima. " - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "search = af.Nautilus(\n", - " path_prefix=Path(\"howtogalaxy\", \"chapter_2\"),\n", - " name=\"tutorial_3_realism_and_complexity__local_maxima\",\n", - " unique_tag=dataset_name,\n", - " n_live=50,\n", - " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - ")\n", - "\n", - "print(\n", - " \"The non-linear search has begun running - checkout the autogalaxy_workspace/output/3_realism_and_complexity\"\n", - " \" folder for live output of the results, images and model.\"\n", - " \" This Jupyter notebook cell with progress once search has completed - this could take some time!\"\n", - ")\n", - "\n", - "result_local_maxima = search.fit(model=model, analysis=analysis)\n", - "\n", - "print(\"Search has finished run - you may now continue the notebook.\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Result__\n", - "\n", - "Inspection of the `info` summary of the result suggests certain parameters have gone to different values to the fit\n", - "performed above." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(result_local_maxima.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Lats look at the fit to the `Imaging` data, which is clearly worse than our original fit above." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_fit_imaging(fit=result_local_maxima.max_log_likelihood_fit)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Finally, just to be sure we hit a local maxima, lets compare the maximum log likelihood values of the two results \n", - "\n", - "The local maxima value is significantly lower, confirming that our non-linear search simply failed to locate lens \n", - "models which fit the data better when it searched parameter space." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(\"Likelihood of Global Model:\")\n", - "print(result.max_log_likelihood_fit.log_likelihood)\n", - "print(\"Likelihood of Local Model:\")\n", - "print(result_local_maxima.max_log_likelihood_fit.log_likelihood)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Wrap Up__\n", - "\n", - "In this example, we intentionally made our non-linear search fail, by using so few live points it had no hope of \n", - "sampling parameter space thoroughly. For modeling real galaxies we wouldn't do this intentionally, but the risk of \n", - "inferring a local maxima is still very real, especially as we make our model more complex.\n", - "\n", - "Lets think about *complexity*. As we make our model more realistic, we also made it more complex. For this \n", - "tutorial, our non-linear parameter space went from 7 dimensions to 13. This means there was a much larger *volume* of \n", - "parameter space to search. As this volume grows, there becomes a higher chance that our non-linear search gets lost \n", - "and infers a local maxima, especially if we don't set it up with enough live points!\n", - "\n", - "At its core, modeling is all about learning how to get a non-linear search to find the global maxima region of \n", - "parameter space, even when the model is complex. This will be the main theme throughout the rest of this chapter\n", - "and is the main subject of chapter 3.\n", - "\n", - "In the next exercise, we'll learn how to deal with failure and begin thinking about how we can ensure our non-linear \n", - "search finds the global-maximum log likelihood solution. First, think about the following:\n", - "\n", - " 1) When you look at an image of a galaxy, do you get a sense of roughly what values certain model \n", - " parameters are?\n", - " \n", - " 2) The non-linear search failed because parameter space was too complex. Could we make it less complex, whilst \n", - " still keeping our model fairly realistic?\n", - " \n", - " 3) The galaxy in this example had only 7 non-linear parameters. Real galaxies may have multiple components (e.g. a \n", - " disk, bulge, bar, star-forming knot) and there may even be more than 1 galaxy! Do you think there is any hope of \n", - " us navigating a parameter space if the galaxies contributes 30+ parameters?" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_2_modeling/tutorial_4_dealing_with_failure.ipynb b/notebooks/howtogalaxy/chapter_2_modeling/tutorial_4_dealing_with_failure.ipynb deleted file mode 100644 index fee39441..00000000 --- a/notebooks/howtogalaxy/chapter_2_modeling/tutorial_4_dealing_with_failure.ipynb +++ /dev/null @@ -1,721 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tutorial 4: Dealing With Failure\n", - "================================\n", - "\n", - "In the previous tutorial we intentionally made our non-linear search infer a local maxima solution and therefore return\n", - "a physically incorrect model. In this tutorial, we will pretend that we have modeled our galaxy and inferred a local\n", - "maxima. We introduce three approaches one can take that changes how we fit the model, all of which have the aim of\n", - "ensuring we infer the global maxima:\n", - "\n", - " 1) Prior Tuning: Tell the non-linear search where to search parameter space.\n", - " 2) Reduce Complexity: Fit a model with fewer parameters and therefore a simpler parameter space.\n", - " 3) Look Harder: Brute force a global maxima by telling the non-linear search to sample parameter space more thoroughly.\n", - "\n", - "Each approach has its benefits and disadvantages and we will discuss them in detail.\n", - "\n", - "In the previous tutorial, when we inferred a local maxima we knew that we had done so. For modeling a real galaxy,\n", - "we do not know the true model and it may be unclear if the solution we inferred is a global or local maxima. The\n", - "methods we learn in this tutorial are therefore equally important for verifying that a solution that looks like a\n", - "global maxima solution is in indeed the global maxima.\n", - "\n", - "__Contents__\n", - "\n", - "**Initial Setup:** Load the dataset and apply a mask.\n", - "**Approach 1: Prior Tuning:** Narrow the priors to guide the search to the correct region of parameter space.\n", - "**Approach 2: Reducing Complexity:** Simplify the model to reduce the dimensionality of parameter space.\n", - "**Approach 3: Look Harder:** Increase the thoroughness of the non-linear search sampling." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autoconf import setup_notebook; setup_notebook()\n", - "\n", - "import numpy as np\n", - "from pathlib import Path\n", - "import autogalaxy as ag\n", - "import autogalaxy.plot as aplt\n", - "import autofit as af" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Initial Setup__\n", - "\n", - "we'll use the same galaxy data as the previous tutorial, where:\n", - "\n", - " - The galaxy's bulge is an `Sersic`.\n", - " - The galaxy's disk is an `Exponential`." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_name = \"simple\"\n", - "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset Auto-Simulation__\n", - "\n", - "If the dataset does not already exist on your system, it will be created by running the corresponding\n", - "simulator script. This ensures that all example scripts can be run without manually simulating data first." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "if not dataset_path.exists():\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.run(\n", - " [sys.executable, \"scripts/imaging/simulator.py\"],\n", - " check=True,\n", - " )\n", - "\n", - "\n", - "dataset = ag.Imaging.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " pixel_scales=0.1,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Mask__\n", - "\n", - "we'll create and use a smaller 2.5\" `Mask2D` again." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "mask = ag.Mask2D.circular(\n", - " shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=2.5\n", - ")\n", - "\n", - "dataset = dataset.apply_mask(mask=mask)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Over Sampling__\n", - "\n", - "Apply adaptive over sampling to ensure the calculation is accurate, you can read up on over-sampling in more detail via \n", - "the `autogalaxy_workspace/*/guides/over_sampling.ipynb` notebook." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "over_sample_size = ag.util.over_sample.over_sample_size_via_radial_bins_from(\n", - " grid=dataset.grid,\n", - " sub_size_list=[8, 4, 1],\n", - " radial_list=[0.3, 0.6],\n", - " centre_list=[(0.0, 0.0)],\n", - ")\n", - "\n", - "dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "When plotted, the galaxy's bulge and disk are clearly visible in the centre of the image." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_imaging_dataset(dataset=dataset)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Approach 1: Prior Tuning__\n", - "\n", - "First, we are going to try giving our non-linear search a helping hand. Our priors tell the non-linear search where \n", - "to look in parameter space. If we tell it to look in the right place (that is, 'tune' our priors), this might mean \n", - "the search finds the global solution when it previously found a local maxima.\n", - "\n", - "We saw in a previous tutorial that we can fully customize priors in **PyAutoGalaxy**, so lets give it a go. I've set up \n", - "a custom search below and specified priors that give the non-linear search a better chance of inferring the global \n", - "maxima solution, alongside discussing how I have changed each prior from the default values specified by the \n", - "`config/priors/default` config files.\n", - "\n", - "In a previous tutorial, we customized the priors of a model by creating a `Galaxy` as a `Model` and customizing each\n", - "prior:" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=ag.lp.Sersic, disk=ag.lp.Exponential)\n", - "\n", - "galaxy.bulge.centre_0 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1)\n", - "galaxy.bulge.centre_1 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1)\n", - "galaxy.disk.centre_0 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1)\n", - "galaxy.disk.centre_1 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can alternatively create the light and bulge profiles as a `Model` and customize their parameters, and then pass them\n", - "to the model galaxy and overall model. These two approaches are equivalent, but in this example the style below \n", - "provides more concise and readable code. We will therefore switch to this code style in this tutorial, but may swap \n", - "back and forth between the two styles throughout **HowToGalaxy** depending on what is more readable." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "bulge = af.Model(ag.lp.Sersic)\n", - "disk = af.Model(ag.lp.Exponential)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "By default, the prior on the $(y,x)$ coordinates of a `LightProfile` is a GaussianPrior with \n", - "`mean=0.0` and `sigma=0.3`. However, visual inspection of our galaxy image tells us that its centre (based on the\n", - "galaxy's luminous emission) is at x = 0.0\" and y = 0.0\", so lets reduce the `sigma` value on this prior so the\n", - "non-linear search looks over a very narrow range of `centre` values in parameter space." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "bulge.centre_0 = af.UniformPrior(lower_limit=-0.05, upper_limit=0.05)\n", - "bulge.centre_1 = af.UniformPrior(lower_limit=-0.05, upper_limit=0.05)\n", - "disk.centre_0 = af.UniformPrior(lower_limit=-0.05, upper_limit=0.05)\n", - "disk.centre_1 = af.UniformPrior(lower_limit=-0.05, upper_limit=0.05)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "By default, the elliptical components of the of our galaxy's elliptical `LightProfile` are `TruncatedGaussianPrior`'s \n", - "with `mean=0.0` and `sigma=0.5`. Note that the solution `ell_comps=(0.0, 0.0)` corresponds to a spherical system\n", - "and that all physical solutions (e.g. with axis-ratios running from 0.0 -> 1.0 and position angles 0.0 -> 180.0 degrees) \n", - "are encapsulated for solutions where each component runs from -1.0 -> 1.0). \n", - "\n", - "However, through visual inspection of the image we can often determine the position angle of the galaxy's light, which \n", - "for this data is clearly 45.0 degrees counter-clockwise from the x-axis. We can update the priors on our elliptical \n", - "components to reflect this. The `lower_limit` and `upper_limit` on a `TruncatedGaussianPrior` ensure the solutions cannot go\n", - "outside the physically plausible range -1.0 -> 1.0." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "bulge.ell_comps.ell_comps_0 = af.TruncatedGaussianPrior(\n", - " mean=0.333333, sigma=0.1, lower_limit=-1.0, upper_limit=1.0\n", - ")\n", - "bulge.ell_comps.ell_comps_1 = af.TruncatedGaussianPrior(\n", - " mean=0.0, sigma=0.1, lower_limit=-1.0, upper_limit=1.0\n", - ")\n", - "\n", - "disk.ell_comps.ell_comps_0 = af.TruncatedGaussianPrior(\n", - " mean=0.333333, sigma=0.1, lower_limit=-1.0, upper_limit=1.0\n", - ")\n", - "disk.ell_comps.ell_comps_1 = af.TruncatedGaussianPrior(\n", - " mean=0.0, sigma=0.1, lower_limit=-1.0, upper_limit=1.0\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `effective_radius` of light profile is its 'half-light' radius, the radius at which 50% of its total luminosity \n", - "is internal to a circle defined within that radius. **PyAutoGalaxy** assumes a `UniformPrior` on this quantity between \n", - "0.0\" and 30.0\". This large range of values is required to cover the size of all possible galaxies that can be \n", - "observed in the Universe.\n", - "\n", - "However, inspection of this image shows the galaxy's light does not extend anywhere near 30.0\", so lets reduce its\n", - "value for both bulge and disk components.\n", - "\n", - "We retain an `upper_limit=np.inf`, which means that the maximum physical value that the effective radius \n", - "can go to is infinity, as it is in the real Universe. Equally, the `lower_limit=0.0` ensures unphysical negative\n", - "values are not sampled.\n", - "\n", - "However, the inputs `mean=1.0` and `sigma=0.8` ensure the majority of samples are drawn around much lower values \n", - "between 0.0\" -> 2.0\"." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "bulge.effective_radius = af.TruncatedGaussianPrior(\n", - " mean=1.0, sigma=0.8, lower_limit=0.0, upper_limit=np.inf\n", - ")\n", - "disk.effective_radius = af.TruncatedGaussianPrior(\n", - " mean=1.0, sigma=0.8, lower_limit=0.0, upper_limit=np.inf\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `sersic_index` defines how concentrated the light profile is. In galaxy structure studies, values of Sersic index\n", - "around 1.0 indicate a disk galaxy (which is the value the `Exponential` uses). \n", - "\n", - "Higher values of 3 or 4 indicate an elliptical galaxy. **PyAutoGalaxy** assumes a `UniformPrior` between 0.8 and 8.0 \n", - "by default on this parameter, as a user could model galaxies\n", - "where the galaxy is of any morphology.\n", - "\n", - "We are assuming the `bulge` component is a bulge, thus we can change its prior on the `sersic_index` to a value near 3." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "bulge.sersic_index = af.TruncatedGaussianPrior(\n", - " mean=3.0, sigma=1.0, lower_limit=0.0, upper_limit=np.inf\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We now compose the overall model, where the galaxy model uses the `Model` components above which had their\n", - "priors customizes.\n", - "\n", - "In this exercise, I'm not going to change any priors on the galaxy. Whilst modeling experts can look at a \n", - "galaxy and often tell you roughly where the galaxy is located, it is something of art \n", - "form. Furthermore, the source's morphology can be pretty complex, making it difficult to come up with a good source \n", - "prior!" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=bulge, disk=disk)\n", - "\n", - "model = af.Collection(galaxies=af.Collection(galaxy=galaxy))" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `info` attribute shows the model, including the priors specified above." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(model.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can now create this custom search and run it. Our non-linear search will now start by sampling higher likelihood \n", - "regions of parameter space, given our improved and more informed priors." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "search = af.Nautilus(\n", - " path_prefix=Path(\"howtogalaxy\", \"chapter_2\"),\n", - " name=\"tutorial_4_custom_priors\",\n", - " unique_tag=dataset_name,\n", - " n_live=100,\n", - " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - ")\n", - "\n", - "analysis = ag.AnalysisImaging(dataset=dataset, use_jax=True)\n", - "\n", - "print(\n", - " \"The non-linear search has begun running - checkout the workspace/output/howtogalaxy/chapter_2/tutorial_4_custom_priors\"\n", - " \" folder for live output of the results, images and model.\"\n", - " \" This Jupyter notebook cell with progress once search has completed - this could take some time!\"\n", - ")\n", - "\n", - "result_custom_priors = search.fit(model=model, analysis=analysis)\n", - "\n", - "print(\"Search has finished run - you may now continue the notebook.\")\n", - "\n", - "print(result_custom_priors.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Result__\n", - "\n", - "Bam! We get a good model, which corresponds to the global maxima. By giving our non-linear search a helping hand and \n", - "informing it of where to sample parameter space, we can increase the odds that we find the global maxima solution." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_fit_imaging(fit=result_custom_priors.max_log_likelihood_fit)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Discussion__\n", - "\n", - "By tuning our priors to the galaxy we fit we increase the chance of inferring the global maxima model. The search\n", - "may also fit the model a lot faster, given it spends less time searches regions of parameter space that do not\n", - "correspond to good solutions. \n", - "\n", - "Before moving onto the next approach, lets think about the advantages and disadvantages of prior tuning:\n", - "\n", - "Advantages: \n", - "\n", - " - We have a higher chance of finding the globally maximum log likelihood solutions in parameter space.\n", - " - The search took less time to run because the non-linear search explored less of parameter space.\n", - "\n", - "Disadvantages: \n", - "\n", - " - If we specified a prior incorrectly the non-linear search will infer an incorrect solution.\n", - " - The priors for the search were tailored to the specific galaxy we fitted. If we are fitting multiple galaxies, \n", - " we would have customize the priors for every single fit, for large samples of galaxies this would take a lot of time!\n", - "\n", - "__Approach 2: Reducing Complexity__\n", - "\n", - "The non-linear search may fail because the model is too complex, making its parameter space too difficult to \n", - "sample accurately. Can we can make the model less complex, whilst keeping it realistic enough to perform our \n", - "scientific study? What assumptions can we make to reduce the number of model parameters and therefore \n", - "dimensionality of non-linear parameter space?" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "bulge = af.Model(ag.lp.Sersic)\n", - "disk = af.Model(ag.lp.Exponential)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "First, we create a model that assumes that the bulge and disk are geometrically aligned. That is, the bulge and\n", - "disk centres and elliptical components are perfectly aligned with one another. This may, or may \n", - "not, be a reasonable assumption, but it`ll remove 4 parameters from the model (the centre and elliptical \n", - "components of the bulge profile), so it is worth trying!\n", - "\n", - "To apply our assumption that the bulge and disk are geometrically aligned, we `pair` the `centre` and `ell_comps` \n", - "parameters by setting them equal to one another. This removes the parameter on the left-hand side of the pairing from \n", - "the galaxy model such that when a model is created it has the same value as the parameter on the right-hand side." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "bulge.centre = disk.centre\n", - "bulge.ell_comps = disk.ell_comps" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We now compose the model, which will have a non-linear parameter space with 4 less dimensions than the fit performed\n", - "previously. " - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=bulge, disk=disk)\n", - "\n", - "model = af.Collection(galaxies=af.Collection(galaxy=galaxy))" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `info` attribute shows the model, including the parameter linking specified above." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(model.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We now create this search and run it." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "search = af.Nautilus(\n", - " path_prefix=Path(\"howtogalaxy\", \"chapter_2\"),\n", - " name=\"tutorial_4_reducing_complexity\",\n", - " unique_tag=dataset_name,\n", - " n_live=100,\n", - " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - ")\n", - "\n", - "print(\n", - " \"The non-linear search has begun running - checkout the workspace/output/howtogalaxy/chapter_2/tutorial_4_reducing_complexity\"\n", - " \" folder for live output of the results, images and model.\"\n", - " \" This Jupyter notebook cell with progress once search has completed - this could take some time!\"\n", - ")\n", - "\n", - "result_bulge_disk_align = search.fit(model=model, analysis=analysis)\n", - "\n", - "print(\"Search has finished run - you may now continue the notebook.\")\n", - "\n", - "print(result_bulge_disk_align.info)\n", - "\n", - "aplt.subplot_fit_imaging(fit=result_bulge_disk_align.max_log_likelihood_fit)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Result__\n", - "\n", - "The results look pretty good. Our galaxy fits the data pretty well and we've clearly inferred a model that looks \n", - "similar to the one above. However, inspection of the residuals shows that the fit was not quite as good as the \n", - "first search.\n", - "\n", - "It turns out that for this simulation, the bulge and disk had different elliptical components. The quality of the fit \n", - "suffered and the highest value of log likelihood for the search inferred was lower as a result.\n", - "\n", - "Herein lies the pitfalls of making assumptions, they may make your model less realistic and your fits worse! \n", - "\n", - "__Discussion__\n", - "\n", - "Again, lets consider the advantages and disadvantages of this approach:\n", - "\n", - "Advantages:\n", - "\n", - " - By reducing parameter space`s complexity we again had a higher chance of inferring the global maximum log \n", - " likelihood and the time required by the search to do this is reducing.\n", - " - Unlike tuned priors, the search was not specific to one galaxy and we could run it on many galaxy images.\n", - " \n", - "Disadvantages:\n", - "\n", - " - Our model was less realistic and our fit suffered as a result.\n", - "\n", - "__Approach 3: Look Harder__\n", - "\n", - "In approaches 1 and 2 we extended our non-linear search an olive branch and helped it find the highest log likelihood \n", - "regions of parameter space. In approach 3 ,we're going to tell it to just `look harder`.\n", - "\n", - "Every non-linear search has settings which govern how thoroughly it searches parameter space, with the number of live\n", - "points that was passed to `Nautilus` an example of such a setting. The more thoroughly the search looks, the more likely \n", - "it is that it`ll find the global maximum model. However, the search will also take longer!\n", - "\n", - "We create a more thorough `nautilus` search, that uses `n_live=200`. What these settings\n", - "are actually changing is discussed in the optional tutorial `HowToGalaxy/chapter_optional/tutorial_searches.ipynb`.\n", - "\n", - "Due to the long run times of this search, we comment it output below so it does not run. Feel free to undo these\n", - "comments so the script runs faster." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=ag.lp.Sersic, disk=ag.lp.Exponential)\n", - "\n", - "model = af.Collection(galaxies=af.Collection(galaxy=galaxy))\n", - "\n", - "search = af.Nautilus(\n", - " path_prefix=Path(\"howtogalaxy\", \"chapter_2\"),\n", - " name=\"tutorial_4_look_harder\",\n", - " unique_tag=dataset_name,\n", - " n_live=200,\n", - " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - ")\n", - "\n", - "print(\n", - " \"The non-linear search has begun running - checkout the workspace/output/howtogalaxy/chapter_2/tutorial_4_look_harder\"\n", - " \" folder for live output of the results, images and model.\"\n", - " \" This Jupyter notebook cell with progress once search has completed - this could take some time!\"\n", - ")\n", - "\n", - "# result_look_harder = search.fit(model=model, analysis=analysis)\n", - "\n", - "print(\"Search has finished run - you may now continue the notebook.\")\n", - "\n", - "# print(result_look_harder.info)\n", - "\n", - "# aplt.subplot_fit_imaging(fit=result_look_harder.max_log_likelihood_fit)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "lets list the advantages and disadvantages of simply adjusting the non-linear search:\n", - "\n", - "Advantages:\n", - "\n", - " - Its easy to setup, we simply change settings of the non-linear search.\n", - " \n", - " - It generalizes to any galaxy.\n", - " \n", - " - We can make our model as complex as we want.\n", - "\n", - "Disadvantage:\n", - " \n", - " - Its potentially expensive. Very expensive. For very complex models, the run times can hours, days, weeks or, dare \n", - " I say it, months!\n", - "\n", - "So, we can now fit galaxies. And when it fails, we know how to get it to work. \n", - "\n", - "In chapter 3 of **HowToGalaxy**, we will introduce a technique called 'non-linear search chaining', which performs a \n", - "model fit by chaining together multiple searches back-to-back . This allows us to combine the 3 different approaches \n", - "discussed and exploit the advantages of each, whilst not being hindered by their disadvantages.\n", - "\n", - "With search chaining, we can:\n", - "\n", - " - Fit simpler models with lower dimensionality parameter spaces in the earlier searches and gradually increase the\n", - " model complexity search-by-search, guiding the model-fit to a sufficiently realistic model. \n", - " \n", - " - In these earlier searches (with easier to sample parameter spaces), use fast non-linear search settings to compute \n", - " the results quickly and switch to slower settings in later searches when we fit more complex models.\n", - "\n", - " - Use 'prior passing' to setup the priors of each parameter in the later searches, based on the models inferred \n", - " by the earlier searches. We can therefore guide each search on how to sample a complex model's parameter space \n", - " in a way that can be fully generalized to any galaxy.\n", - " \n", - "To wrap up chapter 2, we have a few more tutorials, where we will discuss masking in more detail, the `Result` object\n", - "and how to make **PyAutoGalaxy** run faster." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_2_modeling/tutorial_5_linear_profiles.ipynb b/notebooks/howtogalaxy/chapter_2_modeling/tutorial_5_linear_profiles.ipynb deleted file mode 100644 index 8ebc0a73..00000000 --- a/notebooks/howtogalaxy/chapter_2_modeling/tutorial_5_linear_profiles.ipynb +++ /dev/null @@ -1,845 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tutorial 5: Linear Profiles\n", - "===========================\n", - "\n", - "In the previous tutorial we learned how to balance model complexity with our non-linear search in order to infer\n", - "accurate model solutions and avoid failure. We saw how in order to fit a model accurately one may have to\n", - "parameterize and fit a simpler model with fewer non-linear parameters, at the expense of fitting the data less\n", - "accurately.\n", - "\n", - "It would be desirable if we could make our model have more flexibility enabling it to fit more complex galaxy\n", - "structures, but in a way that does not increase (or perhaps even decreases) the number of non-linear parameters.\n", - "This would keep the `nautilus` model-fit efficient and accurate.\n", - "\n", - "This is possible using linear light profiles, which solve for their `intensity` parameter via efficient linear\n", - "algebra, using a process called an inversion. The inversion always computes `intensity` values that give the best\n", - "fit to the data (e.g. they minimize the chi-squared and therefore maximize the likelihood).\n", - "\n", - "This tutorial will first fit a model using two linear light profiles. Because their `intensity` values are solved for\n", - "implicitly, this means they are not a dimension of the non-linear parameter space fitted by `nautilus`, therefore\n", - "reducing the complexity of parameter space and making the fit faster and more accurate.\n", - "\n", - "This tutorial will then show how many linear light profiles can be combined into a `Basis`, which comes from the term\n", - "'basis function'. By combining many linear light profiles models can be composed which are able to fit complex galaxy\n", - "structures (e.g. asymmetries, twists) with just N=6-8 non-linear parameters.\n", - "\n", - "__Contents__\n", - "\n", - "**Initial Setup:** Load the dataset and apply a mask.\n", - "**Linear Light Profiles:** Fit a model using linear light profiles that solve for intensity via linear algebra.\n", - "**Run Time:** Estimate the run time of the model-fit before starting.\n", - "**Result:** Inspect the result and the solved intensity values.\n", - "**Intensities:** Access the intensity values of linear light profiles after the fit.\n", - "**Visualization:** Visualize the fit using linear light profiles.\n", - "**Basis:** Combine many linear light profiles into a basis for fitting complex structures.\n", - "**Model Fit:** Fit the basis model to the data.\n", - "**Disk MGE:** Use a Multi-Gaussian Expansion for the disk component.\n", - "**Multi Gaussian Expansion Benefits:** Discussion of the advantages of MGE models.\n", - "**Positive Only Solver:** Ensure linear algebra solutions have positive intensity values.\n", - "**Other Basis Functions:** Overview of other basis functions like shapelets.\n", - "**Wrap Up:** Summary of linear light profiles and basis functions." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autoconf import setup_notebook; setup_notebook()\n", - "\n", - "import numpy as np\n", - "from pathlib import Path\n", - "import autogalaxy as ag\n", - "import autogalaxy.plot as aplt\n", - "import autofit as af" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Initial Setup__\n", - "\n", - "we'll use the same galaxy data as the previous tutorial, where:\n", - "\n", - " - The galaxy's bulge is an `Sersic`.\n", - " - The galaxy's disk is an `Exponential`." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_name = \"simple\"\n", - "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset Auto-Simulation__\n", - "\n", - "If the dataset does not already exist on your system, it will be created by running the corresponding\n", - "simulator script. This ensures that all example scripts can be run without manually simulating data first." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "if not dataset_path.exists():\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.run(\n", - " [sys.executable, \"scripts/imaging/simulator.py\"],\n", - " check=True,\n", - " )\n", - "\n", - "\n", - "dataset = ag.Imaging.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " pixel_scales=0.1,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Mask__\n", - "\n", - "we'll create and use a smaller 2.5\" `Mask2D` again." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "mask = ag.Mask2D.circular(\n", - " shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=2.5\n", - ")\n", - "\n", - "dataset = dataset.apply_mask(mask=mask)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Over Sampling__\n", - "\n", - "Apply adaptive over sampling to ensure the calculation is accurate, you can read up on over-sampling in more detail via \n", - "the `autogalaxy_workspace/*/guides/over_sampling.ipynb` notebook." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "over_sample_size = ag.util.over_sample.over_sample_size_via_radial_bins_from(\n", - " grid=dataset.grid,\n", - " sub_size_list=[8, 4, 1],\n", - " radial_list=[0.3, 0.6],\n", - " centre_list=[(0.0, 0.0)],\n", - ")\n", - "\n", - "dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "When plotted, the galaxy's bulge and disk are clearly visible in the centre of the image." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_imaging_dataset(dataset=dataset)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Linear Light Profiles__\n", - "\n", - "First, we use a variant of a light profile discussed called a \"linear light profile\", which is accessed via the\n", - "command `ag.lp_linear`. \n", - " \n", - "The `intensity` values of linear light profiles are solved for via linear algebra. We use the `Sersic` \n", - "and `Exponential` linear light profiles, which are identical to the ordinary `Sersic` and `Exponential` \n", - "profiles fitted in previous tutorials, except for their `intensity` parameter now being solved for implicitly.\n", - "\n", - "Because the `intensity` parameter of each light profile is not a free parameter in the model-fit, the dimensionality of \n", - "non-linear parameter space is reduced by 1 for each light profile (in this example, 2). This also removes the \n", - "degeneracies between the `intensity` and other light profile parameters (e.g. `effective_radius`, `sersic_index`), \n", - "making the model-fit more robust.\n", - "\n", - "This is a rare example where we are able to reduce the complexity of parameter space without making the model itself \n", - "any simpler. There is really no downside to using linear light profiles, so I would recommend you adopt them as \n", - "standard for your own model-fits from here on!" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "bulge = af.Model(ag.lp_linear.Sersic)\n", - "disk = af.Model(ag.lp_linear.Exponential)\n", - "\n", - "bulge.centre = disk.centre\n", - "\n", - "galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=bulge, disk=disk)\n", - "\n", - "model = af.Collection(galaxies=af.Collection(galaxy=galaxy))" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `info` attribute shows the model, including the linear light profiles.\n", - "\n", - "Note how the `intensity` is no longer listed and does not have a prior associated with it." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(model.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We now create this search and run it." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "search = af.Nautilus(\n", - " path_prefix=Path(\"howtogalaxy\", \"chapter_2\"),\n", - " name=\"tutorial_5_linear_light_profile\",\n", - " unique_tag=dataset_name,\n", - " n_live=100,\n", - " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - ")\n", - "\n", - "analysis = ag.AnalysisImaging(dataset=dataset, use_jax=True)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Run Time__\n", - "\n", - "For standard light profiles, the log likelihood evaluation time is of order ~0.01 seconds for this dataset.\n", - "\n", - "For linear light profiles, the log likelihood evaluation increases to around ~0.05 seconds per likelihood evaluation.\n", - "This is still fast, but it does mean that the fit may take around five times longer to run.\n", - "\n", - "However, because two free parameters have been removed from the model (the `intensity` of the lens bulge and \n", - "source bulge), the total number of likelihood evaluations will reduce. Furthermore, the simpler parameter space\n", - "likely means that the fit will take less than 10000 per free parameter to converge. This is aided further\n", - "by the reduction in `n_live` to 100.\n", - "\n", - "Fits using standard light profiles and linear light profiles therefore take roughly the same time to run. However,\n", - "the simpler parameter space of linear light profiles means that the model-fit is more reliable, less susceptible to\n", - "converging to an incorrect solution and scales better if even more light profiles are included in the model.\n", - "\n", - "Run the non-linear search." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(\n", - " \"The non-linear search has begun running - checkout the workspace/output/howtogalaxy/chapter_2/tutorial_5_linear_light_profile\"\n", - " \" folder for live output of the results, images and model.\"\n", - " \" This Jupyter notebook cell with progress once search has completed - this could take some time!\"\n", - ")\n", - "\n", - "result_linear_light_profile = search.fit(model=model, analysis=analysis)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Result__\n", - "\n", - "The `info` attribute shows the resulting model, which does not display the `intensity` values for each light profile." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(result_linear_light_profile.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Intensities__\n", - "\n", - "The intensities of linear light profiles are not a part of the model parameterization and therefore are not displayed\n", - "in the `model.results` file.\n", - "\n", - "To extract the `intensity` values of a specific component in the model, we use the `max_log_likelihood_galaxies`,\n", - "which has already performed the inversion and therefore the galaxy light profiles have their solved for\n", - "`intensity`'s associated with them." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "galaxies = result_linear_light_profile.max_log_likelihood_galaxies\n", - "\n", - "print(galaxies[0].bulge.intensity)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `Galaxies` contained in the `max_log_likelihood_fit` also has the solved for `intensity` values:" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "fit = result_linear_light_profile.max_log_likelihood_fit\n", - "\n", - "galaxies = fit.galaxies\n", - "\n", - "print(galaxies[0].bulge.intensity)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Visualization__\n", - "\n", - "Linear light profiles and objects containing them (e.g. galaxies) cannot be plotted because they do not \n", - "have an `intensity` value.\n", - "\n", - "Therefore, the objects created above which replaces all linear light profiles with ordinary light profiles must be\n", - "used for visualization:" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "galaxies = result_linear_light_profile.max_log_likelihood_galaxies\n", - "\n", - "aplt.plot_array(array=galaxies.image_2d_from(grid=dataset.grid), title=\"Image\")\n", - "\n", - "aplt.plot_array(array=galaxies[0].image_2d_from(grid=dataset.grid), title=\"Image\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Basis__\n", - "\n", - "We can use many linear light profiles to build a `Basis`. \n", - "\n", - "For example, below, we make a `Basis` out of 30 elliptical Gaussian linear light profiles which: \n", - "\n", - " - All share the same centre and elliptical components.\n", - " - The `sigma` size of the Gaussians increases in log10 increments.\n", - " \n", - "Because `log10(1.0) = 0.0` the first Gaussian `sigma` value is therefore 0.0001, whereas because `log10(10) = 1.0`\n", - "the size of the final Gaussian is 1.0. \n", - "\n", - "The equation below has therefore been chosen to provide intuition on the scale of the Gaussians." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "total_gaussians = 30\n", - "\n", - "# The sigma values of the Gaussians will be fixed to values spanning 0.01 to the mask radius, 3.0\".\n", - "mask_radius = 3.0\n", - "log10_sigma_list = np.linspace(-2, np.log10(mask_radius), total_gaussians)\n", - "\n", - "# By defining the centre here, it creates two free parameters that are assigned below to all Gaussians.\n", - "\n", - "centre_0 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1)\n", - "centre_1 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1)\n", - "\n", - "bulge_gaussian_list = []\n", - "\n", - "# A list of Gaussian model components whose parameters are customized belows.\n", - "\n", - "gaussian_list = af.Collection(\n", - " af.Model(ag.lp_linear.Gaussian) for _ in range(total_gaussians)\n", - ")\n", - "\n", - "# Iterate over every Gaussian and customize its parameters.\n", - "\n", - "for i, gaussian in enumerate(gaussian_list):\n", - " gaussian.centre.centre_0 = centre_0 # All Gaussians have same y centre.\n", - " gaussian.centre.centre_1 = centre_1 # All Gaussians have same x centre.\n", - " gaussian.ell_comps = gaussian_list[\n", - " 0\n", - " ].ell_comps # All Gaussians have same elliptical components.\n", - " gaussian.sigma = (\n", - " 10 ** log10_sigma_list[i]\n", - " ) # All Gaussian sigmas are fixed to values above.\n", - "\n", - "bulge_gaussian_list += gaussian_list\n", - "\n", - "# The Basis object groups many light profiles together into a single model component.\n", - "\n", - "bulge = af.Model(\n", - " ag.lp_basis.Basis,\n", - " profile_list=bulge_gaussian_list,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "One we have a `Basis`, we can treat it like any other light profile in order to create a `Galaxy` and `Galaxies` and \n", - "use it to fit data." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "galaxy = ag.Galaxy(redshift=0.5, bulge=bulge)\n", - "\n", - "galaxies = ag.Galaxies(galaxies=[galaxy])\n", - "\n", - "fit = ag.FitImaging(dataset=dataset, galaxies=galaxies)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "By plotting the fit, we see that the `Basis` does a reasonable job at capturing the appearance of the lens galaxy.\n", - "\n", - "There are imperfections, but this is because we did not fit the model via a non-linear search in order to determine\n", - "the optimal values of the Gaussians in the basis. In particular, the Gaussians above were all spherical, when the\n", - "lens galaxy is elliptical. \n", - "\n", - "We rectify this below, where we use a non-linear search to determine the optimal values of the Gaussians!" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_fit_imaging(fit=fit)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Model Fit__\n", - "\n", - "To fit a model using `Basis` functions, the API is very similar to that shown throughout this chapter, using both\n", - "the `af.Model()` and `af.Collection()` objects.\n", - "\n", - "In this example we fit a `Basis` model for the bulge where:\n", - "\n", - " - The bulge is a superposition of 30 parametric linear `Gaussian` profiles [4 parameters]. \n", - " - The centres and elliptical components of each family of Gaussians are all linked together.\n", - " - The `sigma` size of the Gaussians increases in log10 increments.\n", - "\n", - "The number of free parameters and therefore the dimensionality of the MGe is just N=4." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "total_gaussians = 30\n", - "\n", - "# The sigma values of the Gaussians will be fixed to values spanning 0.01 to the mask radius, 3.0\".\n", - "mask_radius = 3.0\n", - "log10_sigma_list = np.linspace(-2, np.log10(mask_radius), total_gaussians)\n", - "\n", - "# By defining the centre here, it creates two free parameters that are assigned below to all Gaussians.\n", - "\n", - "centre_0 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1)\n", - "centre_1 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1)\n", - "\n", - "bulge_gaussian_list = []\n", - "\n", - "# A list of Gaussian model components whose parameters are customized belows.\n", - "\n", - "gaussian_list = af.Collection(\n", - " af.Model(ag.lp_linear.Gaussian) for _ in range(total_gaussians)\n", - ")\n", - "\n", - "# Iterate over every Gaussian and customize its parameters.\n", - "\n", - "for i, gaussian in enumerate(gaussian_list):\n", - " gaussian.centre.centre_0 = centre_0 # All Gaussians have same y centre.\n", - " gaussian.centre.centre_1 = centre_1 # All Gaussians have same x centre.\n", - " gaussian.ell_comps = gaussian_list[\n", - " 0\n", - " ].ell_comps # All Gaussians have same elliptical components.\n", - " gaussian.sigma = (\n", - " 10 ** log10_sigma_list[i]\n", - " ) # All Gaussian sigmas are fixed to values above.\n", - "\n", - "bulge_gaussian_list += gaussian_list\n", - "\n", - "# The Basis object groups many light profiles together into a single model component.\n", - "\n", - "bulge = af.Model(\n", - " ag.lp_basis.Basis,\n", - " profile_list=bulge_gaussian_list,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Disk MGE__\n", - "\n", - "The residuals of the fit above showed us that the galaxy in the data is composed of multiple structures (e.g. a bulge\n", - "and disk) which have distinct elliptical coordinates.\n", - "\n", - "We therefore compose a second `Basis` of 10 Gaussians to represent the `disk`. This is parameterized the same as\n", - "the `bulge` (e.g. all Gaussians share the same `centre` and `ell_comps`) but is treated as a completely\n", - "independent set of parameters." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "total_gaussians = 10\n", - "\n", - "# The sigma values of the Gaussians will be fixed to values spanning 0.01 to the mask radius, 3.0\".\n", - "mask_radius = 3.0\n", - "log10_sigma_list = np.linspace(-2, np.log10(mask_radius), total_gaussians)\n", - "\n", - "disk_gaussian_list = []\n", - "\n", - "# A list of Gaussian model components whose parameters are customized belows.\n", - "\n", - "gaussian_list = af.Collection(\n", - " af.Model(ag.lp_linear.Gaussian) for _ in range(total_gaussians)\n", - ")\n", - "\n", - "# Iterate over every Gaussian and customize its parameters.\n", - "\n", - "for i, gaussian in enumerate(gaussian_list):\n", - " gaussian.centre.centre_0 = bulge_gaussian_list[\n", - " 0\n", - " ].centre_0 # All Gaussians have same y centre as bulge.\n", - " gaussian.centre.centre_1 = bulge_gaussian_list[\n", - " 0\n", - " ].centre_1 # All Gaussians have same x centre as bulge.\n", - " gaussian.ell_comps = gaussian_list[\n", - " 0\n", - " ].ell_comps # All Gaussians have same elliptical components.\n", - " gaussian.sigma = (\n", - " 10 ** log10_sigma_list[i]\n", - " ) # All Gaussian sigmas are fixed to values above.\n", - "\n", - "disk_gaussian_list += gaussian_list\n", - "\n", - "# The Basis object groups many light profiles together into a single model component.\n", - "\n", - "disk = af.Model(\n", - " ag.lp_basis.Basis,\n", - " profile_list=disk_gaussian_list,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We now compose the overall model which uses both sets of Gaussians to represent separately the bulge and disk.\n", - "\n", - "The overall dimensionality of non-linear parameter space is just N=6, which is fairly remarkable if you\n", - "think about just how complex the structures are that these two `Basis` of Gaussians can capture!" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=bulge, disk=disk)\n", - "\n", - "model = af.Collection(galaxies=af.Collection(galaxy=galaxy))\n" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `info` attribute shows the model, which is a lot longer than we have seen previously, given that is \n", - "composed of many Gaussians!" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(model.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We now fit the model, with just `n_live=50` given the simiplicity of parameter space." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "search = af.Nautilus(\n", - " path_prefix=Path(\"howtogalaxy\", \"chapter_2\"),\n", - " name=\"tutorial_5_basis\",\n", - " unique_tag=dataset_name,\n", - " n_live=50,\n", - " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - ")\n", - "\n", - "print(\n", - " \"The non-linear search has begun running - checkout the workspace/output/howtogalaxy/chapter_2/tutorial_5_basis\"\n", - " \" folder for live output of the results, images and model.\"\n", - " \" This Jupyter notebook cell with progress once search has completed - this could take some time!\"\n", - ")\n", - "\n", - "result_basis = search.fit(model=model, analysis=analysis)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Result__\n", - "\n", - "The result `info` attribute shows the result, which is again longer than usual given the large number of Gaussians\n", - "used in the fit." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(result_basis.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Visualizing the fit shows that we successfully fit the data to the noise level.\n", - "\n", - "Note that the result objects `max_log_likelihood_galaxies` and `max_log_likelihood_fit` automatically convert\n", - "all linear light profiles to ordinary light profiles, including every single one of the 20 Gaussians fitted\n", - "above. \n", - "\n", - "This means we can use them directly to perform the visualization below." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(result_basis.max_log_likelihood_instance)\n", - "\n", - "aplt.subplot_galaxies(\n", - " galaxies=result_basis.max_log_likelihood_galaxies, grid=result_basis.grids.lp\n", - ")\n", - "\n", - "aplt.subplot_fit_imaging(fit=result_basis.max_log_likelihood_fit)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Multi Gaussian Expansion Benefits__\n", - "\n", - "Symmetric light profiles (e.g. elliptical Sersics) may leave significant residuals, because they fail to capture\n", - "irregular and asymmetric morphological of galaxies (e.g. isophotal twists, an ellipticity which varies radially).\n", - "An MGE fully captures these features and can therefore much better represent the emission of complex galaxies.\n", - "\n", - "The MGE model can be composed in a way that has fewer non-linear parameters than an elliptical Sersic. In this example,\n", - "a groups of Gaussians is used to represent the `bulge` of the galaxy, which in total correspond to just N=4 non-linear \n", - "parameters (a `bulge` and `disk` comprising two linear Sersics has N=10 parameters).\n", - "\n", - "The MGE model parameterization is also composed such that neither the `intensity` parameters or any of the\n", - "parameters controlling the size of the Gaussians (their `sigma` values) are non-linear parameters sampled by Nautilus.\n", - "This removes the most significant degeneracies in parameter space, making the model much more reliable and efficient\n", - "to fit.\n", - "\n", - "Therefore, not only does an MGE fit more complex galaxy morphologies, it does so using fewer non-linear parameters\n", - "in a much simpler non-linear parameter space which has far less significant parameter degeneracies!\n", - "\n", - "__Positive Only Solver__\n", - "\n", - "Many codes which use linear algebra typically rely on a linear algabra solver which allows for positive and negative\n", - "values of the solution (e.g. `np.linalg.solve`), because they are computationally fast. \n", - "\n", - "This is problematic, as it means that negative surface brightnesses values can be computed to represent a galaxy's \n", - "light, which is clearly unphysical. For an MGE, this produces a positive-negative \"ringing\", where the\n", - "Gaussians alternate between large positive and negative values. This is clearly undesirable and unphysical.\n", - "\n", - "**PyAutoGalaxy** (and therefore all examples above) uses a positive only linear algebra solver which has been \n", - "extensively optimized to ensure it is as fast as positive-negative solvers. This ensures that all light profile \n", - "intensities are positive and therefore physical. \n", - "\n", - "__Other Basis Functions__\n", - "\n", - "In addition to the Gaussians used in this example, there is another basis function implemented in PyAutoGalaxy \n", - "that is commonly used to represent the light of galaxies, called a `Shapelet`. \n", - "\n", - "Shapelets are basis functions with analytic properties that are appropriate for capturing the exponential / disk-like \n", - "features of a galaxy. They do so over a wide range of scales, and can often represent features in source galaxies \n", - "that a single Sersic function or MGE cannot.\n", - "\n", - "An example using shapelets is given at `autogalaxy_workspace/scripts/modeling/imaging/features/shapelets.py`.\n", - " \n", - "Feel free to experiment with using shapelets as the galaxy by yourself. However they incur higher computational \n", - "overheads than the MGE and include a free parameter which governs the size of the basis functions and therefore source,\n", - "slowing down convergence of the non-linear search. We have found that MGEs perform better than shapelets in most \n", - "lens modeling problems. \n", - "\n", - "If you have a desire to fit sources with even more complex morphologies we recommend you look at how to reconstruct \n", - "sources using pixelizations in the `modeling/features` section or chapter 4 of **HowToGalaxy**.\n", - "\n", - "__Wrap Up__\n", - "\n", - "In this tutorial we described how linearizing light profiles allows us to fit more complex light profiles to\n", - "galaxies using fewer non-linear parameters, keeping the fit performed by the non-linear search fast, accurate\n", - "and robust.\n", - "\n", - "Perhaps the biggest downside to basis functions is that they are only as good as the features they can capture\n", - "in the data. For example, a baiss of Gaussians still assumes that they have a well defined centre, but there are\n", - "galaxies which may have multiple components with multiple centres (e.g. many star forming knots) which such a \n", - "basis cannot catprue.\n", - "\n", - "In chapter 4 of **HowToGalaxy** we introduce non-parametric pixelizations, which reconstruct the data in way\n", - "that does not make assumptions like a centre and can thus reconstruct even more complex, asymmetric and irregular\n", - "galaxy morphologies." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "basis = ag.lp_basis.Basis(\n", - " profile_list=gaussian_list, regularization=ag.reg.Constant(coefficient=1.0)\n", - ")\n" - ], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_2_modeling/tutorial_6_masking.ipynb b/notebooks/howtogalaxy/chapter_2_modeling/tutorial_6_masking.ipynb deleted file mode 100644 index 09c68cb7..00000000 --- a/notebooks/howtogalaxy/chapter_2_modeling/tutorial_6_masking.ipynb +++ /dev/null @@ -1,257 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tutorial 6: Masking\n", - "===================\n", - "\n", - "We have learnt everything we need to know about non-linear searches to model a galaxy and infer a good lens\n", - "model solution. Now, lets consider masking in more detail, something we have not given much consideration previously.\n", - "We'll also learn a neat trick to improve the speed and accuracy of a non-linear search.\n", - "\n", - "__Contents__\n", - "\n", - "**Initial Setup:** Load the dataset and apply a mask.\n", - "**Mask:** Apply a custom mask to focus the fit on specific regions.\n", - "**Model + Search + Analysis:** Fit the model using the custom mask.\n", - "**Discussion:** How the mask affects the fit quality and run time.\n", - "**Wrap Up:** Summary of masking strategies for galaxy modeling." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autoconf import setup_notebook; setup_notebook()\n", - "\n", - "from pathlib import Path\n", - "import autogalaxy as ag\n", - "import autogalaxy.plot as aplt\n", - "import autofit as af" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Initial Setup__\n", - "\n", - "we'll use the same galaxy data as tutorials 1 & 2, where:\n", - "\n", - " - The galaxy's `LightProfile` is an `Sersic`." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_name = \"simple__sersic\"\n", - "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset Auto-Simulation__\n", - "\n", - "If the dataset does not already exist on your system, it will be created by running the corresponding\n", - "simulator script. This ensures that all example scripts can be run without manually simulating data first." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "if not dataset_path.exists():\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.run(\n", - " [sys.executable, \"scripts/howtogalaxy/simulators/sersic.py\"],\n", - " check=True,\n", - " )\n", - "\n", - "\n", - "dataset = ag.Imaging.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " pixel_scales=0.1,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Mask__\n", - "\n", - "In tutorials 1 and 2 we used a 3.0\" circular mask. \n", - "\n", - "However, there is very faint flux emitted at the outskirts of the galaxy, which the model will benefit from fitting\n", - "by using a larger mask." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "mask = ag.Mask2D.circular(\n", - " shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=6.0\n", - ")\n", - "\n", - "dataset = dataset.apply_mask(mask=mask)\n", - "\n", - "aplt.subplot_imaging_dataset(dataset=dataset)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Over Sampling__\n", - "\n", - "Apply adaptive over sampling to ensure the calculation is accurate, you can read up on over-sampling in more detail via \n", - "the `autogalaxy_workspace/*/guides/over_sampling.ipynb` notebook." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "over_sample_size = ag.util.over_sample.over_sample_size_via_radial_bins_from(\n", - " grid=dataset.grid,\n", - " sub_size_list=[8, 4, 1],\n", - " radial_list=[0.3, 0.6],\n", - " centre_list=[(0.0, 0.0)],\n", - ")\n", - "\n", - "dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Model + Search + Analysis__\n", - "\n", - "Lets fit the data using this mask, by creating the search as per usuag. Note that the `imaging` data with this mask\n", - "applied is passed into the `AnalysisImaging` object, ensuring that this is the mask the model-fit uses. " - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "galaxy = af.Model(ag.Galaxy, redshift=1.0, bulge=ag.lp.Sersic)\n", - "model = af.Collection(galaxies=af.Collection(galaxy=galaxy))\n", - "\n", - "search = af.Nautilus(\n", - " path_prefix=Path(\"howtogalaxy\", \"chapter_2\"),\n", - " name=\"tutorial_6_with_custom_mask\",\n", - " unique_tag=dataset_name,\n", - " n_live=80,\n", - " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - ")\n", - "\n", - "analysis = ag.AnalysisImaging(dataset=dataset, use_jax=True)\n", - "\n", - "search.fit(model=model, analysis=analysis)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Discussion__\n", - "\n", - "So, we can choose the mask we use in a model-fit. We know that we want the mask remove as little of the galaxy's light, \n", - "but is this the 'right' mask? What is the 'right' mask? Maybe we want a bigger mask? a smaller mask?\n", - "\n", - "When it comes to choosing a mask, we are essentially balancing two things: computational run-time and accuracy. When we\n", - "use a bigger mask the model-fit will take longer to perform. Why? Because a bigger mask includes more image-pixels \n", - "in the analysis, and for every additional image-pixel we have to compute its light, blur it with the PSF, compare\n", - "it to the data, etc.\n", - " \n", - "If run-time was not a consideration we would always choose a bigger mask, for two reasons:\n", - "\n", - " 1) The galaxy may have very faint emission that when you choose the mask you simply do not notice. Overly aggressive \n", - " masking runs the risk of us inadvertantly masking out some of the galaxy's light, which would otherwise better \n", - " constrain the model!\n", - " \n", - " 2) When the data is fitted with a model image, the fit is performed only within the masked region. For certain galaxies\n", - " it is possible that it may produce extraneous emission outside of the masked region that is not actually observed in \n", - " the data itself. If this region had not been masked-out, the model would create residuals in these locations and \n", - " reduce the value of likelihood appropriately, whereas if it is masked out this reduction in likelihood is \n", - " not fed through to the analysis. \n", - "\n", - "As you use **PyAutoGalaxy** more you will get a feel for how fast a model-fit will run given the quality of data,\n", - "model complexity, non-linear search settings, etc. As you develop this intuition, I recommend that you always aim to \n", - "use as large of a mask as possible (whilst still achieving reasonable run-times). Aggressive masking will make \n", - "**PyAutoGalaxy** run very fast, but could lead you to infer an incorrect model! \n", - "\n", - "In chapter 3, where we introduce 'non-linear search chaining' we will see how we can use tighter masks in earlier \n", - "searches to achieve faster run times.\n", - "\n", - "If your data includes the light of additional galaxies nearby you may much have no choice but to use a smaller \n", - "circular mask, because it is important these objects do not interfere with the fit. \n", - "\n", - "In fact, you can drawcustom masks that remove their light entirely. You may now wish to checkout \n", - "the `autogalaxy_workspace/*/imaging/preprocess` package. This includes tools for creating custom masks and \n", - "marking the positions on a galaxy (via a GUI) so you can use them in a model-fit.\n", - "\n", - "__Wrap Up__\n", - "\n", - "There are is one thing you should bare in mind in terms of masking:\n", - "\n", - " 1) Customizing the mask for the analysis of one galaxy gets the analysis running fast and can provide accurate \n", - " non-linear sampling. However, for a large sample of galaxies, this high level of customization may take a lot of time. " - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_2_modeling/tutorial_7_results.ipynb b/notebooks/howtogalaxy/chapter_2_modeling/tutorial_7_results.ipynb deleted file mode 100644 index cbfeedd0..00000000 --- a/notebooks/howtogalaxy/chapter_2_modeling/tutorial_7_results.ipynb +++ /dev/null @@ -1,255 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tutorial 5: Results\n", - "===================\n", - "\n", - "In the previous tutorials, each search returned a `Result` object, which we used to plot the maximum log likelihood\n", - "fit each model-fit. In this tutorial, we'll take a look at the result object in a little more detail.\n", - "\n", - "__Contents__\n", - "\n", - "**Initial Setup:** Perform a model-fit to obtain a Result object.\n", - "**Galaxies & Fit:** Access the maximum likelihood galaxies and fit from the result.\n", - "**Samples:** Inspect the non-linear search samples, including parameter estimates and errors.\n", - "**Workspace:** Pointers to more detailed results examples in the workspace.\n", - "**Database:** Overview of the database functionality for managing large numbers of results.\n", - "**Wrap Up:** Summary of the Result object and its key attributes." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autoconf import setup_notebook; setup_notebook()\n", - "\n", - "from pathlib import Path\n", - "import autogalaxy as ag\n", - "import autogalaxy.plot as aplt\n", - "import autofit as af" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Initial Setup__\n", - "\n", - "Lets use the model-fit performed in tutorial 1 to get a `Result` object." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_name = \"simple__sersic\"\n", - "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset Auto-Simulation__\n", - "\n", - "If the dataset does not already exist on your system, it will be created by running the corresponding\n", - "simulator script. This ensures that all example scripts can be run without manually simulating data first." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "if not dataset_path.exists():\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.run(\n", - " [sys.executable, \"scripts/howtogalaxy/simulators/sersic.py\"],\n", - " check=True,\n", - " )\n", - "\n", - "\n", - "dataset = ag.Imaging.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " pixel_scales=0.1,\n", - ")\n", - "\n", - "mask = ag.Mask2D.circular(\n", - " shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=3.0\n", - ")\n", - "\n", - "dataset = dataset.apply_mask(mask=mask)\n", - "\n", - "over_sample_size = ag.util.over_sample.over_sample_size_via_radial_bins_from(\n", - " grid=dataset.grid,\n", - " sub_size_list=[8, 4, 1],\n", - " radial_list=[0.3, 0.6],\n", - " centre_list=[(0.0, 0.0)],\n", - ")\n", - "\n", - "dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size)\n", - "\n", - "model = af.Collection(\n", - " galaxies=af.Collection(galaxy=af.Model(ag.Galaxy, redshift=0.5, mass=ag.lp.Sersic))\n", - ")\n", - "\n", - "search = af.Nautilus(\n", - " path_prefix=Path(\"howtogalaxy\", \"chapter_2\"),\n", - " name=\"tutorial_1_non_linear_search\",\n", - " unique_tag=dataset_name,\n", - " n_live=80,\n", - " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - ")\n", - "\n", - "analysis = ag.AnalysisImaging(dataset=dataset, use_jax=True)\n", - "\n", - "result = search.fit(model=model, analysis=analysis)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Galaxies & Fit__\n", - "\n", - "In the previous tutorials, we saw that this result contains the maximum log likelihood fit, which provide\n", - "a fast way to visualize the result.\n", - "\n", - "It also contains the maximum log likelihood galaxies." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_galaxies(\n", - " galaxies=result.max_log_likelihood_galaxies, grid=mask.derive_grid.all_false\n", - ")\n", - "\n", - "aplt.subplot_fit_imaging(fit=result.max_log_likelihood_fit)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Samples__\n", - "\n", - "The result contains a lot more information about the model-fit. \n", - "\n", - "For example, the `Samples` object contains all of the non-linear search samples, including the parameters \n", - "of every successful model evaluation and their log likelihood values. These are used for computing information \n", - "about the model-fit, such as the most probably parameter estimates and the error inferred for every parameter." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(result.samples)\n", - "print(\"Parameters of 1st Sample:\")\n", - "print(result.samples.parameter_lists[0][:])\n", - "print(\"Log Likelihood of 1st Sample:\")\n", - "print(result.samples.log_likelihood_list[0])" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Workspace__\n", - "\n", - "We are not going into any more detail on the result variable in this tutorial, or in the **HowToGalaxy** lectures.\n", - "\n", - "A comprehensive description of the results can be found at the following packages:\n", - "\n", - " `autogalaxy_workspace/*/results`\n", - " \n", - "The results API for CCD imaging data is the same as for other data types (e.g. interferometer, point-soures). This\n", - "package can therefore be used to learn the API and then translate to other data types.\n", - "\n", - "__Database__\n", - "\n", - "Once a search has completed running, we have a set of results on our hard disk which we can manually inspect and \n", - "analyse. Alternatively, we can return the results from the search.fit() method and manipulate them in a Python script\n", - "or notebook. \n", - "\n", - "However, imagine you have a large dataset consisting of many images of galaxies. You analyse each image \n", - "individually using a search, producing a large library of results on your hard disk. There will be lots of paths and \n", - "directories to navigate, and at some point there will simply be too many results for it to be an efficient use of your \n", - "time to analyse the results by sifting through the outputs on your hard disk one-by-one.\n", - "\n", - "**PyAutoGalaxy**'s database tools solve this problem, by making it possible for us to write the results to a .sqlite \n", - "database file and load the results from hard-disk to a Python script or Jupyter notebook. This database supports\n", - "advanced queries, so specific results can be loaded and inspected.\n", - "\n", - "We won't go into any more detail on the database in this tutorial. If you think the database will be useful, checkout \n", - "the full set of database tutorials which can be found in the folder `autogalaxy_workspace/*/imaging/advanced/database`. \n", - "\n", - "Here, you'll learn how to:\n", - "\n", - " - Use the database to query for results which fit a certain model or give a certain result. \n", - " \n", - " - Use the `Samples` to produce many different results from the fit, including error estimates on parameters and \n", - " plots of the probability density function of parameters in 1D and 2D.\n", - " \n", - " - Visualize results, for example the fit to a lens dataset.\n", - "\n", - "\n", - "__Wrap Up__\n", - "\n", - "Even if you are only modeling a small sample of galaxies, if you anticipate using **PyAutoGalaxy** for the long-term I \n", - "strongly recommend you begin using the database to inspect and analyse your result. \n", - "\n", - "This is because it makes it simple to perform all analyse in a Jupyter notebook, which is the most flexible and \n", - "versatile way to check results and make figures." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_2_modeling/tutorial_8_need_for_speed.ipynb b/notebooks/howtogalaxy/chapter_2_modeling/tutorial_8_need_for_speed.ipynb deleted file mode 100644 index 34d6096e..00000000 --- a/notebooks/howtogalaxy/chapter_2_modeling/tutorial_8_need_for_speed.ipynb +++ /dev/null @@ -1,120 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tutorial 8: Need For Speed\n", - "==========================\n", - "\n", - "In this chapter, we have learnt how to model galaxies and how to balance complexity and realism to ensure that we\n", - "infer a good model.\n", - "\n", - "__Contents__\n", - "\n", - "**Searching Non-linear Parameter Space:** How dimensionality, priors and settings drive search run-times.\n", - "**Algorithmic Optimization:** How PyAutoGalaxy uses numba and JAX for fast computation.\n", - "**Data Quantity:** How the number of image pixels affects run-time.\n", - "**Wrap Up:** Summary of strategies for keeping model-fit run-times manageable.\n", - "\n", - "For fitting more complex models, the final challenge that we face is keeping the run-time low. One can easily end\n", - "up in a situation where a model-fit takes days, or longer, to fit just one image. For fitting complex models and high\n", - "resolution datasets this is somewhat unavoidable. However, it is worth us discussing what drives the long run-times of\n", - "the modeling process and how we might speed it up.\n", - "\n", - "__Searching Non-linear Parameter Space__\n", - "\n", - "The time it takes for the non-linear search to sample parameter space and find the high likelihood models is driven by:\n", - "\n", - " - Dimensionality: A more complex parameter space (e.g. more parameters) takes longer to search.\n", - " - Priors: The broader the priors on each parameter the longer the search.\n", - " - Settings: Non-linear search settings which sample parameter space more thoroughly lead to longer run-times.\n", - "\n", - "When we use only one search to fit a model, we are somewhat restricted in how we can try to achieve faster run\n", - "times by changing these 3 aspects of the search.\n", - "\n", - "In the next chapter, we introduce 'non-linear search chaining', which fits a model using multiple searches that\n", - "are performed back-to-back. A key motivation for this is that it gives us a lot more flexibility in juggling the\n", - "dimensionality, priors and settings so as to perform faster and more efficient modeling.\n", - "\n", - "In the optional **HowToGalaxy** tutorial `chapter_optional/tutorial_searches.ipynb` we discuss other non-linear\n", - "searches supported by **HowToGalaxy** which use a different approach to sample parameter sample than `nautilus`. For\n", - "those familiar with statistical inference, this includes maximum likelihood estimators and MCMC algorithms.\n", - "\n", - "For galaxy modeling, there are maximum likelihood estimator methods (e.g. the Levenberg-Marquardt) that for simple\n", - "models (e.g. a single `Sersic`) can often reliable infer the maximum likelihood model, using 10 times or fewer\n", - "likelihood evaluations than `nautilus` therefore running over ten times or more faster). However, these methods do\n", - "not infer reliable errors and are subject to inferring local maximum. Nevertheless, users modeling large galaxy\n", - "samples may wish to investigate these methods.\n", - "\n", - "__Algorithmic Optimization__\n", - "\n", - "Every operation **PyAutoGalaxy** performs to fit galaxy data with a model takes time, for example:\n", - "\n", - " - Computing the intensity values from a light profile.\n", - " - Convolving the galaxies image with the PSF to compare it to the data.\n", - "\n", - "One can therefore in principle make **PyAutoGalaxy** run faster by using more efficient algorithms. However, I am\n", - "confident that for many tasks and operations we have written code that is already very fast!\n", - "\n", - "I often get asked, given that **PyAutoGalaxy** is written in Python (a synonymously slow programming language), is it\n", - "not really slow? **PyAutoGalaxy** uses a library called `numba` to ensure that it runs fast, which recompiles Python\n", - "functions into C functions before **PyAutoGalaxy** runs. This gives us C-like speed, but in Python code. If you`ve got\n", - "your own code that needs speeding up, I strongly recommend that you look up Numba:\n", - "\n", - "http://numba.pydata.org/\n", - "\n", - "Therefore, **PyAutoGalaxy** is pretty well optimized and there are no 'low hanging fruit' speed ups available by\n", - "writing the code in a different language.\n", - "\n", - "__Data Quantity__\n", - "\n", - "The final factor driving run-speed is the quantity of data that is fitted. For every image-pixel that we fit,\n", - "we have to compute the light profile intensities and convolve it with the telescope's PSF. The larger that PSF is,\n", - "the more convolution operations we have to perform too.\n", - "\n", - "In the previous exercises, we used images with a pixel scale of 0.1\". This value is relatively low resolution: most\n", - "Hubble Space Telescope images have a pixel scale of 0.05\", which is four times the number of pixels! Some telescopes\n", - "observe at scales of 0.03\" or, dare I say it, 0.01\". At these resolutions things can *really* slow down, if we\n", - "do not think carefully about run speed beforehand.\n", - "\n", - "There are ways that we can reduce the number of image-pixels we fit, via masking. If we mask out more of the image,\n", - "we will fit fewer pixels and **PyAutoGalaxy** will run faster. If you want the best, most perfect model possible,\n", - "aggressive masking and cutting the data in this way is a bad idea, as discussed in tutorial 5.\n", - "\n", - "__Wrap Up__\n", - "\n", - "This tutorial simply wanted to get you thinking about *why* a model takes as long to fit as it does." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_3_search_chaining/README.rst b/notebooks/howtogalaxy/chapter_3_search_chaining/README.rst deleted file mode 100644 index b44e0fa6..00000000 --- a/notebooks/howtogalaxy/chapter_3_search_chaining/README.rst +++ /dev/null @@ -1,25 +0,0 @@ -In chapter 3, we introduce non-linear search chaining, whereby modeling pipelines are composed which each fit a -different model. - -**Colab** links to every tutorial are included. - -Files ------ - -`Tutorial 1: Search Chaining `_ -- Breaking the modeling procedure into a chained sequence of model-fits. - -`Tutorial 2: Prior Passing `_ -- How the results of earlier searches are passed to later searches. - -`Tutorial 3: Lens `_ -- Fitting the galaxy's light followed by its mass using chained searches. - -`Tutorial 4: Two Lens galaxies `_ -- Modeling a galaxy with two lens galaxies using chained searches. - -`Tutorial 5: Complex Source `_ -- Using multiple light profiles to fit a complex and irregular source using chained searches. - -`Tutorial 6: SLaM `_ -- Template pipelines for fitting model is standardized ways. \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_3_search_chaining/tutorial_1_search_chaining.ipynb b/notebooks/howtogalaxy/chapter_3_search_chaining/tutorial_1_search_chaining.ipynb deleted file mode 100644 index 2c328385..00000000 --- a/notebooks/howtogalaxy/chapter_3_search_chaining/tutorial_1_search_chaining.ipynb +++ /dev/null @@ -1,546 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tutorial 1: Search Chaining\n", - "===========================\n", - "\n", - "In chapter 2, we learnt how to perform modeling using a non-linear search. In all of the tutorials, we fitted the\n", - "data using just one non-linear search. In this chapter, we introduce a technique called 'non-linear search chaining',\n", - "which fits a model using a sequence of non-linear searches. The initial searches fit simpler models whose parameter\n", - "spaces can be more accurately and efficiently sampled. The results of this search are then passed to later searches\n", - "which fit models of gradually increasing complexity.\n", - "\n", - "Lets think back to tutorial 4 of chapter 2. We learnt there were three approaches one could take fitting a model\n", - "accurately if we found that a model fit failed. These were:\n", - "\n", - " 1) Tuning our priors to the galaxy we're fitting.\n", - " 2) Making our model less complex.\n", - " 3) Searching non-linear parameter space for longer.\n", - "\n", - "However, each of the above approaches has disadvantages. The more we tune our priors, the less we can generalize our\n", - "analysis to a different galaxy. The less complex we make our model, the less realistic it is. And if we rely too\n", - "much on searching parameter space for longer, we could end up with search`s that take days, weeks or months to run.\n", - "\n", - "In this tutorial, we are going to show how search chaining combines these 3 approaches such that we can fit\n", - "complex and realistic models in a way that that can be generalized to many different galaxies. To do this,\n", - "we'll run 2 searches, and chain the model inferred in the first search to the priors of the second search`s lens\n", - "model.\n", - "\n", - "Our first search will make the same bulge-disk alignment assumption we made in the previous tutorial. We saw that this\n", - "gives a reasonable model. However, we'll make a couple of extra simplifying assumptions, to really try and bring\n", - "our model complexity down and get the non-linear search running fast.\n", - "\n", - "The model we infer above will therefore be a lot less realistic. But it does not matter, because in the second search\n", - "we are going to relax these assumptions and fit the more realistic model. The beauty is that, by running the first\n", - "search, we can use its results to tune the priors of our second search. For example:\n", - "\n", - " 1) The first search should give us a pretty good idea of the galaxy's bulge and disk profiles, for example its\n", - " centre, intensity, effective radius.\n", - "\n", - "__Contents__\n", - "\n", - "**Initial Setup:** Load the dataset and apply a mask.\n", - "**Model:** Compose a simplified model for the first search with aligned bulge-disk assumptions.\n", - "**Search + Analysis:** Run the first search with the simplified model.\n", - "**Result:** Inspect the result of the first search.\n", - "**Prior Passing:** Manually pass results from search 1 as priors for search 2.\n", - "**Result:** Inspect the final result of the chained search.\n", - "**Wrap Up:** Summary of how search chaining improves model-fitting reliability." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autoconf import setup_notebook; setup_notebook()\n", - "\n", - "import numpy as np\n", - "from pathlib import Path\n", - "import autogalaxy as ag\n", - "import autogalaxy.plot as aplt\n", - "import autofit as af" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Initial Setup__\n", - "\n", - "we'll use the same galaxy data as tutorial 4 of chapter 2, where:\n", - "\n", - " - The galaxy's bulge is an `Sersic`.\n", - " - The galaxy's disk is an `Exponential`." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_name = \"simple\"\n", - "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset Auto-Simulation__\n", - "\n", - "If the dataset does not already exist on your system, it will be created by running the corresponding\n", - "simulator script. This ensures that all example scripts can be run without manually simulating data first." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "if not dataset_path.exists():\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.run(\n", - " [sys.executable, \"scripts/imaging/simulator.py\"],\n", - " check=True,\n", - " )\n", - "\n", - "\n", - "dataset = ag.Imaging.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " pixel_scales=0.1,\n", - ")\n", - "\n", - "mask = ag.Mask2D.circular(\n", - " shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=2.5\n", - ")\n", - "\n", - "dataset = dataset.apply_mask(mask=mask)\n", - "\n", - "over_sample_size = ag.util.over_sample.over_sample_size_via_radial_bins_from(\n", - " grid=dataset.grid,\n", - " sub_size_list=[8, 4, 1],\n", - " radial_list=[0.3, 0.6],\n", - " centre_list=[(0.0, 0.0)],\n", - ")\n", - "\n", - "dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size)\n", - "\n", - "aplt.subplot_imaging_dataset(dataset=dataset)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Model__\n", - "\n", - "As we've eluded to before, one can look at an image and immediately identify the centre of the galaxy. It's \n", - "that bright blob of light in the middle! Given that we know we're going to make the model more complex in the \n", - "next search, lets take a more liberal approach than before and fix the centre of the bulge and \n", - "disk to $(y,x)$ = (0.0\", 0.0\")..\n", - "\n", - "Now, you might be thinking, doesn`t this prevent our search from generalizing to other galaxies? What if the \n", - "centre of their galaxy isn't at (0.0\", 0.0\")?\n", - "\n", - "Well, this is true if our dataset reduction centres the galaxy somewhere else. But we get to choose where we \n", - "centre it when we make the image. Therefore, I`d recommend you always centre the galaxy at the same location, \n", - "and (0.0\", 0.0\") seems the best choice!" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "bulge = af.Model(ag.lp_linear.Sersic)\n", - "disk = af.Model(ag.lp_linear.Exponential)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You haven't actually seen a line like this one before. By setting a parameter to a number (and not a prior) it is be \n", - "removed from non-linear parameter space and always fixed to that value. Pretty neat, huh?" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "bulge.centre_0 = 0.0\n", - "bulge.centre_1 = 0.0\n", - "disk.centre_0 = 0.0\n", - "disk.centre_1 = 0.0\n", - "\n", - "galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=bulge, disk=disk)\n", - "\n", - "model_1 = af.Collection(galaxies=af.Collection(galaxy=galaxy))" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `info` attribute shows the model in a readable format." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(model_1.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Search + Analysis__\n", - "\n", - "Now lets create the search and analysis." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "search_1 = af.Nautilus(\n", - " path_prefix=Path(\"howtogalaxy\", \"chapter_3\"),\n", - " name=\"tutorial_1_search_chaining_1\",\n", - " unique_tag=dataset_name,\n", - " n_live=100,\n", - " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - ")\n", - "\n", - "analysis_1 = ag.AnalysisImaging(dataset=dataset, use_jax=True)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Lets run the search, noting that our liberal approach to reducing the model complexity has reduced it to just \n", - "6 parameters." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(\n", - " \"The non-linear search has begun running - checkout the workspace/output/5_chaining_searches\"\n", - " \" folder for live output of the results, images and model.\"\n", - " \" This Jupyter notebook cell with progress once search has completed - this could take some time!\"\n", - ")\n", - "\n", - "result_1 = search_1.fit(model=model_1, analysis=analysis_1)\n", - "\n", - "print(\"Search has finished run - you may now continue the notebook.\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Result__\n", - "\n", - "The results are summarised in the `info` attribute." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(result_1.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "And indeed, we get a reasonably good model and fit to the data, in a much shorter space of time!" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_fit_imaging(fit=result_1.max_log_likelihood_fit)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Prior Passing__\n", - "\n", - "Now all we need to do is look at the results of search 1 and pass the results as priors for search 2. Lets setup \n", - "a custom search that does exactly that.\n", - "\n", - "`TruncatedGaussianPrior`'s are a nice way to pass priors. They tell the non-linear search where to look, but leave open the \n", - "possibility that there might be a better solution nearby. In contrast, `UniformPrior`'s put hard limits on what values a \n", - "parameter can or can`t take. It makes it more likely we will accidentally cut-out the global maxima solution.\n", - "\n", - "Note that below the `disk` has become an `Sersic`." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "bulge = af.Model(ag.lp_linear.Sersic)\n", - "disk = af.Model(ag.lp_linear.Sersic)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "What I've done below is looked at the results of search 1 and manually specified a prior for every parameter. If a \n", - "parameter was fixed in the previous search, its prior is based around the previous value. Don't worry about the sigma \n", - "values for now, I've chosen values that I know will ensure reasonable sampling, but we'll cover this later.\n", - "\n", - "__LENS BULGE PRIORS:__" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "bulge.centre.centre_0 = af.TruncatedGaussianPrior(\n", - " mean=0.0, sigma=0.1, lower_limit=-np.inf, upper_limit=np.inf\n", - ")\n", - "bulge.centre.centre_1 = af.TruncatedGaussianPrior(\n", - " mean=0.0, sigma=0.1, lower_limit=-np.inf, upper_limit=np.inf\n", - ")\n", - "bulge.ell_comps.ell_comps_0 = af.TruncatedGaussianPrior(\n", - " mean=0.11, sigma=0.2, lower_limit=-1.0, upper_limit=1.0\n", - ")\n", - "bulge.ell_comps.ell_comps_1 = af.TruncatedGaussianPrior(\n", - " mean=0.05, sigma=0.2, lower_limit=-1.0, upper_limit=1.0\n", - ")\n", - "bulge.effective_radius = af.TruncatedGaussianPrior(\n", - " mean=0.75, sigma=0.4, lower_limit=0.0, upper_limit=np.inf\n", - ")\n", - "bulge.sersic_index = af.TruncatedGaussianPrior(\n", - " mean=4.0, sigma=2.0, lower_limit=0.0, upper_limit=np.inf\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__LENS DISK PRIORS:__" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "disk.centre.centre_0 = af.TruncatedGaussianPrior(\n", - " mean=0.0, sigma=0.1, lower_limit=-np.inf, upper_limit=np.inf\n", - ")\n", - "disk.centre.centre_1 = af.TruncatedGaussianPrior(\n", - " mean=0.0, sigma=0.1, lower_limit=-np.inf, upper_limit=np.inf\n", - ")\n", - "disk.ell_comps.ell_comps_0 = af.TruncatedGaussianPrior(\n", - " mean=0.11, sigma=0.2, lower_limit=-1.0, upper_limit=1.0\n", - ")\n", - "disk.ell_comps.ell_comps_1 = af.TruncatedGaussianPrior(\n", - " mean=0.05, sigma=0.2, lower_limit=-1.0, upper_limit=1.0\n", - ")\n", - "disk.effective_radius = af.TruncatedGaussianPrior(\n", - " mean=1.52, sigma=0.4, lower_limit=0.0, upper_limit=np.inf\n", - ")\n", - "disk.sersic_index = af.TruncatedGaussianPrior(\n", - " mean=1.0, sigma=2.0, lower_limit=0.0, upper_limit=np.inf\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We now compose the model with these components that have had their priors customized. " - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=bulge, disk=disk)\n", - "\n", - "model_2 = af.Collection(galaxies=af.Collection(galaxy=galaxy))" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `info` attribute shows the model, including the priors specified above." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(model_2.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Lets setup and run the search. As expected, it gives us the correct model. However, it does so significantly \n", - "faster than we are used to!" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "batch_size = 50 # Explained chapter 2 tutorial 2\n", - "\n", - "search_2 = af.Nautilus(\n", - " path_prefix=Path(\"howtogalaxy\", \"chapter_3\"),\n", - " name=\"tutorial_1_search_chaining_2\",\n", - " unique_tag=dataset_name,\n", - " n_live=100,\n", - " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - ")\n", - "\n", - "analysis_2 = ag.AnalysisImaging(dataset=dataset, use_jax=True)\n", - "\n", - "print(\n", - " \"The non-linear search has begun running - checkout the workspace/output/5_chaining_searches\"\n", - " \" folder for live output of the results, images and model.\"\n", - " \" This Jupyter notebook cell with progress once search has completed - this could take some time!\"\n", - ")\n", - "\n", - "result_2 = search_2.fit(model=model_2, analysis=analysis_2)\n", - "\n", - "print(\"Search has finished run - you may now continue the notebook.\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Result__\n", - "\n", - "We can again inspect the results via the `info` attribute." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(result_2.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "And a plot of the image shows we get a good model again!" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_fit_imaging(fit=result_2.max_log_likelihood_fit)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Wrap Up__\n", - "\n", - "Chaining two searches together was a huge success. We managed to fit a complex and realistic model, but were able to \n", - "begin by making simplifying assumptions that eased our search of non-linear parameter space. We could apply search 1 to \n", - "pretty much any galaxy and therefore get ourselves a decent model with which to tune search 2`s priors.\n", - "\n", - "You are probably thinking though that there is one huge, giant, glaring flaw in all of this that I've not mentioned. \n", - "Search 2 can`t be generalized to another lens, because its priors are tuned to the image we fitted. If we had a lot \n", - "of galaxies, we`d have to write a new search for every single one. This isn't ideal, is it?\n", - "\n", - "Fortunately, we can pass priors in **PyAutoGalaxy** without specifying the specific values. The API for this technique,\n", - "called prior passing, is the topic of the next tutorial." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_3_search_chaining/tutorial_2_prior_passing.ipynb b/notebooks/howtogalaxy/chapter_3_search_chaining/tutorial_2_prior_passing.ipynb deleted file mode 100644 index ed69eb1c..00000000 --- a/notebooks/howtogalaxy/chapter_3_search_chaining/tutorial_2_prior_passing.ipynb +++ /dev/null @@ -1,522 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tutorial 2: Prior Passing\n", - "=========================\n", - "\n", - "In the previous tutorial, we used non-linear search chaining to break the model-fitting procedure down into two\n", - "non-linear searches. This used an initial search to fit a simple model, whose results were used to tune and\n", - "initialize the priors of a more complex model that was fitted by the second search.\n", - "\n", - "However, the results were passed between searches were passed manually. I explicitly wrote out every result as a prior\n", - "containing the values inferred in the first search. **PyAutoGalaxy** has an API for passing priors in a more generalized\n", - "way, which is the topic of this tutorial.\n", - "\n", - "__Contents__\n", - "\n", - "**Initial Setup:** Load the dataset and apply a mask.\n", - "**Model:** Compose the model for the first search.\n", - "**Search:** Run the first search.\n", - "**Result (Search 1):** Inspect the result of the first search.\n", - "**Prior Passing:** Use the prior passing API to pass results to the second search.\n", - "**Result:** Inspect the final result of the chained search.\n", - "**Wrap Up:** Summary of the prior passing API.\n", - "**Detailed Explanation Of Prior Passing:** In-depth explanation of how priors are passed between searches.\n", - "**EXAMPLE:** A worked example of prior passing for a Sersic sersic_index parameter." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autoconf import setup_notebook; setup_notebook()\n", - "\n", - "import numpy as np\n", - "from pathlib import Path\n", - "import autogalaxy as ag\n", - "import autogalaxy.plot as aplt\n", - "import autofit as af" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Initial Setup__\n", - "\n", - "we'll use the same galaxying data as the previous tutorial, where:\n", - "\n", - " - The galaxy's bulge is an `Sersic`.\n", - " - The galaxy's disk is an `Exponential`.\n", - " \n", - "All the usual steps for setting up a model fit (masking, analysis, etc.) are included below." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_name = \"simple\"\n", - "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset Auto-Simulation__\n", - "\n", - "If the dataset does not already exist on your system, it will be created by running the corresponding\n", - "simulator script. This ensures that all example scripts can be run without manually simulating data first." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "if not dataset_path.exists():\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.run(\n", - " [sys.executable, \"scripts/imaging/simulator.py\"],\n", - " check=True,\n", - " )\n", - "\n", - "\n", - "dataset = ag.Imaging.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " pixel_scales=0.1,\n", - ")\n", - "\n", - "mask = ag.Mask2D.circular(\n", - " shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=2.5\n", - ")\n", - "\n", - "dataset = dataset.apply_mask(mask=mask)\n", - "\n", - "over_sample_size = ag.util.over_sample.over_sample_size_via_radial_bins_from(\n", - " grid=dataset.grid,\n", - " sub_size_list=[8, 4, 1],\n", - " radial_list=[0.3, 0.6],\n", - " centre_list=[(0.0, 0.0)],\n", - ")\n", - "\n", - "dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size)\n", - "\n", - "aplt.subplot_imaging_dataset(dataset=dataset)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Model__\n", - "\n", - "We are going to use the same result of search 1 from the previous tutorial. Thus, we set up an identical model such \n", - "that we instantly load the result from hard-disk." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "bulge = af.Model(ag.lp_linear.Sersic)\n", - "disk = af.Model(ag.lp_linear.Exponential)\n", - "\n", - "bulge.centre_0 = 0.0\n", - "bulge.centre_1 = 0.0\n", - "disk.centre_0 = 0.0\n", - "disk.centre_1 = 0.0\n", - "\n", - "disk.ell_comps = bulge.ell_comps\n", - "\n", - "bulge.sersic_index = 4.0\n", - "\n", - "galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=bulge, disk=disk)\n", - "\n", - "model_1 = af.Collection(galaxies=af.Collection(galaxy=galaxy))" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `info` attribute shows the model in a readable format." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(model_1.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Search__\n", - "\n", - "We also create the same search as the previous tutorial, using the same name to ensure we use the same results, and \n", - "run it." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "analysis_1 = ag.AnalysisImaging(dataset=dataset, use_jax=True)\n", - "\n", - "search_1 = af.Nautilus(\n", - " path_prefix=Path(\"howtogalaxy\", \"chapter_3\"),\n", - " name=\"tutorial_1_search_chaining_1\",\n", - " unique_tag=dataset_name,\n", - " n_live=100,\n", - " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - ")\n", - "\n", - "result_1 = search_1.fit(model=model_1, analysis=analysis_1)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Result (Search 1)__\n", - "\n", - "The results which are used for prior passing are summarised in the `info` attribute." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(result_1.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Prior Passing__\n", - "\n", - "We are now going to use the prior passing API to pass these results, in a way which does not require us to manually \n", - "write out the inferred parameter values of each component. The details of how prior passing is performed will be \n", - "expanded upon at the end of the tutorial.\n", - "\n", - "We start with the bulge, which in the previous search was an `Sersic` with its centre fixed to (0.0, 0.0) \n", - "and its `sersic_index` fixed to 4.0. The API for passing priors is shown below and there are two things worth noting:\n", - "\n", - " 1) We pass the priors using the `model` attribute of the result. This informs **PyAutoGalaxy** to pass the result as a\n", - " model component that is to be fitted for in the next search, using priors that are initialized from the previous\n", - " search's result. Note, if we pass as a `model` a parameter that was fixed in search 1 (e.g. the `sersic_index`) it \n", - " will be fixed to the same value in search 2.\n", - "\n", - " 2) We do not pass the `centre` or `sersic_index` using `model`, because it would be fixed to the values that it was in \n", - " the first search. By omitting the centre, it uses the default priors on a galaxy, whereas we manually tell the \n", - " Sersic index to use a `TruncatedGaussianPrior` centred on 4.0. " - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "bulge = af.Model(ag.lp_linear.Sersic)\n", - "\n", - "bulge.ell_comps = result_1.model.galaxies.galaxy.bulge.ell_comps\n", - "bulge.effective_radius = result_1.model.galaxies.galaxy.bulge.effective_radius\n", - "bulge.sersic_index = af.TruncatedGaussianPrior(\n", - " mean=4.0, sigma=2.0, lower_limit=0.0, upper_limit=5.0\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "For the disk, we are passing the result of an `Exponential` to an `Sersic`.\n", - "\n", - "We do not pass the `ell_comps` because this would pair them to the `bulge`, as was performed in the first \n", - "model-fit." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "disk = af.Model(ag.lp_linear.Sersic)\n", - "\n", - "disk.effective_radius = result_1.model.galaxies.galaxy.disk.effective_radius\n", - "disk.sersic_index = af.TruncatedGaussianPrior(\n", - " mean=1.0, sigma=2.0, lower_limit=0.0, upper_limit=5.0\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We now compose the model with these components that have had their priors customized. " - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=bulge, disk=disk)\n", - "\n", - "model_2 = af.Collection(galaxies=af.Collection(galaxy=galaxy))" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `info` attribute shows the model, including how all priors are updated via prior passing." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(model_2.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Search__\n", - "\n", - "Lets setup and run the search. I have given it a different name to the previous tutorial so we can compare the priors\n", - "that were passed." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "analysis_2 = ag.AnalysisImaging(dataset=dataset, use_jax=True)\n", - "\n", - "search_2 = af.Nautilus(\n", - " path_prefix=Path(\"howtogalaxy\", \"chapter_3\"),\n", - " name=\"tutorial_2_search_chaining_2\",\n", - " unique_tag=dataset_name,\n", - " n_live=100,\n", - " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - ")\n", - "\n", - "print(\n", - " \"The non-linear search has begun running - checkout the workspace/output/5_chaining_searches\"\n", - " \" folder for live output of the results, images and model.\"\n", - " \" This Jupyter notebook cell with progress once search has completed - this could take some time!\"\n", - ")\n", - "\n", - "result_2 = search_2.fit(model=model_2, analysis=analysis_2)\n", - "\n", - "print(\"Search has finished run - you may now continue the notebook.\")\n" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Result__\n", - "\n", - "We can again inspect the results via the `info` attribute." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(result_2.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "And a plot of the image shows we get a good model again!" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_fit_imaging(fit=result_2.max_log_likelihood_fit)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Wrap Up__\n", - "\n", - "We will expand on the prior passing API in the following tutorials. The main thing to note is that we can pass \n", - "entire profiles or galaxies using prior passing, if their model does not change (which for the bulge and disk, was \n", - "not true). The API to pass a whole profile or galaxy is as follows:\n", - " \n", - " bulge = result_1.model.galaxies.galaxy.bulge\n", - " galaxy = result_1.model.galaxies.galaxy\n", - " \n", - "We can also pass priors using an `instance` instead of a `model`. When an `instance` is used, the maximum likelihood\n", - "parameter values are passed as fixed values that are therefore not fitted for by the non-linear search (reducing its\n", - "dimensionality). We will use this in the next tutorial to fit data with two galaxies, where fit one galaxy, fix it to \n", - "the best-fit model in a second search that fits the second galaxy, and then go on to fit both simultaneously in the \n", - "final search.\n", - " \n", - "Lets now think about how priors are passed. Checkout the `model.info` file of the second search of this tutorial. The \n", - "parameters do not use the default priors we saw in search 1 (which are typically broad UniformPriors). Instead, \n", - "they use GaussianPrior`s where:\n", - "\n", - " - The mean values are the median PDF results of every parameter in search 1.\n", - " - The sigma values are specified in the `width_modifier` field of the profile's entry in the `priors.yaml' config \n", - " file (we will discuss why this is used in a moment).\n", - "\n", - "Like the manual `GaussianPrior`'s that were used in tutorial 1, the prior passing API sets up the prior on each \n", - "parameter with a `GaussianPrior` centred on the high likelihood regions of parameter space!\n", - "\n", - "__Detailed Explanation Of Prior Passing__\n", - "\n", - "To end, I provide a detailed overview of how prior passing works and illustrate tools that can be used to customize\n", - "its behaviour. It is up to you whether you want read this, or go ahead to the next tutorial!\n", - "\n", - "Lets say I chain two parameters as follows:\n", - " \n", - " `bulge.effective_radius = result_1.model.galaxies.galaxy.bulge.effective_radius`\n", - "\n", - "By invoking the `model` attribute, the prior is passed following 3 rules:\n", - "\n", - " 1) The new parameter, in this case the einstein radius, uses a `GaussianPrior`.This is ideal, as the 1D pdf results \n", - " we compute at the end of a search are easily summarised as a Gaussian.\n", - "\n", - " 2) The mean of the `GaussianPrior` is the median PDF value of the parameter estimated in search 1.\n", - " \n", - " This ensures that the initial sampling of the new search's non-linear starts by searching the region of non-linear \n", - " parameter space that correspond to highest log likelihood solutions in the previous search. Our priors therefore \n", - " correspond to the `correct` regions of parameter space.\n", - "\n", - " 3) The sigma of the Gaussian uses the value specified for the profile in the `config/priors/*.yaml` config file's \n", - " `width_modifer` field (check these files out now).\n", - "\n", - "The idea here is simple. We want a value of sigma that gives a `GaussianPrior` wide enough to search a broad \n", - "region of parameter space, so that the model can change if a better solution is nearby. However, we want it \n", - "to be narrow enough that we don't search too much of parameter space, as this will be slow or risk leading us \n", - "into an incorrect solution! \n", - "\n", - "The `width_modifier` values in the priors config file have been chosen based on our experience as being a good\n", - "balance broadly sampling parameter space but not being so narrow important solutions are missed.\n", - " \n", - "There are two ways a value is specified using the priors/width file:\n", - "\n", - " 1) Absolute: In this case, the error assumed on the parameter is the value given in the config file. \n", - " For example, if for the width on centre_0 of a light profile, the width modifier reads \"Absolute\" with a value \n", - " 0.05. This means if the error on the parameter centre_0 was less than 0.05 in the previous search, the sigma of \n", - " its `GaussianPrior` in this search will be 0.05.\n", - " \n", - " 2) Relative: In this case, the error assumed on the parameter is the % of the value of the estimated value given in \n", - " the config file. For example, if the intensity estimated in the previous search was 2.0, and the relative error in \n", - " the config file reads \"Relative\" with a value 0.5, then the sigma of the `GaussianPrior` will be 50% of this \n", - " value, i.e. sigma = 0.5 * 2.0 = 1.0.\n", - "\n", - "We use absolute and relative values for different parameters, depending on their properties. For example, using the \n", - "relative value of a parameter like the `Profile` centre makes no sense. If our galaxy is centred at (0.0, 0.0), \n", - "the relative error will always be tiny and thus poorly defined. Therefore, the default configs in **PyAutoGalaxy** use \n", - "absolute errors on the centre.\n", - "\n", - "However, there are parameters where using an absolute value does not make sense. Intensity is a good example of this. \n", - "The intensity of an image depends on its units, S/N, galaxy brightness, etc. There is no single absolute value that \n", - "one can use to generically chain the intensity of any two proflies. Thus, it makes more sense to chain them using \n", - "the relative value from a previous search.\n", - "\n", - "We can customize how priors are passed from the results of a search and non-linear search by editing the\n", - " `prior_passer` settings in the `general.yaml` config file.\n", - "\n", - "__EXAMPLE__\n", - "\n", - "Lets go through an example using a real parameter. Lets say in search 1 we fit the galaxy's light with an \n", - "elliptical Sersic profile, and we estimate that its sersic index is equal to 4.0.\n", - " \n", - "To pass this as a prior to search 2 we write:\n", - "\n", - " galaxy.bulge.sersic_index = result_1.model.galaxy.bulge.sersic_index\n", - "\n", - "The prior on the galaxy's bulge sersic index in search 2 would thus be a `GaussianPrior` with mean=4.0. \n", - "\n", - "The value of the Sersic index `width_modifier` in the priors config file sets sigma. The prior config file specifies \n", - "that we use an \"Absolute\" value of 0.8 to chain this prior. Thus, the `GaussianPrior` in search 2 would have a \n", - "mean=4.0 and sigma=0.8.\n", - "\n", - "If the prior config file had specified that we use an relative value of 0.8, the GaussianPrior in search 2 would have a \n", - "mean=4.0 and sigma = 4.0 * 0.8 = 3.2.\n", - "\n", - "And with that, we're done. Chaining priors is a bit of an art form, but one that works really well. " - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_3_search_chaining/tutorial_3_x2_galaxies.ipynb b/notebooks/howtogalaxy/chapter_3_search_chaining/tutorial_3_x2_galaxies.ipynb deleted file mode 100644 index ed6e2c08..00000000 --- a/notebooks/howtogalaxy/chapter_3_search_chaining/tutorial_3_x2_galaxies.ipynb +++ /dev/null @@ -1,487 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tutorial 3: Two Galaxies\n", - "========================\n", - "\n", - "Up to now, all the images we've fitted had one galaxy. However, we saw in chapter 1 that our galaxies object can\n", - "consist of multiple galaxies which each contribute to the overall emission. Multi-galaxy systems are challenging to\n", - "model, because they add an extra 5-10 parameters to the non-linear search per galaxy and, more problematically, the\n", - "degeneracies between the parameters of the light profiles of the galaxies can be severe.\n", - "\n", - "However, we can still break their analysis down using multiple searches and give ourselves a shot at getting a good\n", - "model. Here, we're going to fit a double galaxy system, fitting as much about each individual galaxy before\n", - "fitting them simultaneously.\n", - "\n", - "Up to now, I've put a focus on an analysis being generag. The script we write in this example is going to be the\n", - "opposite, specific to the image we're modeling. Fitting multiple galaxies is really difficult and writing a\n", - "pipeline that we can generalize to many galaxies isn't currently possible.\n", - "\n", - "__Contents__\n", - "\n", - "**Initial Setup:** Load the double-galaxy dataset and apply a mask.\n", - "**Paths:** Set the output path for chained search results.\n", - "**Search Chaining Approach:** Strategy for fitting two galaxies using a sequence of searches.\n", - "**Model + Search + Analysis + Model-Fit (Search 1):** Fit the first galaxy alone.\n", - "**Result (Search 1):** Inspect the result of fitting the first galaxy.\n", - "**Model (Search 2):** Compose a model for the second galaxy, fixing the first.\n", - "**Search + Analysis + Model-Fit (Search 2):** Fit the second galaxy.\n", - "**Result (Search 2):** Inspect the result of fitting the second galaxy.\n", - "**Model + Search + Analysis + Model-Fit (Search 4):** Fit both galaxies simultaneously.\n", - "**Result (Search 3):** Inspect the final simultaneous fit result.\n", - "**Wrap Up:** Summary of search chaining for multi-galaxy systems." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autoconf import setup_notebook; setup_notebook()\n", - "\n", - "from pathlib import Path\n", - "import autofit as af\n", - "import autogalaxy as ag\n", - "import autogalaxy.plot as aplt" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Initial Setup__\n", - "\n", - "we'll use new galaxying data, where:\n", - "\n", - " - There are two galaxy's whose `LightProfile`'s are both `Sersic`'s." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_name = \"sersic_x2\"\n", - "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset Auto-Simulation__\n", - "\n", - "If the dataset does not already exist on your system, it will be created by running the corresponding\n", - "simulator script. This ensures that all example scripts can be run without manually simulating data first." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "if not dataset_path.exists():\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.run(\n", - " [sys.executable, \"scripts/guides/plot/simulator.py\"],\n", - " check=True,\n", - " )\n", - "\n", - "\n", - "dataset = ag.Imaging.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " pixel_scales=0.05,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Mask__\n", - "\n", - "We need to choose our mask for the analysis. Given the light of both galaxies is present in the image we'll need to \n", - "include all their light in the image, so lets use a large circular mask. \n", - "\n", - "We'll use this mask in all three searches, however you could imagine customizing it on a per-search basis to speed up\n", - "the analysis." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "mask = ag.Mask2D.circular(\n", - " shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=6.0\n", - ")\n", - "\n", - "dataset = dataset.apply_mask(mask=mask)\n", - "\n", - "over_sample_size = ag.util.over_sample.over_sample_size_via_radial_bins_from(\n", - " grid=dataset.grid,\n", - " sub_size_list=[8, 4, 1],\n", - " radial_list=[0.3, 0.6],\n", - " centre_list=[(0.0, -1.0), (0.0, 1.0)],\n", - ")\n", - "\n", - "dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size)\n", - "\n", - "aplt.subplot_imaging_dataset(dataset=dataset)\n" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Paths__\n", - "\n", - "All four searches will use the same `path_prefix`, so we write it here to avoid repetition." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "path_prefix = Path(\"howtogalaxy\") / \"chapter_3\" / \"tutorial_3_x2_galaxies\"" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Search Chaining Approach__\n", - "\n", - "Looking at the image, there are two blobs of light corresponding to the two galaxies. \n", - "\n", - "So, how can we break the modeling up? As follows:\n", - "\n", - " 1) Fit and subtract the light of the left galaxy individually.\n", - " 2) Fit and subtract the light of the right galaxy individually.\n", - " 3) Use these results to initialize a fit which fits both galaxy's simultaneously.\n", - "\n", - "So, with this in mind, we'll perform an analysis using 3 searches:\n", - "\n", - " 1) Fit the light of the galaxy on the left of the image, at coordinates (0.0\", -1.0\").\n", - " 2) Fit the light of the galaxy on the right of the image, at coordinates (0.0\", 1.0\").\n", - " 4) Fit all relevant parameters simultaneously, using priors from searches 1, and 2.\n", - "\n", - "__Model + Search + Analysis + Model-Fit (Search 1)__\n", - "\n", - "Search 1 we fit a model where:\n", - "\n", - " - The left galaxy's light is a parametric linear `DevVaucouleurs` bulge with fixed centre [3 parameters].\n", - "\n", - " - the right galaxy's light is omitted.\n", - "\n", - "The number of free parameters and therefore the dimensionality of non-linear parameter space is N=3.\n", - "\n", - "__Notes__\n", - "\n", - "The `DevVaucouleurs` is an `Sersic` profile with `sersic_index=4`.\n", - "\n", - "We fix the centre of its light to (0.0, -1.0), the pixel we know the left galaxy's light centre peaks.\n", - "\n", - "We use linear light profiles througout this script, given that the model is quite complex and this helps\n", - "simplify it." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "left_galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=ag.lp_linear.DevVaucouleurs)\n", - "left_galaxy.bulge.centre_0 = 0.0\n", - "left_galaxy.bulge.centre_1 = -1.0\n", - "\n", - "model_1 = af.Collection(galaxies=af.Collection(left_galaxy=left_galaxy))" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `info` attribute shows the model in a readable format." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(model_1.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Search + Analysis + Model-Fit (Search 1)__" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "analysis_1 = ag.AnalysisImaging(dataset=dataset, use_jax=True)\n", - "\n", - "search_1 = af.Nautilus(\n", - " path_prefix=path_prefix,\n", - " name=\"search[1]__left_galaxy_light[bulge_linear]\",\n", - " unique_tag=dataset_name,\n", - " n_live=75,\n", - " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - ")\n", - "\n", - "result_1 = search_1.fit(model=model_1, analysis=analysis_1)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Result (Search 1)__\n", - "\n", - "The results which are used for prior passing are summarised in the `info` attribute." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(result_1.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Model (Search 2)__\n", - "\n", - "Search 2 we fit a model where:\n", - "\n", - " - The left galaxy's light is a parametric linear `DevVaucouleurs` bulge [0 parameters: fixed from search 1].\n", - "\n", - " - The right galaxy's light is a parametric linear `DevVaucouleurs` bulge with a fixed centre [3 parameters].\n", - "\n", - " - The galaxy's mass galaxy are omitted.\n", - "\n", - "The number of free parameters and therefore the dimensionality of non-linear parameter space is N=3.\n", - "\n", - "We fix the centre of the right lens's light to (0.0, 1.0), the pixel we know the right galaxy's light centre peaks.\n", - "\n", - "We also pass the result of the `left_galaxy` from search ` as an `instance`, which should improve the fitting of the\n", - "right lens." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "right_galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=ag.lp_linear.DevVaucouleurs)\n", - "right_galaxy.bulge.centre_0 = 0.0\n", - "right_galaxy.bulge.centre_1 = 1.0\n", - "\n", - "model_2 = af.Collection(\n", - " galaxies=af.Collection(\n", - " left_galaxy=result_1.instance.galaxies.left_galaxy, right_galaxy=right_galaxy\n", - " )\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `info` attribute shows the model, including how all priors are updated via prior passing." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(model_2.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Search + Analysis + Model-Fit (Search 2)__" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "analysis_2 = ag.AnalysisImaging(dataset=dataset, use_jax=True)\n", - "\n", - "search_2 = af.Nautilus(\n", - " path_prefix=path_prefix,\n", - " name=\"search[2]__right_galaxy_light[bulge_linear]\",\n", - " unique_tag=dataset_name,\n", - " n_live=75,\n", - " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - ")\n", - "\n", - "result_2 = search_2.fit(model=model_2, analysis=analysis_2)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Result (Search 2)__\n", - "The results which are used for prior passing are summarised in the `info` attribute." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(result_2.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Model + Search + Analysis + Model-Fit (Search 4)__\n", - "\n", - "Search 4 we fit a model where:\n", - "\n", - " - The left galaxy's light is a parametric linear `Sersic` bulge with centre fixed [4 parameters: priors initialized \n", - " from search 1].\n", - "\n", - " - The right galaxy's light is a parametric linear `Sersic` bulge with centre fixed [4 parameters: priors initialized \n", - " from search 2].\n", - "\n", - "The number of free parameters and therefore the dimensionality of non-linear parameter space is N=8.\n", - "\n", - "We can use a special prior passing method to do this, called `take_attributes`. This scans the `DevVaucouleurs`\n", - "passed to the `take_attributes` method for all parameters which have the same name as the `Sersic` model,\n", - "and if their names are the same it passes their prior as a `model` (like we did above). Thus, it will locate all 6\n", - "parameters in common between the two profiles (centre, ell_comps, intensity, effective_radius) and pass those,\n", - "leaving the `sersic_index`'s priors as the default values.\n", - "\n", - "The `take_attributes` method is used in many examples of prior passing, when we pass a simpler parameterization of a\n", - "model to a more complex model. Another good example would be passing the result of a `IsothermalSph` to an\n", - "`Isothermal`." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "left_galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=ag.lp_linear.Sersic)\n", - "left_galaxy.bulge.take_attributes(result_1.model.galaxies.left_galaxy.bulge)\n", - "\n", - "right_galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=ag.lp_linear.Sersic)\n", - "right_galaxy.bulge.take_attributes(result_2.model.galaxies.right_galaxy.bulge)\n", - "\n", - "model_3 = af.Collection(\n", - " galaxies=af.Collection(left_galaxy=left_galaxy, right_galaxy=right_galaxy)\n", - ")\n", - "\n", - "analysis_3 = ag.AnalysisImaging(dataset=dataset, use_jax=True)\n", - "\n", - "search_3 = af.Nautilus(\n", - " path_prefix=path_prefix,\n", - " name=\"search[3]_light_x2[bulge_linear]\",\n", - " unique_tag=dataset_name,\n", - " n_live=100,\n", - " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - ")\n", - "\n", - "result_3 = search_3.fit(model=model_3, analysis=analysis_3)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Result (Search 3)__\n", - "\n", - "The final results are summarised in the `info` attribute." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(result_3.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Wrap Up__\n", - "\n", - "We have successfully fitted multiple galaxies, but fitting each one-by-one." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_4_pixelizations/README.rst b/notebooks/howtogalaxy/chapter_4_pixelizations/README.rst deleted file mode 100644 index 75d680f2..00000000 --- a/notebooks/howtogalaxy/chapter_4_pixelizations/README.rst +++ /dev/null @@ -1,30 +0,0 @@ -In chapter 4, we use **Pixelizations** to reconstruct complex source galaxies on pixelized grids. - -**Colab** links to every tutorial are included. - -Files ------ - -`Tutorial 1: Pixelizations `_ -- Creating a pixel-grid in the source-plane. - -`Tutorial 2: Mappers `_ -- How a pixelization maps source-pixels to image-pixels. - -`Tutorial 3: Inversions `_ -- Inverting the mappings to reconstruct the source's light. - -`Tutorial 4: Bayesian Regularization `_ -- Smoothing the source within a Bayesian framework. - -`Tutorial 5: Borders `_ -- Preventing highly demagnified image-pixels ruining the inversion. - -`Tutorial 6: Lens Modeling `_ -- How to use inversions to fit a model. - -`Tutorial 7: Adaptive Pixelization `_ -- A Voronoi mesh which adapts to the mass model's magnification. - -`Tutorial 8: Model Fit `_ -- An example modeling pipeline which uses an inversion. \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_4_pixelizations/tutorial_1_pixelizations.ipynb b/notebooks/howtogalaxy/chapter_4_pixelizations/tutorial_1_pixelizations.ipynb deleted file mode 100644 index edd2edb7..00000000 --- a/notebooks/howtogalaxy/chapter_4_pixelizations/tutorial_1_pixelizations.ipynb +++ /dev/null @@ -1,304 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tutorial 1: pixelizations\n", - "=========================\n", - "\n", - "In the previous chapters, we used light profiles to model the light of a galaxy, where the light profile was an\n", - "analytic description of how the luminosity varies as a function of radius.\n", - "\n", - "In this chapter, we are instead going to reconstruct the galaxy's light on a pixel-grid, and in this tutorial we will\n", - "learn how to create a pixelization in **PyAutoGalaxy**.\n", - "\n", - "__Contents__\n", - "\n", - "**Initial Setup:** Create a grid for illustration.\n", - "**Mesh:** Set up a rectangular mesh for the pixelization.\n", - "**Wrap Up:** Summary of pixelization concepts." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autoconf import setup_notebook; setup_notebook()\n", - "\n", - "import autogalaxy as ag\n", - "import autogalaxy.plot as aplt\n", - "from autoarray.inversion.plot.mapper_plots import plot_mapper" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Initial Setup__\n", - "\n", - "Lets setup a grid. " - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "grid = ag.Grid2D.uniform(shape_native=(100, 100), pixel_scales=0.05)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Mesh__\n", - "\n", - "Next, lets set up a `Mesh` using the `mesh` module. The mesh represents the pixel-grid used by the pixelization\n", - "to reconstruct the galaxy.\n", - "\n", - "There are multiple `Mesh`'s available in **PyAutoGalaxy**. For now, we'll keep it simple and use a uniform \n", - "rectangular grid, whose `shape` defines its $(y,x)$ dimensions. We will make it the same shape as the 2D grid." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "mesh = ag.mesh.RectangularAdaptDensity(shape=(100, 100))" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We now pass the mesh to a `Pixelization`." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "pixelization = ag.Pixelization(mesh=mesh)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "By itself, a pixelization does not tell us much. It has no grid of $(y,x)$ coordinates, no image, and no information\n", - "about the galaxy we are fitting. \n", - "\n", - "This information comes when we use the pixelization to create up a `Mapper`, which we perform below using the grid \n", - "that we created above." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "interpolator = mesh.interpolator_from(\n", - " source_plane_data_grid=grid, source_plane_mesh_grid=None\n", - ")\n", - "\n", - "mapper = ag.Mapper(interpolator=interpolator)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This `Mapper` is a `RectangularMapper` -- every `Mesh` and `Pixelization` generates it owns mapper." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(type(mapper))" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "By plotting our mapper, we now see our `Pixelization`. Its a fairly boring grid of rectangular pixels." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "plot_mapper(\n", - " mapper=mapper, title=\"Fairly Boring Grid2D of RectangularAdaptDensity Pixels\"\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "However, the `Mapper` does contain lots of interesting information about our `Pixelization`, for example its \n", - "pixelization_grid tells us where the pixel centers are located." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(\"RectangularAdaptDensity Grid2D Pixel Centre 1:\")\n", - "print(mapper.source_plane_mesh_grid[0])\n", - "print(\"RectangularAdaptDensity Grid2D Pixel Centre 2:\")\n", - "print(mapper.source_plane_mesh_grid[1])\n", - "print(\"RectangularAdaptDensity Grid2D Pixel Centre 3:\")\n", - "print(mapper.source_plane_mesh_grid[2])\n", - "print(\"etc.\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can plot these centre on our grid, to make it look slightly less boring!" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "plot_mapper(\n", - " mapper=mapper,\n", - " mesh_grid=mapper.source_plane_mesh_grid,\n", - " title=\"Recntagular Grid With Pixel Cenres\",\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `Mapper` also has the grid that we passed when we set it up. Lets check they`re the same." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(\"Source Grid2D Pixel 1\")\n", - "print(grid[0])\n", - "print(mapper.source_plane_data_grid[0])\n", - "print(\"Source Grid2D Pixel 2\")\n", - "print(grid[1])\n", - "print(mapper.source_plane_data_grid[1])\n", - "print(\"etc.\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can over-lay this grid on the figure, which is starting to look a bit less boring now!" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "plot_mapper(\n", - " mapper=mapper,\n", - " mesh_grid=mapper.source_plane_data_grid,\n", - " title=\"Even less Boring Grid2D of RectangularAdaptDensity Pixels\",\n", - ")\n", - "\n", - "plot_mapper(\n", - " mapper=mapper,\n", - " mesh_grid=mapper.source_plane_data_grid,\n", - " title=\"Zoomed Grid2D of RectangularAdaptDensity Pixels\",\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Finally, the mapper`s `mesh_grid` has lots of information about the pixelization, for example, the arc-second \n", - "size and dimensions." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(mapper.source_plane_mesh_grid.geometry.shape_native_scaled)\n", - "print(mapper.source_plane_mesh_grid.geometry.scaled_maxima)\n", - "print(mapper.source_plane_mesh_grid.geometry.scaled_minima)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Wrap Up__\n", - "\n", - "This was a relatively gentle overview of pixelizations, but one that was hopefully easy to follow. Think about the \n", - "following questions before moving on to the next tutorial:\n", - "\n", - " 1) The rectangular pixelization`s edges are aligned with the most exterior coordinates of the source-grid. This is \n", - " intentional, why do you think this is?" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_4_pixelizations/tutorial_2_mappers.ipynb b/notebooks/howtogalaxy/chapter_4_pixelizations/tutorial_2_mappers.ipynb deleted file mode 100644 index ba60fee6..00000000 --- a/notebooks/howtogalaxy/chapter_4_pixelizations/tutorial_2_mappers.ipynb +++ /dev/null @@ -1,323 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tutorial 2: Mappers\n", - "===================\n", - "\n", - "In the previous tutorial, we used a pixelization to create made a `Mapper`. However, it was not clear what a `Mapper`\n", - "does, why it was called a mapper and whether it was mapping anything at all!\n", - "\n", - "Therefore, in this tutorial, we'll cover mappers in more detail.\n", - "\n", - "__Contents__\n", - "\n", - "**Initial Setup:** Load the dataset for illustration.\n", - "**Mappers:** Understand how mappers map image-plane pixels to pixelization pixels.\n", - "**Mask:** Apply a mask and see how it affects the mapper.\n", - "**Wrap Up:** Summary of mapper concepts." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autoconf import setup_notebook; setup_notebook()\n", - "\n", - "from pathlib import Path\n", - "import autogalaxy as ag\n", - "import autogalaxy.plot as aplt\n", - "import autoarray.plot as aaplt" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Initial Setup__\n", - "\n", - "we'll use complex galaxy data, where:\n", - "\n", - " - The galaxy's bulge is an `Sersic`.\n", - " - The galaxy's disk is an `Exponential`.\n", - " - The galaxy's has four star forming clumps which are `Sersic` profiles." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_name = \"simple\"\n", - "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset Auto-Simulation__\n", - "\n", - "If the dataset does not already exist on your system, it will be created by running the corresponding\n", - "simulator script. This ensures that all example scripts can be run without manually simulating data first." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "if not dataset_path.exists():\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.run(\n", - " [sys.executable, \"scripts/imaging/simulator.py\"],\n", - " check=True,\n", - " )\n", - "\n", - "dataset = ag.Imaging.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " pixel_scales=0.1,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now, lets set up our `Grid2D` (using the image above)." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "grid = ag.Grid2D.uniform(\n", - " shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Mappers__\n", - "\n", - "We now setup a `Pixelization` and use it to create a `Mapper` via the plane`s source-plane grid, just like we did in\n", - "the previous tutorial.\n", - "\n", - "We will make its pixelization resolution half that of the grid above." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "mesh = ag.mesh.RectangularAdaptDensity(\n", - " shape=(dataset.shape_native[0] / 2, dataset.shape_native[1] / 2)\n", - ")\n", - "\n", - "pixelization = ag.Pixelization(mesh=mesh)\n", - "\n", - "interpolator = mesh.interpolator_from(\n", - " source_plane_data_grid=grid, source_plane_mesh_grid=None\n", - ")\n", - "\n", - "mapper = ag.Mapper(interpolator=interpolator)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We now plot the `Mapper` alongside the image we used to generate the source-plane grid.\n", - "\n", - "Using the `Visuals2D` object we are also going to highlight specific grid coordinates certain colors, such that we\n", - "can see how they map from the image grid to the pixelization grid. " - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "indexes = [range(250), [150, 250, 350, 450, 550, 650, 750, 850, 950, 1050]]\n", - "\n", - "aaplt.subplot_image_and_mapper(mapper=mapper, image=dataset.data)\n" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Using a mapper, we can now make these mappings appear the other way round. That is, we can input a pixelization pixel\n", - "index (of our rectangular grid) and highlight how all of the image-pixels that it contains map to the image-plane. \n", - "\n", - "Lets map source pixel 313, the central source-pixel, to the image. We observe that for a given rectangular pixelization\n", - "pixel, there are four image pixels." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "pix_indexes = [[312]]\n", - "\n", - "indexes = mapper.slim_indexes_for_pix_indexes(pix_indexes=pix_indexes)\n", - "\n", - "aaplt.subplot_image_and_mapper(mapper=mapper, image=dataset.data)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Okay, so I think we can agree, mapper's map things! More specifically, they map pixelization pixels to multiple pixels \n", - "in the observed image of a galaxy.\n", - "\n", - "__Mask__\n", - "\n", - "Finally, lets repeat the steps that we performed above, but now using a masked image. By applying a `Mask2D`, the \n", - "mapper only maps image-pixels that are not removed by the mask. This removes the (many) image pixels at the edge of the \n", - "image, where the galaxy is not present.\n", - "\n", - "Lets just have a quick look at these edges pixels:\n", - "\n", - "Lets use an circular `Mask2D`, which will capture the central galaxy light and clumps." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "mask = ag.Mask2D.circular(\n", - " shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=2.0\n", - ")\n", - "\n", - "dataset = dataset.apply_mask(mask=mask)\n", - "aplt.plot_array(array=dataset.data, title=\"Data\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can now use the masked grid to create a new `Mapper` (using the same rectangular 25 x 25 pixelization \n", - "as before)." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "interpolator = mesh.interpolator_from(\n", - " source_plane_data_grid=dataset.grids.pixelization, source_plane_mesh_grid=None\n", - ")\n", - "mapper = ag.Mapper(interpolator=interpolator)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Lets plot it." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aaplt.subplot_image_and_mapper(mapper=mapper, image=dataset.data)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "First, We can see a red circle of dots in both the image and pixelization, showing where the edge of the mask\n", - "maps too in the pixelization.\n", - "\n", - "Now lets show that when we plot pixelization pixel indexes, they still appear in the same place in the image." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "pix_indexes = [[312], [314], [316], [318]]\n", - "\n", - "indexes = mapper.slim_indexes_for_pix_indexes(pix_indexes=pix_indexes)\n", - "\n", - "aaplt.subplot_image_and_mapper(mapper=mapper, image=dataset.data)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Wrap Up__\n", - "\n", - "In this tutorial, we learnt about mappers, and we used them to understand how the image and pixelization map to one \n", - "another. Your exercises are:\n", - " \n", - " 1) Think about how this could help us actually model galaxies. We have said we're going to reconstruct our galaxies \n", - " on the pixel-grid. So, how does knowing how each pixel maps to the image actually help us? If you`ve not got \n", - " any bright ideas, then worry not, that exactly what we're going to cover in the next tutorial." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_4_pixelizations/tutorial_3_inversions.ipynb b/notebooks/howtogalaxy/chapter_4_pixelizations/tutorial_3_inversions.ipynb deleted file mode 100644 index 7909c9fc..00000000 --- a/notebooks/howtogalaxy/chapter_4_pixelizations/tutorial_3_inversions.ipynb +++ /dev/null @@ -1,394 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tutorial 3: Inversions\n", - "======================\n", - "\n", - "In the previous two tutorials, we introduced:\n", - "\n", - " - `Pixelization`'s: which place a pixel-grid over the image data.\n", - " - `Mappers`'s: which describe how each pixelization pixel maps to one or more image pixels.\n", - "\n", - "However, non of this has actually helped us fit galaxy data or reconstruct the galaxy. This is the subject\n", - "of this tutorial, where the process of reconstructing the galaxy's light on the pixelization is called an `Inversion`.\n", - "\n", - "__Contents__\n", - "\n", - "**Initial Setup:** Load the dataset for illustration.\n", - "**Pixelization:** Create a pixelization and perform an inversion to reconstruct the galaxy.\n", - "**Positive Only Solver:** Ensure the reconstruction has only positive intensity values.\n", - "**Wrap Up:** Summary of inversion concepts.\n", - "**Detailed Explanation:** In-depth explanation of the linear algebra behind inversions." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autoconf import setup_notebook; setup_notebook()\n", - "\n", - "from pathlib import Path\n", - "import autogalaxy as ag\n", - "import autogalaxy.plot as aplt\n", - "from autoarray.inversion.plot.inversion_plots import subplot_of_mapper" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Initial Setup__\n", - "\n", - "we'll use the same complex galaxy data as the previous tutorial, where:\n", - "\n", - " - The galaxy's bulge is an `Sersic`.\n", - " - The galaxy's disk is an `Exponential`.\n", - " - The galaxy's has four star forming clumps which are `Sersic` profiles." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_name = \"simple\"\n", - "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset Auto-Simulation__\n", - "\n", - "If the dataset does not already exist on your system, it will be created by running the corresponding\n", - "simulator script. This ensures that all example scripts can be run without manually simulating data first." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "if not dataset_path.exists():\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.run(\n", - " [sys.executable, \"scripts/imaging/simulator.py\"],\n", - " check=True,\n", - " )\n", - "\n", - "dataset = ag.Imaging.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " pixel_scales=0.1,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Lets create a circular mask which contains the galaxy's emission:" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "mask = ag.Mask2D.circular(\n", - " shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=2.0\n", - ")\n", - "\n", - "aplt.plot_array(array=dataset.data, title=\"Data\", mask=mask)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We now create the masked imaging, as we did in the previous tutorial." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset = dataset.apply_mask(mask=mask)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "we again use the rectangular pixelization to create the mapper.\n", - "\n", - "(Ignore the regularization input below for now, we will cover this in the next tutorial)." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "mesh = ag.mesh.RectangularAdaptDensity(shape=dataset.shape_native)\n", - "\n", - "pixelization = ag.Pixelization(mesh=mesh)\n", - "\n", - "interpolator = mesh.interpolator_from(\n", - " source_plane_data_grid=dataset.grids.pixelization,\n", - " source_plane_mesh_grid=None,\n", - ")\n", - "\n", - "mapper = ag.Mapper(\n", - " interpolator=interpolator,\n", - " regularization=ag.reg.Constant(coefficient=1.0),\n", - ")\n", - "\n", - "aplt.subplot_image_and_mapper(mapper=mapper, image=dataset.data)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Pixelization__\n", - "\n", - "Finally, we can now use the `Mapper` to reconstruct the galaxy via an `Inversion`. I'll explain how this works in a \n", - "second, but lets just go ahead and create the inversion first. " - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "inversion = ag.Inversion(dataset=dataset, linear_obj_list=[mapper])" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The inversion has reconstructed the galaxy's light on the rectangular pixel grid, which is called the \n", - "`reconstruction`. \n", - "\n", - "This reconstruction can be mapped back to the same resolution as the image to produce the `mapped_reconstructed_operated_data`." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(inversion.reconstruction)\n", - "print(inversion.mapped_reconstructed_operated_data)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Both of these can be plotted using `subplot_of_mapper`.\n", - "\n", - "It is possible for an inversion to have multiple `Mapper`'s, therefore for certain figures we specify the index \n", - "of the mapper we wish to plot. In this case, because we only have one mapper we specify the index 0." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(array=inversion.reconstruction_to_native, title=\"Reconstruction\")\n", - "subplot_of_mapper(inversion=inversion, mapper_index=0)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "There we have it, we have successfully reconstructed the galaxy using a rectangular pixel-grid. This has reconstructed\n", - "the complex blobs of light of the galaxy.\n", - "\n", - "Pretty great, huh? If you ran the complex source pipeline in chapter 3, you'll remember that getting a model image \n", - "that looked this good simply *was not possible*. With an inversion, we can do this with ease and without having to \n", - "perform model-fitting with 20+ parameters for the galaxy's light!\n", - "\n", - "We will now briefly discuss how an inversion actually works, however the explanation I give in this tutorial will be \n", - "overly-simplified. To be good at modeling you do not need to understand the details of how an inversion works, you \n", - "simply need to be able to use an inversion to model a galaxy. \n", - "\n", - "To begin, lets consider some random mappings between our mapper`s pixelization pixels and the image." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "pix_indexes = [[445], [285], [313], [132], [11]]\n", - "\n", - "indexes = mapper.slim_indexes_for_pix_indexes(pix_indexes=pix_indexes)\n", - "\n", - "aplt.subplot_image_and_mapper(mapper=mapper, image=dataset.data)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "These mappings are known before the inversion reconstructs the galaxy, which means before this inversion is performed \n", - "we know two key pieces of information:\n", - "\n", - " 1) The mappings between every pixelization pixel and a set of image-pixels.\n", - " 2) The flux values in every observed image-pixel, which are the values we want to fit successfully.\n", - "\n", - "It turns out that with these two pieces of information we can linearly solve for the set of pixelization pixel fluxes \n", - "that best-fit (e.g. maximize the log likelihood) our observed image. Essentially, we set up the mappings between\n", - "pixelization and image pixels as a large matrix and solve for the pixelization pixel fluxes in an analogous fashion to \n", - "how you would solve a set of simultaneous linear equations. This process is called a `linear inversion`.\n", - "\n", - "There are three more things about a linear inversion that are worth knowing:\n", - "\n", - " 1) When performing fits using light profiles, we discussed how a `model_image` was generated by convolving the light\n", - " profile images with the data's PSF. A similar blurring operation is incorporated into the inversion, such that it \n", - " reconstructs a galaxy (and therefore image) which fully accounts for the telescope optics and effect of the PSF.\n", - "\n", - " 2) You may be familiar with image sub-gridding, which splits each image-pixel into a sub-pixel (if you are not \n", - " familiar then feel free to checkout the optional **HowToGalaxy** tutorial on sub-gridding. If a sub-grid is used, it is \n", - " the mapping between every sub-pixel -pixel that is computed and used to perform the inversion. This prevents \n", - " aliasing effects degrading the image reconstruction. By default **PyAutoGalaxy** uses sub-gridding of degree 4x4.\n", - "\n", - " 3) The inversion`s solution is regularized. But wait, that`s what we'll cover in the next tutorial!\n", - "\n", - "Finally, let me show you how easy it is to fit an image with an `Inversion` using a `FitImaging` object. Instead of \n", - "giving the galaxy a light profile, we simply pass it a `Pixelization` and regularization, and pass it to a \n", - "galaxies." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "pixelization = ag.Pixelization(\n", - " mesh=ag.mesh.RectangularAdaptDensity(shape=(25, 25)),\n", - " regularization=ag.reg.Constant(coefficient=1.0),\n", - ")\n", - "\n", - "galaxy = ag.Galaxy(redshift=1.0, pixelization=pixelization)\n", - "\n", - "galaxies = ag.Galaxies(galaxies=[galaxy])" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Then, like before, we pass the imaging and galaxies `FitImaging` object. \n", - "\n", - "We see some pretty good looking residuals, albeit there is faint flux leftover. We will consider how we can address \n", - "this in the next tutorial. \n", - "\n", - "We can use the `subplot_of_galaxies` method to specifically visualize the inversion and plot the reconstruction." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "fit = ag.FitImaging(dataset=dataset, galaxies=galaxies)\n", - "\n", - "aplt.subplot_fit_imaging(fit=fit)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Positive Only Solver__\n", - "\n", - "All pixelized source reconstructions use a positive-only solver, meaning that every source-pixel is only allowed\n", - "to reconstruct positive flux values. This ensures that the source reconstruction is physical and that we don't\n", - "reconstruct negative flux values that don't exist in the real source galaxy (a common systematic solution in lens\n", - "analysis).\n", - "\n", - "It may be surprising to hear that this is a feature worth pointing out, but it turns out setting up the linear algebra\n", - "to enforce positive reconstructions is difficult to make efficient. A lot of development time went into making this\n", - "possible, where a bespoke fast non-negative linear solver was developed to achieve this.\n", - "\n", - "Other methods in the literature often do not use a positive only solver, and therefore suffer from these \n", - "unphysical solutions, which can degrade the results of lens model in general.\n", - "\n", - "__Wrap Up__\n", - "\n", - "And, we're done, here are a few questions to get you thinking about inversions:\n", - "\n", - " 1) The inversion provides the maximum log likelihood solution to the observed image. Is there a problem with seeking \n", - " the highest likelihood solution? Is there a risk that we're going to fit other things in the image than just the \n", - " galaxy? What happens if you reduce the `coefficient` of the regularization object above to zero?\n", - "\n", - " 2) The exterior pixels in the rectangular pixel-grid have no image-pixels in them. However, they are still given a \n", - " reconstructed flux. Given these pixels do not map to the data, where is this value coming from?\n", - " \n", - "__Detailed Explanation__\n", - "\n", - "If you are interested in a more detailed description of how inversions work, then checkout the file\n", - "`autogalaxy_workspace/*/imaging/log_likelihood_function/inversion.ipynb` which gives a visual step-by-step\n", - "guide of the process alongside equations and references to literature on the subject." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_4_pixelizations/tutorial_4_bayesian_regularization.ipynb b/notebooks/howtogalaxy/chapter_4_pixelizations/tutorial_4_bayesian_regularization.ipynb deleted file mode 100644 index cce58410..00000000 --- a/notebooks/howtogalaxy/chapter_4_pixelizations/tutorial_4_bayesian_regularization.ipynb +++ /dev/null @@ -1,435 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tutorial 4: Bayesian Regularization\n", - "===================================\n", - "\n", - "So far, we have:\n", - "\n", - " - Used pixelizations and mappers to map pixelization pixels to image-pixels and visa versa.\n", - " - Successfully used an inversion to reconstruct a galaxy.\n", - " - Seen that this reconstruction provides a good fit of the observed image, providing a high likelihood solution.\n", - "\n", - "The explanation of *how* an inversion works has so far been overly simplified. You'll have noted the regularization\n", - "inputs which we have not so far discussed. This will be the topic of this tutorial, and where inversions become more\n", - "conceptually challenging!\n", - "\n", - "__Contents__\n", - "\n", - "**Initial Setup:** Load the dataset for illustration.\n", - "**Convenience Function:** A helper function for performing inversions.\n", - "**Pixelization:** Perform inversions with different regularization coefficients.\n", - "**Regularization:** Understand how regularization smooths the reconstruction.\n", - "**Bayesian Evidence:** Use the Bayesian evidence to objectively choose the regularization coefficient.\n", - "**Non-Linear and Linear:** Discussion of how regularization interacts with the non-linear search.\n", - "**Detailed Description:** In-depth explanation of how the Bayesian evidence penalizes overfitting." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autoconf import setup_notebook; setup_notebook()\n", - "\n", - "from pathlib import Path\n", - "import autogalaxy as ag\n", - "import autogalaxy.plot as aplt\n", - "from autoarray.inversion.plot.inversion_plots import subplot_of_mapper" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Initial Setup__\n", - "\n", - "we'll use the same complex galaxy data as the previous tutorial, where:\n", - "\n", - " - The galaxy's bulge is an `Sersic`.\n", - " - The galaxy's disk is an `Exponential`.\n", - " - The galaxy's has four star forming clumps which are `Sersic` profiles." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_name = \"simple\"\n", - "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset Auto-Simulation__\n", - "\n", - "If the dataset does not already exist on your system, it will be created by running the corresponding\n", - "simulator script. This ensures that all example scripts can be run without manually simulating data first." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "if not dataset_path.exists():\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.run(\n", - " [sys.executable, \"scripts/imaging/simulator.py\"],\n", - " check=True,\n", - " )\n", - "\n", - "dataset = ag.Imaging.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " pixel_scales=0.1,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Convenience Function__\n", - "\n", - "we're going to perform a lot of fits using an `Inversion` this tutorial. This would create a lot of code, so to keep \n", - "things tidy, I've setup this function which handles it all for us." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "\n", - "def perform_fit_with_galaxy(dataset, galaxy):\n", - " mask = ag.Mask2D.circular(\n", - " shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=2.0\n", - " )\n", - "\n", - " dataset = dataset.apply_mask(mask=mask)\n", - "\n", - " galaxies = ag.Galaxies(galaxies=[galaxy])\n", - "\n", - " return ag.FitImaging(dataset=dataset, galaxies=galaxies)\n" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Pixelization__\n", - "\n", - "Okay, so lets look at our fit from the previous tutorial in more detail." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "pixelization = ag.Pixelization(\n", - " mesh=ag.mesh.RectangularAdaptDensity(shape=(50, 50)),\n", - " regularization=ag.reg.Constant(coefficient=1.0),\n", - ")\n", - "\n", - "galaxy = ag.Galaxy(redshift=1.0, pixelization=pixelization)\n", - "\n", - "fit = perform_fit_with_galaxy(dataset=dataset, galaxy=galaxy)\n", - "\n", - "aplt.subplot_fit_imaging(fit=fit)\n", - "subplot_of_mapper(inversion=fit.inversion, mapper_index=0)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Regularization__\n", - "\n", - "The galaxy reconstruction looks pretty good! \n", - "\n", - "However, the high quality of this solution was possible because I chose a `coefficient` for the regularization input of\n", - "1.0. If we reduce this `coefficient` to 0.01, the galaxy reconstruction goes *very* weird." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "pixelization = ag.Pixelization(\n", - " mesh=ag.mesh.RectangularAdaptDensity(shape=(50, 50)),\n", - " regularization=ag.reg.Constant(coefficient=0.01),\n", - ")\n", - "\n", - "galaxy = ag.Galaxy(redshift=1.0, pixelization=pixelization)\n", - "\n", - "no_regularization_fit = perform_fit_with_galaxy(dataset=dataset, galaxy=galaxy)\n", - "\n", - "aplt.subplot_fit_imaging(fit=no_regularization_fit)\n", - "subplot_of_mapper(inversion=no_regularization_fit.inversion, mapper_index=0)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "So, what is happening here? Why does reducing the `coefficient` do this to our reconstruction? First, we need\n", - "to understand what regularization actually does!\n", - "\n", - "When the inversion reconstructs the galaxy, it does not *only* compute the set of pixelization pixel fluxes that \n", - "best-fit the image. It also regularizes this solution, whereby it goes to every pixel on the rectangular grid \n", - "and computes the different between the reconstructed flux values of every pixel with its 4 neighboring pixels. \n", - "If the difference in flux is large the solution is penalized, reducing its log likelihood. You can think of this as \n", - "us applying a 'smoothness prior' on the reconstructed galaxy's light.\n", - "\n", - "This smoothing adds a 'penalty term' to the log likelihood of an inversion which is the summed difference between the \n", - "reconstructed fluxes of every pixelization pixel pair multiplied by the `coefficient`. By setting the regularization \n", - "coefficient to zero, we set this penalty term to zero, meaning that regularization is completely omitted.\n", - "\n", - "Why do we need to regularize our solution? We just saw why, if we do not apply this smoothness prior to the galaxy \n", - "reconstruction, we `over-fit` the image and reconstruct a noisy galaxy with lots of extraneous features. This is what \n", - "the aliasing chequer-board effect is caused by. If the inversions's sole aim is to maximize the log likelihood, it can \n", - "do this by fitting *everything* accurately, including the noise.\n", - "\n", - "Over-fitting is why regularization is necessary. Solutions like this will completely ruin our attempts to model a \n", - "galaxy. By smoothing our galaxy reconstruction we ensure it does not over fit noise in the image. \n", - "\n", - "So, what happens if we apply a high value for the regularization coefficient?" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "pixelization = ag.Pixelization(\n", - " mesh=ag.mesh.RectangularAdaptDensity(shape=(50, 50)),\n", - " regularization=ag.reg.Constant(coefficient=100.0),\n", - ")\n", - "\n", - "galaxy = ag.Galaxy(redshift=1.0, pixelization=pixelization)\n", - "\n", - "high_regularization_fit = perform_fit_with_galaxy(dataset=dataset, galaxy=galaxy)\n", - "\n", - "aplt.subplot_fit_imaging(fit=high_regularization_fit)\n", - "subplot_of_mapper(inversion=high_regularization_fit.inversion, mapper_index=0)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The figure above shows that we completely remove over-fitting. However, we now fit the image data less poorly,\n", - "due to the much higher level of smoothing.\n", - "\n", - "So, we now understand what regularization is and why it is necessary. There is one nagging question that remains, how \n", - "do I choose the regularization coefficient value? We can not use the log likelihood, as decreasing the regularization\n", - "coefficient will always increase the log likelihood, because less smoothing allows the reconstruction to fit \n", - "the data better." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(\"Likelihood Without Regularization:\")\n", - "print(no_regularization_fit.log_likelihood_with_regularization)\n", - "print(\"Likelihood With Normal Regularization:\")\n", - "print(fit.log_likelihood_with_regularization)\n", - "print(\"Likelihood With High Regularization:\")\n", - "print(high_regularization_fit.log_likelihood_with_regularization)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Bayesian Evidence__\n", - "\n", - "For inversions, we therefore need a different goodness-of-fit measure to choose the appropriate level of regularization. \n", - "\n", - "For this, we invoke the `Bayesian Evidence`, which quantifies the goodness of the fit as follows:\n", - "\n", - " - It requires that the residuals of the fit are consistent with Gaussian noise (which is the type of noise expected \n", - " in the imaging data). If this Gaussian pattern is not visible in the residuals, the noise must have been over-fitted\n", - " by the inversion. The Bayesian evidence will therefore decrease. If the image is fitted poorly due to over smoothing, \n", - " the residuals will again not appear Gaussian either, again producing a decrease in the Bayesian evidence value.\n", - "\n", - " - There can be many solutions which fit the data to the noise level, without over-fitting. To determine the best \n", - " solutions from these solutions, the Bayesian evidence therefore also quantifies the complexity of the galaxy \n", - " reconstruction. If an inversion requires many pixels and a low level of regularization to achieve a good fit, the \n", - " Bayesian evidence will decrease. The evidence penalizes solutions which are complex, which, in a Bayesian sense, are \n", - " less probable (you may want to look up `Occam`s Razor`).\n", - "\n", - "The Bayesian evidence therefore ensures we only invoke a more complex galaxy reconstruction when the data absolutely \n", - "necessitates it.\n", - "\n", - "Lets take a look at the Bayesian evidence of the fits that we performed above, which is accessible from a `FitImaging` \n", - "object via the `log_evidence` property:" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(\"Bayesian Evidence Without Regularization:\")\n", - "print(no_regularization_fit.log_evidence)\n", - "print(\"Bayesian Evidence With Normal Regularization:\")\n", - "print(fit.log_evidence)\n", - "print(\"Bayesian Evidence With High Regularization:\")\n", - "print(high_regularization_fit.log_evidence)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "As expected, the solution that we could see `by-eye` was the best solution corresponds to the highest log evidence \n", - "solution.\n", - "\n", - "__Non-Linear and Linear__\n", - "\n", - "Before we end, lets consider which aspects of an inversion are linear and which are non-linear.\n", - "\n", - "The linear part of the inversion is the step that solves for the reconstruct pixelization pixel fluxes, including \n", - "accounting for the smoothing via regularizaton. We do not have to perform a non-linear search to determine the pixel\n", - "fluxes or compute the Bayesian evidence discussed above.\n", - "\n", - "However, determining the regularization `coefficient` that maximizes the Bayesian log evidence is a non-linear problem \n", - "that requires a non-linear search. The Bayesian evidence also depends on the grid resolution, which means the \n", - "pixel-grid's `shape` parameter may also now become dimensions of non linear parameter space (albeit it is common\n", - "practise for us to simply use the resolution of the image data, or a multiple of this). \n", - "\n", - "Nevertheless, these total only 3 non-linear parameters, far fewer than the 20+ that are required when modeling such a\n", - "complex galaxy using light profiles for every individual clump! \n", - "\n", - "Here are a few questions for you to think about.\n", - "\n", - " 1) We maximize the log evidence by using simpler galaxy reconstructions. Therefore, decreasing the pixel-grid \n", - " size should provide a higher log_evidence, provided it still has sufficiently high resolution to fit the image well \n", - " (and provided that the regularization coefficient is set to an appropriate value). Can you increase the log evidence \n", - " from the value above by changing these parameters, I've set you up with a code to do so below." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "pixelization = ag.Pixelization(\n", - " mesh=ag.mesh.RectangularAdaptDensity(shape=(50, 50)),\n", - " regularization=ag.reg.Constant(coefficient=1.0),\n", - ")\n", - "\n", - "galaxy = ag.Galaxy(redshift=1.0, pixelization=pixelization)\n", - "\n", - "fit = perform_fit_with_galaxy(dataset=dataset, galaxy=galaxy)\n", - "\n", - "print(\"Previous Bayesian Evidence:\")\n", - "print(3988.0716851250163)\n", - "print(\"New Bayesian Evidence:\")\n", - "print(fit.log_evidence)\n", - "\n", - "aplt.subplot_fit_imaging(fit=fit)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Detailed Description__\n", - "\n", - "Below, I provide a more detailed discussion of the Bayesian evidence. It is not paramount that you understand this to\n", - "use **PyAutoGalaxy**, but I recommend you give it a read to get an intuition for how the evidence works.\n", - "\n", - "The Bayesian log evidence quantifies the following 3 aspects of a fit to galaxy imaging data:\n", - "\n", - "1) *The quality of the image reconstruction:* The galaxy reconstruction is a linear inversion which uses the observed\n", - " values in the image-data to fit it and reconstruct the galaxy. It is in principle able to perfectly reconstruct the\n", - " image regardless of the image\u2019s noise or the accuracy of the model (e.g. at infinite resolution without\n", - " regularization). The problem is therefore \u2018ill-posed\u2019 and this is why regularization is necessary.\n", - "\n", - " However, this raises the question of what constitutes a \u2018good\u2019 solution? The Bayesian evidence defines this by\n", - " assuming that the image data consists of independent Gaussian noise in every image pixel. A \u2018good\u2019 solution is one\n", - " whose chi-squared residuals are consistent with Gaussian noise, producing a reduced chi-squared near 1.0 .Solutions\n", - " which give a reduced chi squared below 1 are penalized for being overly complex and fitting the image\u2019s noise, whereas\n", - " solutions with a reduced chi-squared above are penalized for not invoking a more complex galaxy model when the data it\n", - " is necessary to fit the data bettter. In both circumstances, these penalties reduce the inferred Bayesian evidence!\n", - "\n", - "2) *The complexity of the galaxy reconstruction:* The log evidence estimates the number of pixelization pixels that are used \n", - " to reconstruct the image, after accounting for their correlation with one another due to regularization. Solutions that\n", - " require fewer correlated galaxy pixels increase the Bayesian evidence. Thus, simpler and less complex galaxy \n", - " reconstructions are favoured.\n", - "\n", - "3) *The signal-to-noise (S/N) of the image that is fitted:* The Bayesian evidence favours models which fit higher S/N\n", - " realizations of the observed data (where the S/N is determined using the image-pixel variances, e.g. the noise-map). Up \n", - " to now, all **PyAutoGalaxy** fits assumed fixed variances, meaning that this aspect of the Bayeisan evidence has no impact \n", - " on the inferred evidence values. \n", - " \n", - " The premise is that whilst increasing the variances of image pixels lowers their S/N values and therefore also\n", - " decreases the log evidence, doing so may produce a net increase in log evidence. This occurs when the chi-squared \n", - " values of the image pixels whose variances are increased were initially very high (e.g. they were fit poorly by the \n", - " model).\n", - "\n", - "In summary, the log evidence is maximized for solutions which most accurately reconstruct the highest S/N realization of\n", - "the observed image, without over-fitting its noise and using the fewest correlated pixelization pixels. By employing \n", - "this framework throughout, **PyAutoGalaxy** objectively determines the final model following the principles of Bayesian\n", - "analysis and Occam\u2019s Razor." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_4_pixelizations/tutorial_5_model_fit.ipynb b/notebooks/howtogalaxy/chapter_4_pixelizations/tutorial_5_model_fit.ipynb deleted file mode 100644 index f184585c..00000000 --- a/notebooks/howtogalaxy/chapter_4_pixelizations/tutorial_5_model_fit.ipynb +++ /dev/null @@ -1,359 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tutorial 5: Model-Fit\n", - "=====================\n", - "\n", - "In the previous tutorials we used an inversion to reconstruct a complex galaxy. However, from the perspective of\n", - "a scientific analysis, it is not clear how useful this was. When I fit a galaxy with light profiles, I learn about\n", - "its brightness (`intensity`), size (`effective_radius`), compactness (`sersic_index`), etc.\n", - "\n", - "What did I learn about the galaxy I reconstructed? Not a lot, perhaps.\n", - "\n", - "Inversions are most useful when combined with light profiles. For the complex galaxy we used throughout this tutorial,\n", - "we can fit it with light profiles to quantify the properties of its `bulge` and `disk` components, whilst\n", - "simultaneously fitting the clumps with the inversion so as to ensure they do not impact the fit.\n", - "\n", - "To illustrate modeling using an inversion this tutorial therefore revisits the complex galaxy model-fit that we\n", - "performed in tutorial 4 of chapter 3. This time, as you have probably guessed, we will fit part of the complex galaxy\n", - "using an inversion.\n", - "\n", - "We will use search chaining to do this, first fitting the main galaxy components with light profiles, thereby\n", - "initializing the bulge and disk components. In the later searches we will switch to an `Inversion`.\n", - "\n", - "__Contents__\n", - "\n", - "**Initial Setup:** Load the complex galaxy dataset and apply a mask.\n", - "**Model + Search + Analysis + Model-Fit (Search 1):** Fit light profiles to the main galaxy components.\n", - "**Mesh Shape:** Discussion of how mesh shape affects the inversion.\n", - "**Model + Search + Analysis + Model-Fit (Search 2):** Fit with a pixelization for residual structure.\n", - "**Model + Search (Search 3):** Final simultaneous fit of light profiles and pixelization.\n", - "**Wrap Up:** Summary of combining light profiles with pixelizations via search chaining." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autoconf import setup_notebook; setup_notebook()\n", - "\n", - "from pathlib import Path\n", - "import autofit as af\n", - "import autogalaxy as ag\n", - "import autogalaxy.plot as aplt" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Initial Setup__\n", - "\n", - "we'll use complex galaxy data, where:\n", - "\n", - " - The galaxy's bulge is an `Sersic`.\n", - " - The galaxy's disk is an `Exponential`.\n", - " - The galaxy's has four star forming clumps which are `Sersic` profiles." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_name = \"simple\"\n", - "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset Auto-Simulation__\n", - "\n", - "If the dataset does not already exist on your system, it will be created by running the corresponding\n", - "simulator script. This ensures that all example scripts can be run without manually simulating data first." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "if not dataset_path.exists():\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.run(\n", - " [sys.executable, \"scripts/imaging/simulator.py\"],\n", - " check=True,\n", - " )\n", - "\n", - "dataset = ag.Imaging.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " pixel_scales=0.1,\n", - ")\n", - "\n", - "mask = ag.Mask2D.circular(\n", - " shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=2.0\n", - ")\n", - "\n", - "\n", - "dataset = dataset.apply_mask(mask=mask)\n", - "\n", - "over_sample_size = ag.util.over_sample.over_sample_size_via_radial_bins_from(\n", - " grid=dataset.grid,\n", - " sub_size_list=[8, 4, 1],\n", - " radial_list=[0.3, 0.6],\n", - " centre_list=[(0.0, 0.0)],\n", - ")\n", - "\n", - "dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size)\n", - "\n", - "aplt.subplot_imaging_dataset(dataset=dataset)\n" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Model + Search + Analysis + Model-Fit (Search 1)__\n", - "\n", - "Search 1 we fit a model where:\n", - "\n", - " - The galaxy's bulge is an `Sersic` with fixed centre [5 parameters].\n", - " \n", - " - The galaxy's disk is an `Exponential` with fixed centre [4 parameters].\n", - " \n", - "The number of free parameters and therefore the dimensionality of non-linear parameter space is N=9." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "bulge = af.Model(ag.lp.Sersic)\n", - "disk = af.Model(ag.lp.Exponential)\n", - "\n", - "bulge.centre_0 = 0.0\n", - "bulge.centre_1 = 0.0\n", - "disk.centre_0 = 0.0\n", - "disk.centre_1 = 0.0\n", - "\n", - "galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=bulge, disk=disk)\n", - "\n", - "model_1 = af.Collection(galaxies=af.Collection(galaxy=galaxy))\n", - "\n", - "search_1 = af.Nautilus(\n", - " path_prefix=Path(\"howtogalaxy\", \"chapter_4\"),\n", - " name=\"search[1]\",\n", - " unique_tag=dataset_name,\n", - " n_live=100,\n", - " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - ")\n", - "\n", - "analysis_1 = ag.AnalysisImaging(dataset=dataset, use_jax=True)\n", - "\n", - "result_1 = search_1.fit(model=model_1, analysis=analysis_1)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Mesh Shape__\n", - "\n", - "The `mesh_shape` parameter defines number of pixels used by the rectangular mesh to reconstruct the source,\n", - "set below to 28 x 28. \n", - "\n", - "The `mesh_shape` must be fixed before modeling and cannot be a free parameter of the model, because JAX uses the\n", - "mesh shape to define static shaped arrays which use the mesh to reconstruct the source. For a rectangular\n", - "mesh, the same number of pixels must be used in the y and x directions." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "mesh_pixels_yx = 28\n", - "mesh_shape = (mesh_pixels_yx, mesh_pixels_yx)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Model + Search + Analysis + Model-Fit (Search 2)__\n", - "\n", - "We use the results of search 1 to create the model fitted in search 2, where:\n", - "\n", - " - The galaxy's bulge is an `Sersic` [0 parameters: parameters fixed from search 1].\n", - " \n", - " - The galaxy's disk is an `Exponential` [0 parameters: parameters fixed from search 1].\n", - "\n", - " - The galaxy's clumps are reconstructed `RectangularAdaptDensity` mesh with resolution as free parameters [2 parameters].\n", - "\n", - " - This pixelization is regularized using a `Constant` scheme [1 parameter]. \n", - "\n", - "The number of free parameters and therefore the dimensionality of non-linear parameter space is N=3.\n", - "\n", - "This search allows us to very efficiently set up the resolution of the mesh and regularization coefficient \n", - "of the regularization scheme, before using these models to refit the galaxy mass model." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "pixelization = af.Model(\n", - " ag.Pixelization,\n", - " mesh=ag.mesh.RectangularAdaptDensity(shape=mesh_shape),\n", - " regularization=ag.reg.GaussianKernel,\n", - ")\n", - "\n", - "galaxy = af.Model(\n", - " ag.Galaxy,\n", - " redshift=0.5,\n", - " bulge=result_1.instance.galaxies.galaxy.bulge,\n", - " disk=result_1.instance.galaxies.galaxy.disk,\n", - " pixelization=pixelization,\n", - ")\n", - "\n", - "model_2 = af.Collection(galaxies=af.Collection(galaxy=galaxy))\n", - "\n", - "search_2 = af.Nautilus(\n", - " path_prefix=Path(\"howtogalaxy\", \"chapter_4\"),\n", - " name=\"search[2]\",\n", - " unique_tag=dataset_name,\n", - " n_live=50,\n", - " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - ")\n", - "\n", - "analysis_2 = ag.AnalysisImaging(\n", - " dataset=dataset,\n", - " settings=ag.Settings(use_border_relocator=True),\n", - " use_jax=True,\n", - ")\n", - "\n", - "result_2 = search_2.fit(model=model_2, analysis=analysis_2)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Model + Search (Search 3)__\n", - "\n", - "We use the results of searches 1 and 2 to create the model fitted in search 3, where:\n", - "\n", - " - The galaxy's bulge is an `Sersic` [7 parameters: priors initialized from search 1].\n", - " \n", - " - The galaxy's disk is an `Exponential` [6 parameters: priors initialized from search 1].\n", - "\n", - " - The galaxy's light uses a `RectangularAdaptDensity` mesh[parameters fixed to results of search 2].\n", - "\n", - " - This pixelization is regularized using a `Constant` scheme [parameters fixed to results of search 2]. \n", - "\n", - "The number of free parameters and therefore the dimensionality of non-linear parameter space is N=13.\n", - "\n", - "This search allows us to refit the bulge and disk components with an inversion that takes care of the clumps." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "bulge = af.Model(ag.lp.Sersic)\n", - "bulge.ell_comps = result_1.model.galaxies.galaxy.bulge.ell_comps\n", - "bulge.intensity = result_1.model.galaxies.galaxy.bulge.intensity\n", - "bulge.effective_radius = result_1.model.galaxies.galaxy.bulge.effective_radius\n", - "bulge.sersic_index = result_1.model.galaxies.galaxy.bulge.sersic_index\n", - "\n", - "disk = af.Model(ag.lp.Sersic)\n", - "disk.ell_comps = result_1.model.galaxies.galaxy.disk.ell_comps\n", - "disk.intensity = result_1.model.galaxies.galaxy.disk.intensity\n", - "disk.effective_radius = result_1.model.galaxies.galaxy.disk.effective_radius\n", - "\n", - "pixelization = af.Model(\n", - " ag.Pixelization,\n", - " mesh=ag.mesh.RectangularAdaptDensity(shape=mesh_shape),\n", - " regularization=ag.reg.GaussianKernel,\n", - ")\n", - "\n", - "galaxy = af.Model(\n", - " ag.Galaxy, redshift=0.5, bulge=bulge, disk=disk, pixelization=pixelization\n", - ")\n", - "\n", - "model_3 = af.Collection(galaxies=af.Collection(galaxy=galaxy))\n", - "\n", - "search_3 = af.Nautilus(\n", - " path_prefix=Path(\"howtogalaxy\", \"chapter_4\"),\n", - " name=\"search[3]\",\n", - " unique_tag=dataset_name,\n", - " n_live=100,\n", - " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - ")\n", - "\n", - "analysis_3 = ag.AnalysisImaging(dataset=dataset, use_jax=True)\n", - "\n", - "result_3 = search_3.fit(model=model_3, analysis=analysis_3)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Wrap Up__\n", - "\n", - "And with that, we now have a pipeline to model galaxies using an inversion! " - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/howtogalaxy/chapter_optional/tutorial_searches.ipynb b/notebooks/howtogalaxy/chapter_optional/tutorial_searches.ipynb deleted file mode 100644 index 562f3d62..00000000 --- a/notebooks/howtogalaxy/chapter_optional/tutorial_searches.ipynb +++ /dev/null @@ -1,446 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tutorial: Alternative Searches\n", - "==============================\n", - "\n", - "Up to now, we've always used the non-linear search Nautilus and not considered the input parameters that control its\n", - "sampling. In this tutorial, we'll consider how we can change these setting to balance finding the global maxima\n", - "solution with fast run time.\n", - "\n", - "We will also discuss other types of non-linear searches, such as MCMC and optimizers, which we can use to perform lens\n", - "modeling. So far, we have no found any of these alternatives to give anywhere near as robust and efficient results as\n", - "Nautilus, and we recommend users use Nautilus unless they are particularly interested in investigating different\n", - "model-fitting techniques.\n", - "\n", - "__Contents__\n", - "\n", - "**Nested Sampling:** Customize Nautilus settings and use Dynesty as an alternative nested sampler.\n", - "**Optimizers:** Use maximum likelihood optimizers like LBFGS for fast but less robust fitting.\n", - "**MCMC:** Use Markov Chain Monte Carlo methods like Emcee for parameter estimation." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autoconf import setup_notebook; setup_notebook()\n", - "\n", - "from pathlib import Path\n", - "import autogalaxy as ag\n", - "import autogalaxy.plot as aplt\n", - "import autofit as af" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "we'll use new galaxying data, where:\n", - "\n", - " - The galaxy's light is an `Sersic`.\n", - " - The galaxy's total mass distribution is an `Isothermal` and `ExternalShear`.\n", - " - The source galaxy's `LightProfile` is an `Sersic`." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_name = \"simple__sersic\"\n", - "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset Auto-Simulation__\n", - "\n", - "If the dataset does not already exist on your system, it will be created by running the corresponding\n", - "simulator script. This ensures that all example scripts can be run without manually simulating data first." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "if not dataset_path.exists():\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.run(\n", - " [sys.executable, \"scripts/howtogalaxy/simulators/sersic.py\"],\n", - " check=True,\n", - " )\n", - "\n", - "\n", - "dataset = ag.Imaging.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " pixel_scales=0.1,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "we'll create and use a smaller 2.0\" `Mask2D` again." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "mask = ag.Mask2D.circular(\n", - " shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=2.6\n", - ")\n", - "\n", - "dataset = dataset.apply_mask(mask=mask)\n", - "\n", - "aplt.subplot_imaging_dataset(dataset=dataset)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Nested Sampling__\n", - "\n", - "Lets first perform the model-fit using Nautilus, but look at different parameters that control how long it takes to run. \n", - "We'll therefore discuss in a bit more detail how Nautilus works, but still keep this description conceptually simple \n", - "and avoid technical terms and jargon. For a complete description of Nautilus you should check out the Nautilus \n", - "publication `https://arxiv.org/abs/2306.16923`.\n", - "\n", - "nlive:\n", - "\n", - "Nautilus is a `nested sampling` algorithm. As we described in chapter 2, it throws down a set of `live points` in \n", - "parameter space, where each live point corresponds to a model with a given set of parameters. These points are\n", - "initially distributed according to our priors, hence why tuning our priors allows us to sample parameter space faster.\n", - " \n", - "The number of live points is set by the parameter `n_live`. More points provide a more thorough sampling of \n", - "parameter space, increasing the probability that we locate the global maxima solution. Therefore, if you think your \n", - "model-fit has gone to a local maxima, you should try increasing `n_live`. The downside of this is Nautilus will \n", - "take longer to sample parameter space and converge on a solution. Ideally, we will use as few live points as possible \n", - "to locate the global maxima as quickly as possible.\n", - "\n", - "f_live:\n", - "\n", - "A nested sampling algorithm estimates the *Bayesian Evidence* of the model-fit, which is quantity the non-linear \n", - "search algorithms we introduce later do not. The Bayesian evidence quantifies how well the model as a whole fits\n", - "the data, following a principle called Occam's Razor (`https://simple.wikipedia.org/wiki/Occam%27s_razor`). This \n", - "penalizes models for being more complex (e.g. more parameters) and requires that their additional complexity improve \n", - "their overall fit to the data compared to a simpler model. By computing the comparing the Bayesian evidence of \n", - "different models one can objectively choose the model that best fits the data.\n", - "\n", - "A nested sampling algorithm stops sampling when it estimates that continuing sampling will not increase the Bayesian \n", - "evidence (called the `log_evidence`) by more than the `f_live`. As Nautilus progresses and converges on the\n", - "solution, the rate of increase of the estimated Bayesian evidence slows down. Therefore, higher `f_live`s \n", - "mean Nautilus terminate sooner.\n", - " \n", - "A high `f_live` will make the errors estimated on every parameter unreliable and its value must be kept \n", - "below 0.8 for reliable error estimates. However, when chaining searches, we typically *do not care* about the errors \n", - "in the first search, therefore setting a high evidence tolerance can be an effective means to make Nautilus converge\n", - "faster (we'll estimate reliable errors in the second search when the `f_live is 0.8 or less). \n", - "\n", - "\n", - "Lets perform two fits, where:\n", - "\n", - " - One has many live points and a higher evidence tolerance, causing the non-linear search to\n", - " take a longer time to run.\n", - " \n", - " - One has few live points, a high sampling efficiency and evidence tolerance, causing the non-linear search to\n", - " converge and end quicker." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "model = af.Collection(\n", - " galaxies=af.Collection(\n", - " lens=af.Model(\n", - " ag.Galaxy, redshift=0.5, bulge=ag.lp.Sersic, mass=ag.mp.Isothermal\n", - " ),\n", - " source=af.Model(ag.Galaxy, redshift=1.0, bulge=ag.lp.Sersic),\n", - " )\n", - ")\n", - "\n", - "search = af.Nautilus(\n", - " path_prefix=Path(\"howtogalaxy\", \"chapter_optional\"),\n", - " name=\"tutorial_searches_slow\",\n", - " unique_tag=dataset_name,\n", - " n_live=400,\n", - " n_batch=50, # GPU lens model fits are batched and run simultaneously, see VRAM section below.\n", - ")\n", - "\n", - "analysis = ag.AnalysisImaging(dataset=dataset, use_jax=True)\n", - "\n", - "print(\n", - " \"The non-linear search has begun running - checkout the workspace/output\"\n", - " \" folder for live output of the results, images and model.\"\n", - " \" This Jupyter notebook cell with progress once search has completed - this could take some time!\"\n", - ")\n", - "\n", - "result_slow = search.fit(model=model, analysis=analysis)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Lets check that we get a good model and fit to the data." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_fit_imaging(fit=result_slow.max_log_likelihood_fit)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can use the result to tell us how many iterations Nautilus took to convergence on the solution." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(\"Total Nautilus Iterations (If you skip running the search, this is ~ 500000):\")\n", - "print(result_slow.samples.total_samples)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now lets run the search with fast settings, so we can compare the total number of iterations required." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "search = af.Nautilus(\n", - " path_prefix=Path(\"howtogalaxy\", \"chapter_2\"),\n", - " name=\"tutorial_searches_fast\",\n", - " unique_tag=dataset_name,\n", - " n_live=75,\n", - " n_batch=50, # GPU lens model fits are batched and run simultaneously, see VRAM section below.\n", - ")\n", - "\n", - "print(\n", - " \"The non-linear search has begun running - checkout the workspace/output\"\n", - " \" folder for live output of the results, images and model.\"\n", - " \" This Jupyter notebook cell with progress once search has completed - this could take some time!\"\n", - ")\n", - "\n", - "result_fast = search.fit(model=model, analysis=analysis)\n", - "\n", - "print(\"Search has finished run - you may now continue the notebook.\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Lets check that this search, despite its faster sampling settings, still gives us the global maxima solution." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_fit_imaging(fit=result_fast.max_log_likelihood_fit)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "And now lets confirm it uses significantly fewer iterations." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(\"Total Nautilus Iterations:\")\n", - "print(\"Slow settings: ~500000\")\n", - "print(result_slow.samples.total_samples)\n", - "print(\"Fast settings: \", result_fast.samples.total_samples)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Optimizers__\n", - "\n", - "Nested sampling algorithms like Nautilus provides the errors on all of the model parameters, by fully mapping out all \n", - "of the high likelihood regions of parameter space. This provides knowledge on the complete *range* of models that do \n", - "and do not provide high likelihood fits to the data, but takes many extra iterations to perform. If we require precise \n", - "error estimates (perhaps this is our final model fit before we publish the results in a paper), these extra\n", - "iterations are acceptable. \n", - "\n", - "However, we often don't care about the errors. For example, in the previous tutorial when chaining searches, the only\n", - "result we used from the fit performed in the first search was the maximum log likelihood model, omitting the errors\n", - "entirely! Its seems wasteful to use a nested sampling algorithm like Nautilus to map out the entirity of parameter\n", - "space when we don't use this information!\n", - "\n", - "There are a class of non-linear searches called `optimizers`, which seek to optimize just one thing, the log\n", - "likelihood. They want to find the model that maximizes the log likelihood, with no regard for the errors, thus not\n", - "wasting time mapping out in intricate detail every facet of parameter space.\n", - "\n", - "PyAutoFit supports the LBFGS optimizer (from scipy), which can be used as an alternative to nested sampling when\n", - "only the maximum likelihood model is needed. However, optimizers generally need a good starting point to work well,\n", - "and in our experience the parameter spaces fitted by models are often too complex for optimizers without careful\n", - "setup of initialization priors.\n", - "\n", - "__MCMC__\n", - "\n", - "For users familiar with Markov Chain Monte Carlo (MCMC) non-linear samplers, PyAutoFit supports the non-linear\n", - "searches `Emcee` and `Zeus`. Like LBFGS, these also need a good starting point, and are generally less effective at\n", - "modeling than Nautilus.\n", - "\n", - "I've included an example runs of Emcee and Zeus below, where the model is set up using `UniformPriors` to give\n", - "the starting point of the MCMC walkers. " - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "lens_bulge = af.Model(ag.lp.Sersic)\n", - "lens_bulge.centre.centre_0 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1)\n", - "lens_bulge.centre.centre_1 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1)\n", - "lens_bulge.ell_comps.ell_comps_0 = af.UniformPrior(lower_limit=-0.3, upper_limit=0.3)\n", - "lens_bulge.ell_comps.ell_comps_1 = af.UniformPrior(lower_limit=-0.3, upper_limit=0.3)\n", - "lens_bulge.intensity = af.UniformPrior(lower_limit=0.5, upper_limit=1.5)\n", - "lens_bulge.effective_radius = af.UniformPrior(lower_limit=0.2, upper_limit=1.6)\n", - "lens_bulge.sersic_index = af.UniformPrior(lower_limit=3.0, upper_limit=5.0)\n", - "\n", - "\n", - "mass = af.Model(ag.mp.Isothermal)\n", - "mass.centre.centre_0 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1)\n", - "mass.centre.centre_1 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1)\n", - "mass.ell_comps.ell_comps_0 = af.UniformPrior(lower_limit=-0.3, upper_limit=0.3)\n", - "mass.ell_comps.ell_comps_1 = af.UniformPrior(lower_limit=-0.3, upper_limit=0.3)\n", - "mass.einstein_radius = af.UniformPrior(lower_limit=1.0, upper_limit=2.0)\n", - "\n", - "shear = af.Model(ag.mp.ExternalShear)\n", - "shear.gamma_1 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1)\n", - "shear.gamma_2 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1)\n", - "\n", - "bulge = af.Model(ag.lp.Sersic)\n", - "bulge.centre.centre_0 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1)\n", - "bulge.centre.centre_1 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1)\n", - "bulge.ell_comps.ell_comps_0 = af.UniformPrior(lower_limit=-0.3, upper_limit=0.3)\n", - "bulge.ell_comps.ell_comps_1 = af.UniformPrior(lower_limit=-0.3, upper_limit=0.3)\n", - "bulge.intensity = af.UniformPrior(lower_limit=0.1, upper_limit=0.5)\n", - "bulge.effective_radius = af.UniformPrior(lower_limit=0.0, upper_limit=0.4)\n", - "bulge.sersic_index = af.UniformPrior(lower_limit=0.5, upper_limit=2.0)\n", - "\n", - "lens = af.Model(ag.Galaxy, redshift=0.5, mass=mass, shear=shear)\n", - "source = af.Model(ag.Galaxy, redshift=1.0, bulge=bulge)\n", - "\n", - "model = af.Collection(galaxies=af.Collection(lens=lens, source=source))\n", - "\n", - "search = af.Zeus(\n", - " path_prefix=Path(\"howtogalaxy\", \"chapter_2\"),\n", - " name=\"tutorial_searches_zeus\",\n", - " unique_tag=dataset_name,\n", - " nwalkers=50,\n", - " nsteps=1000,\n", - ")\n", - "\n", - "print(\n", - " \"Zeus has begun running - checkout the workspace/output\"\n", - " \" folder for live output of the results, images and model.\"\n", - " \" This Jupyter notebook cell with progress once search has completed - this could take some time!\"\n", - ")\n", - "\n", - "result_zeus = search.fit(model=model, analysis=analysis)\n", - "\n", - "print(\"Zeus has finished run - you may now continue the notebook.\")\n", - "\n", - "aplt.subplot_fit_imaging(fit=result_zeus.max_log_likelihood_fit)\n", - "\n", - "\n", - "search = af.Emcee(\n", - " path_prefix=Path(\"howtogalaxy\", \"chapter_2\"),\n", - " name=\"tutorial_searches_emcee\",\n", - " unique_tag=dataset_name,\n", - " nwalkers=50,\n", - " nsteps=1000,\n", - ")\n", - "\n", - "print(\n", - " \"The non-linear search has begun running - checkout the workspace/output\"\n", - " \" folder for live output of the results, images and model.\"\n", - " \" This Jupyter notebook cell with progress once search has completed - this could take some time!\"\n", - ")\n", - "\n", - "result_emcee = search.fit(model=model, analysis=analysis)\n", - "\n", - "print(\"The search has finished run - you may now continue the notebook.\")\n", - "\n", - "aplt.subplot_fit_imaging(fit=result_emcee.max_log_likelihood_fit)\n" - ], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/howtogalaxy/simulators/sersic.ipynb b/notebooks/howtogalaxy/simulators/sersic.ipynb deleted file mode 100644 index d69f5a03..00000000 --- a/notebooks/howtogalaxy/simulators/sersic.ipynb +++ /dev/null @@ -1,320 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Simulator: Sersic\n", - "=================\n", - "\n", - "This script simulates `Imaging` of a galaxy using light profiles where:\n", - "\n", - " - The galaxy's bulge is an `Sersic`.\n", - "\n", - "__Contents__\n", - "\n", - "**Dataset Paths:** Set the output path for the simulated dataset.\n", - "**Grid:** Create a 2D grid with adaptive over-sampling for simulation.\n", - "**Galaxies:** Define the galaxy Sersic light profile used for simulation.\n", - "**Output:** Save the simulated dataset to FITS files.\n", - "**Visualize:** Output subplot and image PNGs of the simulated dataset.\n", - "**Plane Output:** Save the Galaxies object as a JSON file.\n", - "\n", - "__Start Here Notebook__\n", - "\n", - "If any code in this script is unclear, refer to the `simulators/start_here.ipynb` notebook." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autoconf import setup_notebook; setup_notebook()\n", - "\n", - "from pathlib import Path\n", - "import autogalaxy as ag\n", - "import autogalaxy.plot as aplt" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset Paths__\n", - "\n", - "The `dataset_type` describes the type of data being simulated and `dataset_name` gives it a descriptive name. " - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_type = \"imaging\"\n", - "dataset_name = \"simple__sersic\"\n", - "\n", - "dataset_path = Path(\"dataset\", dataset_type, dataset_name)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Grid__\n", - "\n", - "Simulate the image using a (y,x) grid with the adaptive over sampling scheme." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "grid = ag.Grid2D.uniform(\n", - " shape_native=(100, 100),\n", - " pixel_scales=0.1,\n", - ")\n", - "\n", - "over_sample_size = ag.util.over_sample.over_sample_size_via_radial_bins_from(\n", - " grid=grid,\n", - " sub_size_list=[32, 8, 2],\n", - " radial_list=[0.3, 0.6],\n", - " centre_list=[(0.0, 0.0)],\n", - ")\n", - "\n", - "grid = grid.apply_over_sampling(over_sample_size=over_sample_size)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Simulate a simple Gaussian PSF for the image." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "psf = ag.Convolver.from_gaussian(\n", - " shape_native=(11, 11), sigma=0.1, pixel_scales=grid.pixel_scales\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Create the simulator for the imaging data, which defines the exposure time, background sky, noise levels and psf." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "simulator = ag.SimulatorImaging(\n", - " exposure_time=300.0,\n", - " psf=psf,\n", - " background_sky_level=0.1,\n", - " add_poisson_noise_to_data=True,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Galaxies__\n", - "\n", - "Setup the galaxy with a bulge (elliptical Sersic) for this simulation." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "galaxy = ag.Galaxy(\n", - " redshift=0.5,\n", - " bulge=ag.lp.Sersic(\n", - " centre=(0.0, 0.0),\n", - " ell_comps=ag.convert.ell_comps_from(axis_ratio=0.9, angle=45.0),\n", - " intensity=1.0,\n", - " effective_radius=0.8,\n", - " sersic_index=4.0,\n", - " ),\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Use these galaxies to generate the image for the simulated `Imaging` dataset." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "galaxies = ag.Galaxies(galaxies=[galaxy])\n", - "aplt.plot_array(array=galaxies.image_2d_from(grid=grid), title=\"Image\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Pass the simulator galaxies, which creates the image which is simulated as an imaging dataset." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset = simulator.via_galaxies_from(galaxies=galaxies, grid=grid)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Plot the simulated `Imaging` dataset before outputting it to fits." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_imaging_dataset(dataset=dataset)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Output__\n", - "\n", - "Output the simulated dataset to the dataset path as .fits files." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.fits_imaging(\n", - " dataset=dataset,\n", - " data_path=dataset_path / \"data.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " overwrite=True,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Visualize__\n", - "\n", - "Output a subplot of the simulated dataset, the image and the galaxies quantities to the dataset path as .png files." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_imaging_dataset(\n", - " dataset=dataset, output_path=dataset_path, output_format=\"png\"\n", - ")\n", - "aplt.plot_array(\n", - " array=dataset.data, title=\"Data\", output_path=dataset_path, output_format=\"png\"\n", - ")\n", - "\n", - "aplt.subplot_galaxies(\n", - " galaxies=galaxies, grid=grid, output_path=dataset_path, output_format=\"png\"\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Plane Output__\n", - "\n", - "Save the `Galaxies` in the dataset folder as a .json file, ensuring the true light profiles and galaxies\n", - "are safely stored and available to check how the dataset was simulated in the future. \n", - "\n", - "This can be loaded via the method `galaxies = ag.from_json()`." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "ag.output_to_json(\n", - " obj=galaxies,\n", - " file_path=Path(dataset_path, \"galaxies.json\"),\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The dataset can be viewed in the folder `autogalaxy_workspace/imaging/simple__sersic`." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/imaging/features/pixelization/fit.ipynb b/notebooks/imaging/features/pixelization/fit.ipynb index d5232008..95a6120b 100644 --- a/notebooks/imaging/features/pixelization/fit.ipynb +++ b/notebooks/imaging/features/pixelization/fit.ipynb @@ -148,7 +148,7 @@ " import sys\n", "\n", " subprocess.run(\n", - " [sys.executable, \"scripts/howtogalaxy/simulators/sersic.py\"],\n", + " [sys.executable, \"scripts/imaging/simulator_sersic.py\"],\n", " check=True,\n", " )\n", "\n", diff --git a/notebooks/imaging/features/pixelization/modeling.ipynb b/notebooks/imaging/features/pixelization/modeling.ipynb index b54ca0f8..83c21797 100644 --- a/notebooks/imaging/features/pixelization/modeling.ipynb +++ b/notebooks/imaging/features/pixelization/modeling.ipynb @@ -189,7 +189,7 @@ " import sys\n", "\n", " subprocess.run(\n", - " [sys.executable, \"scripts/howtogalaxy/simulators/sersic.py\"],\n", + " [sys.executable, \"scripts/imaging/simulator_sersic.py\"],\n", " check=True,\n", " )\n", "\n", diff --git a/notebooks/imaging/features/pixelization/source_science.ipynb b/notebooks/imaging/features/pixelization/source_science.ipynb index c23884cb..fcc17c54 100644 --- a/notebooks/imaging/features/pixelization/source_science.ipynb +++ b/notebooks/imaging/features/pixelization/source_science.ipynb @@ -80,7 +80,7 @@ " import sys\n", "\n", " subprocess.run(\n", - " [sys.executable, \"scripts/howtogalaxy/simulators/sersic.py\"],\n", + " [sys.executable, \"scripts/imaging/simulator_sersic.py\"],\n", " check=True,\n", " )\n", "\n", diff --git a/notebooks/imaging/features/shapelets/fit.ipynb b/notebooks/imaging/features/shapelets/fit.ipynb index cbe90356..021e733d 100644 --- a/notebooks/imaging/features/shapelets/fit.ipynb +++ b/notebooks/imaging/features/shapelets/fit.ipynb @@ -129,7 +129,7 @@ " import sys\n", "\n", " subprocess.run(\n", - " [sys.executable, \"scripts/howtogalaxy/simulators/sersic.py\"],\n", + " [sys.executable, \"scripts/imaging/simulator_sersic.py\"],\n", " check=True,\n", " )\n", "\n", diff --git a/notebooks/imaging/features/shapelets/modeling.ipynb b/notebooks/imaging/features/shapelets/modeling.ipynb index ee8e7aae..bd5bfdf9 100644 --- a/notebooks/imaging/features/shapelets/modeling.ipynb +++ b/notebooks/imaging/features/shapelets/modeling.ipynb @@ -131,7 +131,7 @@ " import sys\n", "\n", " subprocess.run(\n", - " [sys.executable, \"scripts/howtogalaxy/simulators/sersic.py\"],\n", + " [sys.executable, \"scripts/imaging/simulator_sersic.py\"],\n", " check=True,\n", " )\n", "\n", diff --git a/notebooks/imaging/modeling.ipynb b/notebooks/imaging/modeling.ipynb index 9ad1e0e2..02f78085 100644 --- a/notebooks/imaging/modeling.ipynb +++ b/notebooks/imaging/modeling.ipynb @@ -1,760 +1,760 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Modeling: Start Here\n", - "====================\n", - "\n", - "This script is the starting point for modeling of CCD imaging data (E.g. Hubble Space Telescope, Euclid) with\n", - "**PyAutoGalaxy** and it provides an overview of the modeling API.\n", - "\n", - "After reading this script, the `features`, `customize` and `searches` folders provide example for performing lens\n", - "modeling in different ways and customizing the analysis.\n", - "\n", - "__Model__\n", - "\n", - "This script fits an `Imaging` dataset of a galaxy with a model where:\n", - "\n", - " - The galaxy's light is a linear parametric `Sersic` bulge and `Exponential` disk.\n", - "\n", - "__Plotters__\n", - "\n", - "To produce images of the data `Plotter` objects are used, which are high-level wrappers of matplotlib\n", - "code which produce high quality visualization of galaxies.\n", - "\n", - "The `PLotter` API is described in the script `autogalaxy_workspace/*/guides/plot`.\n", - "\n", - "__Simulation__\n", - "\n", - "This script fits a simulated `Imaging` dataset of a galaxy, which is produced in the\n", - "script `autogalaxy_workspace/*/simulators/imaging/start_here.py`\n", - "\n", - "__Data Preparation__\n", - "\n", - "The `Imaging` dataset fitted in this example confirms to a number of standard that make it suitable to be fitted in\n", - "**PyAutoGalaxy**.\n", - "\n", - "If you are intending to fit your own data, you will need to ensure it conforms to these standards, which are\n", - "described in the script `autogalaxy_workspace/*/data_preparation/imaging/start_here.ipynb`.\n", - "\n", - "__Contents__\n", - "\n", - "**Dataset:** Loading the imaging dataset from FITS files.\n", - "**Dataset Auto-Simulation:** Automatically simulating the dataset if it does not already exist.\n", - "**Mask:** Defining a 2D mask for the region of interest around the galaxy.\n", - "**Over Sampling:** Applying adaptive over-sampling for accurate light profile evaluation.\n", - "**Model:** Composing the galaxy model with linear Sersic bulge and Exponential disk.\n", - "**Search:** Configuring the Nautilus nested sampling non-linear search.\n", - "**Analysis:** Creating the AnalysisImaging object for likelihood evaluation.\n", - "**VRAM Use:** Estimating GPU VRAM usage for JAX-accelerated fitting.\n", - "**Run Times:** Discussion of computational run times and how to estimate them.\n", - "**Model-Fit:** Running the model-fit and monitoring output.\n", - "**Output Folder:** Description of the results written to the output folder.\n", - "**Result:** Inspecting the result object, maximum likelihood model and posteriors.\n", - "**Features:** Links to advanced modeling features in the workspace.\n", - "**Data Preparation:** Links to data preparation resources.\n", - "**HowToGalaxy:** Links to the HowToGalaxy tutorial lectures." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autoconf import setup_notebook; setup_notebook()\n", - "\n", - "from pathlib import Path\n", - "import autofit as af\n", - "import autogalaxy as ag\n", - "import autogalaxy.plot as aplt" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset__\n", - "\n", - "Load the strong dataset `simple` via .fits files, which is a data format used by astronomers to store images.\n", - "\n", - "The `pixel_scales` define the arc-second to pixel conversion factor of the image, which for the dataset we are using \n", - "is 0.1\" / pixel." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_name = \"simple\"\n", - "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset Auto-Simulation__\n", - "\n", - "If the dataset does not already exist on your system, it will be created by running the corresponding\n", - "simulator script. This ensures that all example scripts can be run without manually simulating data first." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "if ag.util.dataset.should_simulate(str(dataset_path)):\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.run(\n", - " [sys.executable, \"scripts/imaging/simulator.py\"],\n", - " check=True,\n", - " )\n", - "\n", - "\n", - "dataset = ag.Imaging.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " pixel_scales=0.1,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Use an `Imaging` the plot the data, including: \n", - "\n", - " - `data`: The image of the strong lens.\n", - " - `noise_map`: The noise-map of the image, which quantifies the noise in every pixel as their RMS values.\n", - " - `psf`: The point spread function of the image, which describes the blurring of the image by the telescope optics.\n", - " - `signal_to_noise_map`: Quantifies the signal-to-noise in every pixel." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_imaging_dataset(dataset=dataset)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Mask__\n", - "\n", - "The model-fit requires a 2D mask defining the regions of the image we fit the model to the data. \n", - "\n", - "We create a 3.0 arcsecond circular mask and apply it to the `Imaging` object that the model fits." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "mask = ag.Mask2D.circular(\n", - " shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=3.0\n", - ")\n", - "\n", - "dataset = dataset.apply_mask(mask=mask)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "If we plot the masked data, the mask removes the exterior regions of the image where there is no emission from the \n", - "galaxy.\n", - "\n", - "The mask used to fit the data can be customized, as described in \n", - "the script `autogalaxy_workspace/*/modeling/imaging/customize/custom_mask.py`" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_imaging_dataset(dataset=dataset)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Over Sampling__\n", - "\n", - "Over sampling is a numerical technique where the images of light profiles and galaxies are evaluated \n", - "on a higher resolution grid than the image data to ensure the calculation is accurate. \n", - "\n", - "For a new user, the details of over-sampling are not important, therefore just be aware that below we make it so that \n", - "all calculations use an adaptive over sampling scheme which ensures high accuracy and precision.\n", - "\n", - "Once you are more experienced, you should read up on over-sampling in more detail via \n", - "the `autogalaxy_workspace/*/guides/over_sampling.ipynb` notebook." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "over_sample_size = ag.util.over_sample.over_sample_size_via_radial_bins_from(\n", - " grid=dataset.grid,\n", - " sub_size_list=[8, 4, 1],\n", - " radial_list=[0.3, 0.6],\n", - " centre_list=[(0.0, 0.0)],\n", - ")\n", - "\n", - "dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The imaging subplot updates the bottom two panels to reflect the update to over sampling, which now uses a higher\n", - "values in the centre.\n", - "\n", - "Whilst you may not yet understand the details of over-sampling, you can at least track it visually in the plots\n", - "and later learnt more about it in the `over_sampling.ipynb` guide." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_imaging_dataset(dataset=dataset)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Model__\n", - "\n", - "In this example we compose a model where:\n", - "\n", - " - The galaxy's bulge is a linear parametric `Sersic` bulge [6 parameters]. \n", - "\n", - " - The galaxy's disk is a linear parametric `Exponential` disk, whose centre is aligned with the bulge [3 parameters].\n", - " \n", - "The number of free parameters and therefore the dimensionality of non-linear parameter space is N=11.\n", - "\n", - "__Linear Light Profiles__\n", - "\n", - "The model below uses a `linear light profile` for the bulge and disk, via the API `lp_linear`. This is a specific type \n", - "of light profile that solves for the `intensity` of each profile that best fits the data via a linear inversion. \n", - "This means it is not a free parameter, reducing the dimensionality of non-linear parameter space. \n", - "\n", - "Linear light profiles significantly improve the speed, accuracy and reliability of modeling and they are used\n", - "by default in every modeling example. A full description of linear light profiles is provided in the\n", - "`autogalaxy_workspace/*/modeling/imaging/features/linear_light_profiles.py` example.\n", - "\n", - "A standard light profile can be used if you change the `lp_linear` to `lp`, but it is not recommended.\n", - "\n", - "__Model Composition__\n", - "\n", - "The API below for composing a model uses the `Model` and `Collection` objects, which are imported from \n", - "**PyAutoGalaxy**'s parent project **PyAutoFit** \n", - "\n", - "The API is fairly self explanatory and is straight forward to extend, for example adding more light profiles\n", - "to the galaxy.\n", - "\n", - "__Model Cookbook__\n", - "\n", - "A full description of model composition is provided by the model cookbook: \n", - "\n", - "https://pyautogalaxy.readthedocs.io/en/latest/general/model_cookbook.html\n", - "\n", - "__Coordinates__\n", - "\n", - "The model fitting default settings assume that the galaxy centre is near the coordinates (0.0\", 0.0\"). \n", - "\n", - "If for your dataset the galaxy is not centred at (0.0\", 0.0\"), we recommend that you either: \n", - "\n", - " - Reduce your data so that the centre is (`autogalaxy_workspace/*/preprocess`). \n", - " - Manually override the model priors (`autogalaxy_workspace/*/modeling/imaging/customize/priors.py`)." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "bulge = af.Model(ag.lp_linear.Sersic)\n", - "disk = af.Model(ag.lp_linear.Exponential)\n", - "bulge.centre = disk.centre\n", - "\n", - "galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=bulge, disk=disk)\n", - "\n", - "model = af.Collection(galaxies=af.Collection(galaxy=galaxy))" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `info` attribute shows the model in a readable format.\n", - "\n", - "[The `info` below may not display optimally on your computer screen, for example the whitespace between parameter\n", - "names on the left and parameter priors on the right may lead them to appear across multiple lines. This is a\n", - "common issue in Jupyter notebooks.\n", - "\n", - "The`info_whitespace_length` parameter in the file `config/generag.yaml` in the [output] section can be changed to \n", - "increase or decrease the amount of whitespace (The Jupyter notebook kernel will need to be reset for this change to \n", - "appear in a notebook).]" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(model.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Search__\n", - "\n", - "The model is fitted to the data using a non-linear search. \n", - "\n", - "All examples in the autogalaxy workspace use the nested sampling algorithm \n", - "Nautilus (https://nautilus-sampler.readthedocs.io/en/latest/), which extensive testing has revealed gives the most \n", - "accurate and efficient modeling results.\n", - "\n", - "Nautilus has one main setting that trades-off accuracy and computational run-time, the number of `live_points`. \n", - "A higher number of live points gives a more accurate result, but increases the run-time. A lower value may give \n", - "less reliable modeling (e.g. the fit may infer a local maxima), but is faster. \n", - "\n", - "The suitable value depends on the model complexity whereby models with more parameters require more live points. \n", - "The default value of 200 is sufficient for the vast majority of common galaxy models. Lower values often given reliable\n", - "results though, and speed up the run-times. In this example, given the model is quite simple (N=21 parameters), we \n", - "reduce the number of live points to 100 to speed up the run-time.\n", - "\n", - "__Unique Identifier__\n", - "\n", - "In the path above, the `unique_identifier` appears as a collection of characters, where this identifier is generated \n", - "based on the model, search and dataset that are used in the fit.\n", - " \n", - "An identical combination of model and search generates the same identifier, meaning that rerunning the script will use \n", - "the existing results to resume the model-fit. In contrast, if you change the model or search, a new unique identifier \n", - "will be generated, ensuring that the model-fit results are output into a separate folder.\n", - "\n", - "We additionally want the unique identifier to be specific to the dataset fitted, so that if we fit different datasets\n", - "with the same model and search results are output to a different folder. We achieve this below by passing \n", - "the `dataset_name` to the search's `unique_tag`.\n", - "\n", - "__Iterations Per Update__\n", - "\n", - "Every N iterations, the non-linear search outputs the current results to the folder `autogalaxy_workspace/output`,\n", - "which includes producing visualization. \n", - "\n", - "Depending on how long it takes for the model to be fitted to the data (see discussion about run times below), \n", - "this can take up a large fraction of the run-time of the non-linear search.\n", - "\n", - "For this fit, the fit is very fast, thus we set a high value of `iterations_per_quick_update=10000` to ensure these updates\n", - "so not slow down the overall speed of the model-fit. \n", - "# \n", - "**If the iteration per update is too low, the model-fit may be significantly slowed down by the time it takes to\n", - "output results and visualization frequently to hard-disk. If your fit is consistent displaying a log saying that it\n", - "is outputting results, try increasing this value to ensure the model-fit runs efficiently.**" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "search = af.Nautilus(\n", - " path_prefix=Path(\"imaging\") / \"features\",\n", - " name=\"start_here\",\n", - " unique_tag=dataset_name,\n", - " n_live=200,\n", - " n_batch=50, # GPU lens model fits are batched and run simultaneously, see VRAM section below.\n", - " iterations_per_quick_update=10000,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Analysis__\n", - "\n", - "We next create an `AnalysisImaging` object, which can be given many inputs customizing how the model is fitted to the \n", - "data (in this example they are omitted for simplicity).\n", - "\n", - "Internally, this object defines the `log_likelihood_function` used by the non-linear search to fit the model to \n", - "the `Imaging` dataset. \n", - "\n", - "It is not vital that you as a user understand the details of how the `log_likelihood_function` fits a model to \n", - "data, but interested readers can find a step-by-step guide of the likelihood \n", - "function at ``autogalaxy_workspace/*/imaging/log_likelihood_function`\n", - "\n", - "__JAX__\n", - "\n", - "Analysis uses JAX under the hood for fast GPU/CPU acceleration. If JAX is installed with GPU\n", - "support, your fits will run much faster (around 10 minutes instead of an hour). If only a CPU is available,\n", - "JAX will still provide a speed up via multithreading, with fits taking around 20-30 minutes.\n", - "\n", - "If you don\u2019t have a GPU locally, consider Google Colab which provides free GPUs, so your modeling runs are much faster." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "analysis = ag.AnalysisImaging(dataset=dataset, use_jax=True)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__VRAM Use__\n", - "\n", - "When running with JAX on a GPU, the analysis must fit within the GPU\u2019s\n", - "available VRAM. If insufficient VRAM is available, the analysis will fail with an\n", - "out-of-memory error, typically during JIT compilation or the first likelihood call.\n", - "\n", - "Two factors dictate the VRAM usage of an analysis:\n", - "\n", - "- The number of arrays and other data structures JAX must store in VRAM to fit the model\n", - " to the data in the likelihood function. This is dictated by the model complexity and dataset size.\n", - "\n", - "- The `batch_size` sets how many likelihood evaluations are performed simultaneously.\n", - " Increasing the batch size increases VRAM usage but can reduce overall run time,\n", - " while decreasing it lowers VRAM usage at the cost of slower execution.\n", - "\n", - "Before running an analysis, users should check that the estimated VRAM usage for the\n", - "chosen batch size is comfortably below their GPU\u2019s total VRAM.\n", - "\n", - "The method below prints the VRAM usage estimate for the analysis and model with the specified batch size,\n", - "it takes about 20-30 seconds to run so you may want to comment it out once you are familiar with your GPU's VRAM limits.\n", - "\n", - "For a MGE model with the low resolution dataset fitted in this example VRAM use is relatively low (~0.027GB) For other \n", - "models (e.g. pixelized sources) and higher resolution datasets it can be much higher (> 1GB going beyond 10GB)." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "analysis.print_vram_use(model=model, batch_size=search.batch_size)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Run Times__\n", - "\n", - "Modeling can be a computationally expensive process. When fitting complex models to high resolution datasets \n", - "run times can be of order hours, days, weeks or even months.\n", - "\n", - "Run times are dictated by two factors:\n", - "\n", - " - The log likelihood evaluation time: the time it takes for a single `instance` of the model to be fitted to \n", - " the dataset such that a log likelihood is returned.\n", - "\n", - " - The number of iterations (e.g. log likelihood evaluations) performed by the non-linear search: more complex\n", - " models require more iterations to converge to a solution.\n", - "\n", - "For this analysis, the log likelihood evaluation time is ~0.03 seconds, which is extremely fast for modeling. More advanced\n", - "modeling features (e.g. shapelets, multi Gaussian expansions, pixelizations) have slower log likelihood evaluation\n", - "times (1-3 seconds), and you should be wary of this when using these features.PointDataset\n", - "\n", - "To estimate the expected overall run time of the model-fit we multiply the log likelihood evaluation time by an \n", - "estimate of the number of iterations the non-linear search will perform. \n", - "\n", - "Estimating this quantity is more tricky, as it varies depending on the model complexity (e.g. number of parameters)\n", - "and the properties of the dataset and model being fitted.\n", - "\n", - "For this example, we conservatively estimate that the non-linear search will perform ~10000 iterations per free \n", - "parameter in the model. This is an upper limit, with models typically converging in far fewer iterations.\n", - "\n", - "If you perform the fit over multiple CPUs, you can divide the run time by the number of cores to get an estimate of\n", - "the time it will take to fit the model. Parallelization with Nautilus scales well, it speeds up the model-fit by the \n", - "`number_of_cores` for N < 8 CPUs and roughly `0.5*number_of_cores` for N > 8 CPUs. This scaling continues \n", - "for N> 50 CPUs, meaning that with super computing facilities you can always achieve fast run times!\n", - "\n", - "__Model-Fit__\n", - "\n", - "We can now begin the model-fit by passing the model and analysis object to the search, which performs a non-linear\n", - "search to find which models fit the data with the highest likelihood.\n", - "\n", - "Checkout the output folder for live outputs of the results of the fit, including on-the-fly visualization of the best \n", - "fit model!\n", - "\n", - "**Run Time Error:** On certain operating systems (e.g. Windows, Linux) and Python versions, the code below may produce \n", - "an error. If this occurs, see the `autolens_workspace/guides/modeling/bug_fix` example for a fix." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "result = search.fit(model=model, analysis=analysis)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Output Folder__\n", - "\n", - "Now this is running you should checkout the `autogalaxy_workspace/output` folder. This is where the results of the \n", - "search are written to hard-disk (in the `start_here` folder), where all outputs are human readable (e.g. as .json,\n", - ".csv or text files).\n", - "\n", - "As the fit progresses, results are written to the `output` folder on the fly using the highest likelihood model found\n", - "by the non-linear search so far. This means you can inspect the results of the model-fit as it runs, without having to\n", - "wait for the non-linear search to terminate.\n", - " \n", - "The `output` folder includes:\n", - "\n", - " - `model.info`: Summarizes the model, its parameters and their priors discussed in the next tutorial.\n", - " \n", - " - `model.results`: Summarizes the highest likelihood model inferred so far including errors.\n", - " \n", - " - `images`: Visualization of the highest likelihood model-fit to the dataset, (e.g. a fit subplot showing the \n", - " galaxies, model data and residuals).\n", - " \n", - " - `files`: A folder containing .fits files of the dataset, the model as a human-readable .json file, \n", - " a `.csv` table of every non-linear search sample and other files containing information about the model-fit.\n", - " \n", - " - search.summary: A file providing summary statistics on the performance of the non-linear search.\n", - " \n", - " - `search_internal`: Internal files of the non-linear search (in this case Nautilus) used for resuming the fit and\n", - " visualizing the search.\n", - "\n", - "__Result__\n", - "\n", - "The search returns a result object, which whose `info` attribute shows the result in a readable format.\n", - "\n", - "[Above, we discussed that the `info_whitespace_length` parameter in the config files could b changed to make \n", - "the `model.info` attribute display optimally on your computer. This attribute also controls the whitespace of the\n", - "`result.info` attribute.]" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(result.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `Result` object also contains:\n", - "\n", - " - The model corresponding to the maximum log likelihood solution in parameter space.\n", - " - The corresponding maximum log likelihood `Galaxies` and `FitImaging` objects." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(result.max_log_likelihood_instance)\n", - "\n", - "aplt.subplot_galaxies(galaxies=result.max_log_likelihood_galaxies, grid=result.grids.lp)\n", - "\n", - "aplt.subplot_fit_imaging(fit=result.max_log_likelihood_fit)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The result contains the full posterior information of our non-linear search, including all parameter samples, \n", - "log likelihood values and tools to compute the errors on the model. \n", - "\n", - "There are built in visualization tools for plotting this.\n", - "\n", - "The plot is labeled with short hand parameter names (e.g. `sersic_index` is mapped to the short hand \n", - "parameter `n`). These mappings ate specified in the `config/notation.yaml` file and can be customized by users.\n", - "\n", - "The superscripts of labels correspond to the name each component was given in the model (e.g. for the `Isothermal`\n", - "mass its name `mass` defined when making the `Model` above is used)." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.corner_cornerpy(samples=result.samples)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Loading From Output Folder__\n", - "\n", - "Everything the `Result` object contains has also been written to hard-disk, inside the fit's output folder. Each\n", - "file loads back into a full Python object with a single line \u2014 much faster and simpler than re-running the fit.\n", - "\n", - "For example, the maximum log likelihood `Galaxies` is saved as a `.json` file and the per-galaxy model images as\n", - "a `.fits` file:" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "from autoconf.dictable import from_json\n", - "\n", - "result_path = search.paths.output_path # Points at the fit's unique output folder.\n", - "\n", - "if (result_path / \"files\" / \"galaxies.json\").exists():\n", - " galaxies = from_json(file_path=result_path / \"files\" / \"galaxies.json\")\n", - "\n", - " galaxy_images = ag.Array2D.from_fits(\n", - " file_path=result_path / \"image\" / \"galaxy_images.fits\", hdu=0, pixel_scales=0.1\n", - " )" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The output folder also contains `model.json`, `samples.csv`, `dataset.fits`, `fit.fits` and more. A full walkthrough\n", - "of loading results from the output folder is given in:\n", - "\n", - " `autogalaxy_workspace/*/guides/results/start_here.py`\n", - "\n", - "For multi-fit, memory-efficient workflows using the aggregator instead, see:\n", - "\n", - " `autogalaxy_workspace/*/guides/results/aggregator/start_here.py`" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "# %%\n", - "'''\n", - "This script gives a concise overview of the PyAutoGalaxy modeling API, fitting one the simplest models possible.\n", - "So, what next? \n", - "\n", - "__Features__\n", - "\n", - "The examples in the `autogalaxy_workspace/*/modeling/imaging/features` package illustrate other modeling features. \n", - "\n", - "We recommend you checkout the following features, because the make modeling in general more reliable and \n", - "efficient (you will therefore benefit from using these features irrespective of the quality of your data and \n", - "scientific topic of study).\n", - "\n", - "We recommend you now checkout the following feature:\n", - "\n", - "- ``linear_light_profiles``: The model light profiles use linear algebra to solve for their intensity, reducing model complexity.\n", - "\n", - "All other features may be useful to specific users with specific datasets and scientific goals, but are not useful\n", - "for general modeling.\n", - "\n", - "The folders `autogalaxy_workspace/*/modeling/imaging/searches` and `autogalaxy_workspace/*/modeling/imaging/customize`\n", - "provide guides on how to customize many other aspects of the model-fit. Check them out to see if anything\n", - "sounds useful, but for most users you can get by without using these forms of customization!\n", - " \n", - "__Data Preparation__\n", - "\n", - "If you are looking to fit your own CCD imaging data of a galaxy, checkout \n", - "the `autogalaxy_workspace/*/data_preparation/imaging/start_here.ipynb` script for an overview of how data should be \n", - "prepared before being modeled.\n", - "\n", - "__HowToGalaxy__\n", - "\n", - "This `start_here.py` script, and the features examples above, do not explain many details of how modeling is \n", - "performed, for example:\n", - "\n", - " - How does PyAutoGalaxy perform calculations which determine how a gaalxy emits light and therefore fit a model?\n", - " - How is the model fitted to data? What quantifies the goodness of fit (e.g. how is a log likelihood computed?).\n", - " - How does Nautilus find the highest likelihood models? What exactly is a \"non-linear search\"?\n", - "\n", - "You do not need to be able to answer these questions in order to fit models with PyAutoGalaxy and do science.\n", - "However, having a deeper understanding of how it all works is both interesting and will benefit you as a scientist\n", - "\n", - "This deeper insight is offered by the **HowToGalaxy** Jupyter notebook lectures, found \n", - "at `autogalaxy_workspace/*/howtogalaxy`. \n", - "\n", - "I recommend that you check them out if you are interested in more details!\n", - "'''" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Modeling: Start Here\n", + "====================\n", + "\n", + "This script is the starting point for modeling of CCD imaging data (E.g. Hubble Space Telescope, Euclid) with\n", + "**PyAutoGalaxy** and it provides an overview of the modeling API.\n", + "\n", + "After reading this script, the `features`, `customize` and `searches` folders provide example for performing lens\n", + "modeling in different ways and customizing the analysis.\n", + "\n", + "__Model__\n", + "\n", + "This script fits an `Imaging` dataset of a galaxy with a model where:\n", + "\n", + " - The galaxy's light is a linear parametric `Sersic` bulge and `Exponential` disk.\n", + "\n", + "__Plotters__\n", + "\n", + "To produce images of the data `Plotter` objects are used, which are high-level wrappers of matplotlib\n", + "code which produce high quality visualization of galaxies.\n", + "\n", + "The `PLotter` API is described in the script `autogalaxy_workspace/*/guides/plot`.\n", + "\n", + "__Simulation__\n", + "\n", + "This script fits a simulated `Imaging` dataset of a galaxy, which is produced in the\n", + "script `autogalaxy_workspace/*/simulators/imaging/start_here.py`\n", + "\n", + "__Data Preparation__\n", + "\n", + "The `Imaging` dataset fitted in this example confirms to a number of standard that make it suitable to be fitted in\n", + "**PyAutoGalaxy**.\n", + "\n", + "If you are intending to fit your own data, you will need to ensure it conforms to these standards, which are\n", + "described in the script `autogalaxy_workspace/*/data_preparation/imaging/start_here.ipynb`.\n", + "\n", + "__Contents__\n", + "\n", + "**Dataset:** Loading the imaging dataset from FITS files.\n", + "**Dataset Auto-Simulation:** Automatically simulating the dataset if it does not already exist.\n", + "**Mask:** Defining a 2D mask for the region of interest around the galaxy.\n", + "**Over Sampling:** Applying adaptive over-sampling for accurate light profile evaluation.\n", + "**Model:** Composing the galaxy model with linear Sersic bulge and Exponential disk.\n", + "**Search:** Configuring the Nautilus nested sampling non-linear search.\n", + "**Analysis:** Creating the AnalysisImaging object for likelihood evaluation.\n", + "**VRAM Use:** Estimating GPU VRAM usage for JAX-accelerated fitting.\n", + "**Run Times:** Discussion of computational run times and how to estimate them.\n", + "**Model-Fit:** Running the model-fit and monitoring output.\n", + "**Output Folder:** Description of the results written to the output folder.\n", + "**Result:** Inspecting the result object, maximum likelihood model and posteriors.\n", + "**Features:** Links to advanced modeling features in the workspace.\n", + "**Data Preparation:** Links to data preparation resources.\n", + "**HowToGalaxy:** Links to the HowToGalaxy tutorial lectures." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "\n", + "from autoconf import setup_notebook; setup_notebook()\n", + "\n", + "from pathlib import Path\n", + "import autofit as af\n", + "import autogalaxy as ag\n", + "import autogalaxy.plot as aplt" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Dataset__\n", + "\n", + "Load the strong dataset `simple` via .fits files, which is a data format used by astronomers to store images.\n", + "\n", + "The `pixel_scales` define the arc-second to pixel conversion factor of the image, which for the dataset we are using \n", + "is 0.1\" / pixel." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset_name = \"simple\"\n", + "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Dataset Auto-Simulation__\n", + "\n", + "If the dataset does not already exist on your system, it will be created by running the corresponding\n", + "simulator script. This ensures that all example scripts can be run without manually simulating data first." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "if ag.util.dataset.should_simulate(str(dataset_path)):\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.run(\n", + " [sys.executable, \"scripts/imaging/simulator.py\"],\n", + " check=True,\n", + " )\n", + "\n", + "\n", + "dataset = ag.Imaging.from_fits(\n", + " data_path=dataset_path / \"data.fits\",\n", + " psf_path=dataset_path / \"psf.fits\",\n", + " noise_map_path=dataset_path / \"noise_map.fits\",\n", + " pixel_scales=0.1,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Use an `Imaging` the plot the data, including: \n", + "\n", + " - `data`: The image of the strong lens.\n", + " - `noise_map`: The noise-map of the image, which quantifies the noise in every pixel as their RMS values.\n", + " - `psf`: The point spread function of the image, which describes the blurring of the image by the telescope optics.\n", + " - `signal_to_noise_map`: Quantifies the signal-to-noise in every pixel." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.subplot_imaging_dataset(dataset=dataset)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Mask__\n", + "\n", + "The model-fit requires a 2D mask defining the regions of the image we fit the model to the data. \n", + "\n", + "We create a 3.0 arcsecond circular mask and apply it to the `Imaging` object that the model fits." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "mask = ag.Mask2D.circular(\n", + " shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=3.0\n", + ")\n", + "\n", + "dataset = dataset.apply_mask(mask=mask)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "If we plot the masked data, the mask removes the exterior regions of the image where there is no emission from the \n", + "galaxy.\n", + "\n", + "The mask used to fit the data can be customized, as described in \n", + "the script `autogalaxy_workspace/*/modeling/imaging/customize/custom_mask.py`" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.subplot_imaging_dataset(dataset=dataset)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Over Sampling__\n", + "\n", + "Over sampling is a numerical technique where the images of light profiles and galaxies are evaluated \n", + "on a higher resolution grid than the image data to ensure the calculation is accurate. \n", + "\n", + "For a new user, the details of over-sampling are not important, therefore just be aware that below we make it so that \n", + "all calculations use an adaptive over sampling scheme which ensures high accuracy and precision.\n", + "\n", + "Once you are more experienced, you should read up on over-sampling in more detail via \n", + "the `autogalaxy_workspace/*/guides/over_sampling.ipynb` notebook." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "over_sample_size = ag.util.over_sample.over_sample_size_via_radial_bins_from(\n", + " grid=dataset.grid,\n", + " sub_size_list=[8, 4, 1],\n", + " radial_list=[0.3, 0.6],\n", + " centre_list=[(0.0, 0.0)],\n", + ")\n", + "\n", + "dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The imaging subplot updates the bottom two panels to reflect the update to over sampling, which now uses a higher\n", + "values in the centre.\n", + "\n", + "Whilst you may not yet understand the details of over-sampling, you can at least track it visually in the plots\n", + "and later learnt more about it in the `over_sampling.ipynb` guide." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.subplot_imaging_dataset(dataset=dataset)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Model__\n", + "\n", + "In this example we compose a model where:\n", + "\n", + " - The galaxy's bulge is a linear parametric `Sersic` bulge [6 parameters]. \n", + "\n", + " - The galaxy's disk is a linear parametric `Exponential` disk, whose centre is aligned with the bulge [3 parameters].\n", + " \n", + "The number of free parameters and therefore the dimensionality of non-linear parameter space is N=11.\n", + "\n", + "__Linear Light Profiles__\n", + "\n", + "The model below uses a `linear light profile` for the bulge and disk, via the API `lp_linear`. This is a specific type \n", + "of light profile that solves for the `intensity` of each profile that best fits the data via a linear inversion. \n", + "This means it is not a free parameter, reducing the dimensionality of non-linear parameter space. \n", + "\n", + "Linear light profiles significantly improve the speed, accuracy and reliability of modeling and they are used\n", + "by default in every modeling example. A full description of linear light profiles is provided in the\n", + "`autogalaxy_workspace/*/modeling/imaging/features/linear_light_profiles.py` example.\n", + "\n", + "A standard light profile can be used if you change the `lp_linear` to `lp`, but it is not recommended.\n", + "\n", + "__Model Composition__\n", + "\n", + "The API below for composing a model uses the `Model` and `Collection` objects, which are imported from \n", + "**PyAutoGalaxy**'s parent project **PyAutoFit** \n", + "\n", + "The API is fairly self explanatory and is straight forward to extend, for example adding more light profiles\n", + "to the galaxy.\n", + "\n", + "__Model Cookbook__\n", + "\n", + "A full description of model composition is provided by the model cookbook: \n", + "\n", + "https://pyautogalaxy.readthedocs.io/en/latest/general/model_cookbook.html\n", + "\n", + "__Coordinates__\n", + "\n", + "The model fitting default settings assume that the galaxy centre is near the coordinates (0.0\", 0.0\"). \n", + "\n", + "If for your dataset the galaxy is not centred at (0.0\", 0.0\"), we recommend that you either: \n", + "\n", + " - Reduce your data so that the centre is (`autogalaxy_workspace/*/preprocess`). \n", + " - Manually override the model priors (`autogalaxy_workspace/*/modeling/imaging/customize/priors.py`)." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "bulge = af.Model(ag.lp_linear.Sersic)\n", + "disk = af.Model(ag.lp_linear.Exponential)\n", + "bulge.centre = disk.centre\n", + "\n", + "galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=bulge, disk=disk)\n", + "\n", + "model = af.Collection(galaxies=af.Collection(galaxy=galaxy))" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The `info` attribute shows the model in a readable format.\n", + "\n", + "[The `info` below may not display optimally on your computer screen, for example the whitespace between parameter\n", + "names on the left and parameter priors on the right may lead them to appear across multiple lines. This is a\n", + "common issue in Jupyter notebooks.\n", + "\n", + "The`info_whitespace_length` parameter in the file `config/generag.yaml` in the [output] section can be changed to \n", + "increase or decrease the amount of whitespace (The Jupyter notebook kernel will need to be reset for this change to \n", + "appear in a notebook).]" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(model.info)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Search__\n", + "\n", + "The model is fitted to the data using a non-linear search. \n", + "\n", + "All examples in the autogalaxy workspace use the nested sampling algorithm \n", + "Nautilus (https://nautilus-sampler.readthedocs.io/en/latest/), which extensive testing has revealed gives the most \n", + "accurate and efficient modeling results.\n", + "\n", + "Nautilus has one main setting that trades-off accuracy and computational run-time, the number of `live_points`. \n", + "A higher number of live points gives a more accurate result, but increases the run-time. A lower value may give \n", + "less reliable modeling (e.g. the fit may infer a local maxima), but is faster. \n", + "\n", + "The suitable value depends on the model complexity whereby models with more parameters require more live points. \n", + "The default value of 200 is sufficient for the vast majority of common galaxy models. Lower values often given reliable\n", + "results though, and speed up the run-times. In this example, given the model is quite simple (N=21 parameters), we \n", + "reduce the number of live points to 100 to speed up the run-time.\n", + "\n", + "__Unique Identifier__\n", + "\n", + "In the path above, the `unique_identifier` appears as a collection of characters, where this identifier is generated \n", + "based on the model, search and dataset that are used in the fit.\n", + " \n", + "An identical combination of model and search generates the same identifier, meaning that rerunning the script will use \n", + "the existing results to resume the model-fit. In contrast, if you change the model or search, a new unique identifier \n", + "will be generated, ensuring that the model-fit results are output into a separate folder.\n", + "\n", + "We additionally want the unique identifier to be specific to the dataset fitted, so that if we fit different datasets\n", + "with the same model and search results are output to a different folder. We achieve this below by passing \n", + "the `dataset_name` to the search's `unique_tag`.\n", + "\n", + "__Iterations Per Update__\n", + "\n", + "Every N iterations, the non-linear search outputs the current results to the folder `autogalaxy_workspace/output`,\n", + "which includes producing visualization. \n", + "\n", + "Depending on how long it takes for the model to be fitted to the data (see discussion about run times below), \n", + "this can take up a large fraction of the run-time of the non-linear search.\n", + "\n", + "For this fit, the fit is very fast, thus we set a high value of `iterations_per_quick_update=10000` to ensure these updates\n", + "so not slow down the overall speed of the model-fit. \n", + "# \n", + "**If the iteration per update is too low, the model-fit may be significantly slowed down by the time it takes to\n", + "output results and visualization frequently to hard-disk. If your fit is consistent displaying a log saying that it\n", + "is outputting results, try increasing this value to ensure the model-fit runs efficiently.**" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "search = af.Nautilus(\n", + " path_prefix=Path(\"imaging\") / \"features\",\n", + " name=\"start_here\",\n", + " unique_tag=dataset_name,\n", + " n_live=200,\n", + " n_batch=50, # GPU lens model fits are batched and run simultaneously, see VRAM section below.\n", + " iterations_per_quick_update=10000,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Analysis__\n", + "\n", + "We next create an `AnalysisImaging` object, which can be given many inputs customizing how the model is fitted to the \n", + "data (in this example they are omitted for simplicity).\n", + "\n", + "Internally, this object defines the `log_likelihood_function` used by the non-linear search to fit the model to \n", + "the `Imaging` dataset. \n", + "\n", + "It is not vital that you as a user understand the details of how the `log_likelihood_function` fits a model to \n", + "data, but interested readers can find a step-by-step guide of the likelihood \n", + "function at ``autogalaxy_workspace/*/imaging/log_likelihood_function`\n", + "\n", + "__JAX__\n", + "\n", + "Analysis uses JAX under the hood for fast GPU/CPU acceleration. If JAX is installed with GPU\n", + "support, your fits will run much faster (around 10 minutes instead of an hour). If only a CPU is available,\n", + "JAX will still provide a speed up via multithreading, with fits taking around 20-30 minutes.\n", + "\n", + "If you don\u2019t have a GPU locally, consider Google Colab which provides free GPUs, so your modeling runs are much faster." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "analysis = ag.AnalysisImaging(dataset=dataset, use_jax=True)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__VRAM Use__\n", + "\n", + "When running with JAX on a GPU, the analysis must fit within the GPU\u2019s\n", + "available VRAM. If insufficient VRAM is available, the analysis will fail with an\n", + "out-of-memory error, typically during JIT compilation or the first likelihood call.\n", + "\n", + "Two factors dictate the VRAM usage of an analysis:\n", + "\n", + "- The number of arrays and other data structures JAX must store in VRAM to fit the model\n", + " to the data in the likelihood function. This is dictated by the model complexity and dataset size.\n", + "\n", + "- The `batch_size` sets how many likelihood evaluations are performed simultaneously.\n", + " Increasing the batch size increases VRAM usage but can reduce overall run time,\n", + " while decreasing it lowers VRAM usage at the cost of slower execution.\n", + "\n", + "Before running an analysis, users should check that the estimated VRAM usage for the\n", + "chosen batch size is comfortably below their GPU\u2019s total VRAM.\n", + "\n", + "The method below prints the VRAM usage estimate for the analysis and model with the specified batch size,\n", + "it takes about 20-30 seconds to run so you may want to comment it out once you are familiar with your GPU's VRAM limits.\n", + "\n", + "For a MGE model with the low resolution dataset fitted in this example VRAM use is relatively low (~0.027GB) For other \n", + "models (e.g. pixelized sources) and higher resolution datasets it can be much higher (> 1GB going beyond 10GB)." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "analysis.print_vram_use(model=model, batch_size=search.batch_size)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Run Times__\n", + "\n", + "Modeling can be a computationally expensive process. When fitting complex models to high resolution datasets \n", + "run times can be of order hours, days, weeks or even months.\n", + "\n", + "Run times are dictated by two factors:\n", + "\n", + " - The log likelihood evaluation time: the time it takes for a single `instance` of the model to be fitted to \n", + " the dataset such that a log likelihood is returned.\n", + "\n", + " - The number of iterations (e.g. log likelihood evaluations) performed by the non-linear search: more complex\n", + " models require more iterations to converge to a solution.\n", + "\n", + "For this analysis, the log likelihood evaluation time is ~0.03 seconds, which is extremely fast for modeling. More advanced\n", + "modeling features (e.g. shapelets, multi Gaussian expansions, pixelizations) have slower log likelihood evaluation\n", + "times (1-3 seconds), and you should be wary of this when using these features.PointDataset\n", + "\n", + "To estimate the expected overall run time of the model-fit we multiply the log likelihood evaluation time by an \n", + "estimate of the number of iterations the non-linear search will perform. \n", + "\n", + "Estimating this quantity is more tricky, as it varies depending on the model complexity (e.g. number of parameters)\n", + "and the properties of the dataset and model being fitted.\n", + "\n", + "For this example, we conservatively estimate that the non-linear search will perform ~10000 iterations per free \n", + "parameter in the model. This is an upper limit, with models typically converging in far fewer iterations.\n", + "\n", + "If you perform the fit over multiple CPUs, you can divide the run time by the number of cores to get an estimate of\n", + "the time it will take to fit the model. Parallelization with Nautilus scales well, it speeds up the model-fit by the \n", + "`number_of_cores` for N < 8 CPUs and roughly `0.5*number_of_cores` for N > 8 CPUs. This scaling continues \n", + "for N> 50 CPUs, meaning that with super computing facilities you can always achieve fast run times!\n", + "\n", + "__Model-Fit__\n", + "\n", + "We can now begin the model-fit by passing the model and analysis object to the search, which performs a non-linear\n", + "search to find which models fit the data with the highest likelihood.\n", + "\n", + "Checkout the output folder for live outputs of the results of the fit, including on-the-fly visualization of the best \n", + "fit model!\n", + "\n", + "**Run Time Error:** On certain operating systems (e.g. Windows, Linux) and Python versions, the code below may produce \n", + "an error. If this occurs, see the `autolens_workspace/guides/modeling/bug_fix` example for a fix." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "result = search.fit(model=model, analysis=analysis)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Output Folder__\n", + "\n", + "Now this is running you should checkout the `autogalaxy_workspace/output` folder. This is where the results of the \n", + "search are written to hard-disk (in the `start_here` folder), where all outputs are human readable (e.g. as .json,\n", + ".csv or text files).\n", + "\n", + "As the fit progresses, results are written to the `output` folder on the fly using the highest likelihood model found\n", + "by the non-linear search so far. This means you can inspect the results of the model-fit as it runs, without having to\n", + "wait for the non-linear search to terminate.\n", + " \n", + "The `output` folder includes:\n", + "\n", + " - `model.info`: Summarizes the model, its parameters and their priors discussed in the next tutorial.\n", + " \n", + " - `model.results`: Summarizes the highest likelihood model inferred so far including errors.\n", + " \n", + " - `images`: Visualization of the highest likelihood model-fit to the dataset, (e.g. a fit subplot showing the \n", + " galaxies, model data and residuals).\n", + " \n", + " - `files`: A folder containing .fits files of the dataset, the model as a human-readable .json file, \n", + " a `.csv` table of every non-linear search sample and other files containing information about the model-fit.\n", + " \n", + " - search.summary: A file providing summary statistics on the performance of the non-linear search.\n", + " \n", + " - `search_internal`: Internal files of the non-linear search (in this case Nautilus) used for resuming the fit and\n", + " visualizing the search.\n", + "\n", + "__Result__\n", + "\n", + "The search returns a result object, which whose `info` attribute shows the result in a readable format.\n", + "\n", + "[Above, we discussed that the `info_whitespace_length` parameter in the config files could b changed to make \n", + "the `model.info` attribute display optimally on your computer. This attribute also controls the whitespace of the\n", + "`result.info` attribute.]" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(result.info)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The `Result` object also contains:\n", + "\n", + " - The model corresponding to the maximum log likelihood solution in parameter space.\n", + " - The corresponding maximum log likelihood `Galaxies` and `FitImaging` objects." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(result.max_log_likelihood_instance)\n", + "\n", + "aplt.subplot_galaxies(galaxies=result.max_log_likelihood_galaxies, grid=result.grids.lp)\n", + "\n", + "aplt.subplot_fit_imaging(fit=result.max_log_likelihood_fit)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The result contains the full posterior information of our non-linear search, including all parameter samples, \n", + "log likelihood values and tools to compute the errors on the model. \n", + "\n", + "There are built in visualization tools for plotting this.\n", + "\n", + "The plot is labeled with short hand parameter names (e.g. `sersic_index` is mapped to the short hand \n", + "parameter `n`). These mappings ate specified in the `config/notation.yaml` file and can be customized by users.\n", + "\n", + "The superscripts of labels correspond to the name each component was given in the model (e.g. for the `Isothermal`\n", + "mass its name `mass` defined when making the `Model` above is used)." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.corner_cornerpy(samples=result.samples)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Loading From Output Folder__\n", + "\n", + "Everything the `Result` object contains has also been written to hard-disk, inside the fit's output folder. Each\n", + "file loads back into a full Python object with a single line \u2014 much faster and simpler than re-running the fit.\n", + "\n", + "For example, the maximum log likelihood `Galaxies` is saved as a `.json` file and the per-galaxy model images as\n", + "a `.fits` file:" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "from autoconf.dictable import from_json\n", + "\n", + "result_path = search.paths.output_path # Points at the fit's unique output folder.\n", + "\n", + "if (result_path / \"files\" / \"galaxies.json\").exists():\n", + " galaxies = from_json(file_path=result_path / \"files\" / \"galaxies.json\")\n", + "\n", + " galaxy_images = ag.Array2D.from_fits(\n", + " file_path=result_path / \"image\" / \"galaxy_images.fits\", hdu=0, pixel_scales=0.1\n", + " )" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The output folder also contains `model.json`, `samples.csv`, `dataset.fits`, `fit.fits` and more. A full walkthrough\n", + "of loading results from the output folder is given in:\n", + "\n", + " `autogalaxy_workspace/*/guides/results/start_here.py`\n", + "\n", + "For multi-fit, memory-efficient workflows using the aggregator instead, see:\n", + "\n", + " `autogalaxy_workspace/*/guides/results/aggregator/start_here.py`" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "\n", + "# %%\n", + "'''\n", + "This script gives a concise overview of the PyAutoGalaxy modeling API, fitting one the simplest models possible.\n", + "So, what next? \n", + "\n", + "__Features__\n", + "\n", + "The examples in the `autogalaxy_workspace/*/modeling/imaging/features` package illustrate other modeling features. \n", + "\n", + "We recommend you checkout the following features, because the make modeling in general more reliable and \n", + "efficient (you will therefore benefit from using these features irrespective of the quality of your data and \n", + "scientific topic of study).\n", + "\n", + "We recommend you now checkout the following feature:\n", + "\n", + "- ``linear_light_profiles``: The model light profiles use linear algebra to solve for their intensity, reducing model complexity.\n", + "\n", + "All other features may be useful to specific users with specific datasets and scientific goals, but are not useful\n", + "for general modeling.\n", + "\n", + "The folders `autogalaxy_workspace/*/modeling/imaging/searches` and `autogalaxy_workspace/*/modeling/imaging/customize`\n", + "provide guides on how to customize many other aspects of the model-fit. Check them out to see if anything\n", + "sounds useful, but for most users you can get by without using these forms of customization!\n", + " \n", + "__Data Preparation__\n", + "\n", + "If you are looking to fit your own CCD imaging data of a galaxy, checkout \n", + "the `autogalaxy_workspace/*/data_preparation/imaging/start_here.ipynb` script for an overview of how data should be \n", + "prepared before being modeled.\n", + "\n", + "__HowToGalaxy__\n", + "\n", + "This `start_here.py` script, and the features examples above, do not explain many details of how modeling is \n", + "performed, for example:\n", + "\n", + " - How does PyAutoGalaxy perform calculations which determine how a gaalxy emits light and therefore fit a model?\n", + " - How is the model fitted to data? What quantifies the goodness of fit (e.g. how is a log likelihood computed?).\n", + " - How does Nautilus find the highest likelihood models? What exactly is a \"non-linear search\"?\n", + "\n", + "You do not need to be able to answer these questions in order to fit models with PyAutoGalaxy and do science.\n", + "However, having a deeper understanding of how it all works is both interesting and will benefit you as a scientist\n", + "\n", + "This deeper insight is offered by the **HowToGalaxy** Jupyter notebook lectures, found \n", + "at https://github.com/PyAutoLabs/HowToGalaxy.\n", + "\n", + "I recommend that you check them out if you are interested in more details!\n", + "'''" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "code", + "metadata": {}, + "source": [], + "outputs": [], + "execution_count": null + } + ], + "metadata": { + "anaconda-cloud": {}, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.1" + } + }, + "nbformat": 4, + "nbformat_minor": 4 } \ No newline at end of file diff --git a/notebooks/interferometer/modeling.ipynb b/notebooks/interferometer/modeling.ipynb index 4ee81ab3..ba4f24bd 100644 --- a/notebooks/interferometer/modeling.ipynb +++ b/notebooks/interferometer/modeling.ipynb @@ -1,578 +1,578 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Modeling: Start Here\n", - "====================\n", - "\n", - "This script is the starting point for modeling of interferometer datasets with less than 1000\n", - "visibilities (e.g. SMA, ALMA) and it provides an overview of the modeling API.\n", - "\n", - "__Number of Visibilities__\n", - "\n", - "This example fits a **low-resolution interferometric dataset** with a small number of visibilities (273). The\n", - "dataset is intentionally minimal so that the example runs quickly and allows you to become familiar with the API\n", - "and modeling workflow. The code demonstrated in this example can feasible fit datasets with up to around 10000\n", - "visibilities, above which computational time and VRAM use become significant for this modeling approach.\n", - "\n", - "High-resolution datasets with many visibilities (e.g. high-quality ALMA observations\n", - "with **millions hundreds of millions of visibilities**) can be modeled efficiently. However, this requires\n", - "using the more advanced **pixelized source reconstructions** modeling approach. These large datasets fully\n", - "exploit **JAX acceleration**, enable modeling to run in **hours on a modern GPU**.\n", - "\n", - "If your dataset contains many visibilities, you should start by working through this example and the other examples\n", - "in the `interferometer` folder. Once you are comfortable with the API, the `feature/pixelization` package provides a\n", - "guided path toward efficiently modeling large interferometric datasets.\n", - "\n", - "The threshold between a dataset having many visibilities and therefore requiring pixelized source reconstructions, or\n", - "being small enough to be modeled with light profiles, is around **10,000 visibilities**.\n", - "\n", - "__Contents__\n", - "\n", - "**Number of Visibilities:** Discussion of dataset size and when to use pixelized source reconstructions.\n", - "**Model:** Description of the galaxy model fitted in this example.\n", - "**Mask:** Defining the real-space mask for the interferometer grid.\n", - "**Dataset:** Loading the interferometer dataset from FITS files.\n", - "**Dataset Auto-Simulation:** Automatically simulating data if it does not exist.\n", - "**Over Sampling:** Why over sampling is not needed for interferometer data.\n", - "**Model:** Composing a Sersic bulge and Exponential disk galaxy model using linear light profiles.\n", - "**Search:** Configuring the Nautilus nested sampling non-linear search.\n", - "**Analysis:** Setting up the AnalysisInterferometer object with JAX acceleration.\n", - "**VRAM Use:** Estimating GPU VRAM requirements for the model fit.\n", - "**Run Times:** Estimating the computational cost of the model fit.\n", - "**Model-Fit:** Running the non-linear search to fit the model to data.\n", - "**Output Folder:** Description of the results written to hard-disk during and after the fit.\n", - "**Result:** Inspecting the result object and maximum likelihood model.\n", - "**Features:** Overview of advanced interferometer modeling features like pixelizations.\n", - "**Data Preparation:** Pointers to data preparation scripts for your own data.\n", - "**HowToGalaxy:** Pointers to the HowToGalaxy lecture series for deeper understanding.\n", - "**Modeling Customization:** Overview of alternative non-linear searches and model customization.\n", - "\n", - "__Model__\n", - "\n", - "This script fits `Interferometer` dataset of a galaxy with a model where:\n", - "\n", - " - The galaxy's light is a linear parametric `Sersic` bulge and `Exponential` disk." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autoconf import setup_notebook; setup_notebook()\n", - "\n", - "from pathlib import Path\n", - "import autofit as af\n", - "import autogalaxy as ag\n", - "import autogalaxy.plot as aplt\n", - "import numpy as np" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Mask__\n", - "\n", - "We define the \u2018real_space_mask\u2019 which defines the grid the image the galaxy is evaluated using." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "mask_radius = 4.0\n", - "\n", - "real_space_mask = ag.Mask2D.circular(\n", - " shape_native=(256, 256),\n", - " pixel_scales=0.1,\n", - " radius=mask_radius,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset__\n", - "\n", - "Load and plot the galaxy `Interferometer` dataset `simple__sersic` from .fits files, which we will fit \n", - "with the model.\n", - "\n", - "This includes the method used to Fourier transform the real-space image of the galaxy to the uv-plane and compare \n", - "directly to the visiblities. We use a non-uniform fast Fourier transform, which is the most efficient method for \n", - "interferometer datasets containing ~1-10 million visibilities." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_name = \"simple\"\n", - "dataset_path = Path(\"dataset\") / \"interferometer\" / dataset_name" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset Auto-Simulation__\n", - "\n", - "If the dataset does not already exist on your system, it will be created by running the corresponding\n", - "simulator script. This ensures that all example scripts can be run without manually simulating data first." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "if not dataset_path.exists():\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.run(\n", - " [sys.executable, \"scripts/interferometer/simulator.py\"],\n", - " check=True,\n", - " )\n", - "\n", - "\n", - "dataset = ag.Interferometer.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " uv_wavelengths_path=dataset_path / \"uv_wavelengths.fits\",\n", - " real_space_mask=real_space_mask,\n", - " transformer_class=ag.TransformerDFT,\n", - ")\n", - "\n", - "aplt.subplot_interferometer_dirty_images(dataset=dataset)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Over Sampling__\n", - "\n", - "If you are familiar with using imaging data, you may have seen that a numerical technique called over sampling is used, \n", - "which evaluates light profiles on a higher resolution grid than the image data to ensure the calculation is accurate.\n", - "\n", - "Interferometer does not observe galaxies in a way where over sampling is necessary, therefore all interferometer\n", - "calculations are performed without over sampling.\n", - "\n", - "__Model__\n", - "\n", - "We compose our model using `Model` objects, which represent the galaxies we fit to our data. In this \n", - "example we fit a model where:\n", - "\n", - " - The galaxy's light is a linear parametric `Sersic` bulge and `Exponential` disk, the centres of \n", - " which are aligned [10 parameters].\n", - " \n", - "The number of free parameters and therefore the dimensionality of non-linear parameter space is N=10.\n", - "\n", - "__Linear Light Profiles__\n", - "\n", - "The model below uses a `linear light profile` for the bulge and disk, via the API `lp_linear`. This is a specific type \n", - "of light profile that solves for the `intensity` of each profile that best fits the data via a linear inversion. \n", - "This means it is not a free parameter, reducing the dimensionality of non-linear parameter space. \n", - "\n", - "Linear light profiles significantly improve the speed, accuracy and reliability of modeling and they are used\n", - "by default in every modeling example. A full description of linear light profiles is provided in the\n", - "`autogalaxy_workspace/*/modeling/imaging/features/linear_light_profiles.py` example.\n", - "\n", - "A standard light profile can be used if you change the `lp_linear` to `lp`, but it is not recommended.\n", - "\n", - "__Coordinates__\n", - "\n", - "The model fitting default settings assume that the galaxy centre is near the coordinates (0.0\", 0.0\"). \n", - "\n", - "If for your dataset the galaxy is not centred at (0.0\", 0.0\"), we recommend that you either: \n", - "\n", - " - Reduce your data so that the centre is (`autogalaxy_workspace/*/preprocess`). \n", - " - Manually override the model priors (`autogalaxy_workspace/*/modeling/imaging/customize/priors.py`)." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "bulge = af.Model(ag.lp_linear.Sersic)\n", - "disk = af.Model(ag.lp_linear.Exponential)\n", - "bulge.centre = disk.centre\n", - "\n", - "galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=bulge, disk=disk)\n", - "\n", - "model = af.Collection(galaxies=af.Collection(galaxy=galaxy))" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `info` attribute shows the model in a readable format.\n", - "\n", - "[The `info` below may not display optimally on your computer screen, for example the whitespace between parameter\n", - "names on the left and parameter priors on the right may lead them to appear across multiple lines. This is a\n", - "common issue in Jupyter notebooks.\n", - "\n", - "The`info_whitespace_length` parameter in the file `config/generag.yaml` in the [output] section can be changed to \n", - "increase or decrease the amount of whitespace (The Jupyter notebook kernel will need to be reset for this change to \n", - "appear in a notebook).]" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(model.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Search__\n", - "\n", - "The model is fitted to the data using a non-linear search. In this example, we use the nested sampling algorithm \n", - "Nautilus (https://nautilus.readthedocs.io/en/latest/).\n", - "\n", - "The folders: \n", - "\n", - " - `autogalaxy_workspace/*/modeling/imaging/searches`.\n", - " - `autogalaxy_workspace/*/modeling/imaging/customize`\n", - " \n", - "Give overviews of the non-linear searches **PyAutoGalaxy** supports and more details on how to customize the\n", - "model-fit, including the priors on the model. \n", - "\n", - "The `name` and `path_prefix` below specify the path where results are stored in the output folder: \n", - "\n", - " `/autogalaxy_workspace/output/imaging/simple__sersic/mass[sie]/unique_identifier`.\n", - "\n", - "__Unique Identifier__\n", - "\n", - "In the path above, the `unique_identifier` appears as a collection of characters, where this identifier is generated \n", - "based on the model, search and dataset that are used in the fit.\n", - " \n", - "An identical combination of model, search and dataset generates the same identifier, meaning that rerunning the\n", - "script will use the existing results to resume the model-fit. In contrast, if you change the model, search or dataset,\n", - "a new unique identifier will be generated, ensuring that the model-fit results are output into a separate folder." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "search = af.Nautilus(\n", - " path_prefix=Path(\"interferometer\", \"modeling\"),\n", - " name=\"start_here\",\n", - " unique_tag=dataset_name,\n", - " n_live=100,\n", - " n_batch=50, # GPU model fits are batched and run simultaneously, see VRAM section below.\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Analysis__\n", - "\n", - "The `AnalysisInterferometer` object defines the `log_likelihood_function` used by the non-linear search to fit the \n", - "model to the `Interferometer`dataset.\n", - "\n", - "__JAX__\n", - "\n", - "PyAutouses JAX under the hood for fast GPU/CPU acceleration. If JAX is installed with GPU\n", - "support, your fits will run much faster (around 10 minutes instead of an hour). If only a CPU is available,\n", - "JAX will still provide a speed up via multithreading, with fits taking around 20-30 minutes.\n", - "\n", - "If you don\u2019t have a GPU locally, consider Google Colab which provides free GPUs, so your modeling runs are much faster." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "analysis = ag.AnalysisInterferometer(dataset=dataset, use_jax=True)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__VRAM Use__\n", - "\n", - "When running Autowith JAX on a GPU, the analysis must fit within the GPU\u2019s available VRAM. If insufficient \n", - "VRAM is available, the analysis will fail with an out-of-memory error, typically during JIT compilation or the \n", - "first likelihood call.\n", - "\n", - "Two factors dictate the VRAM usage of an analysis:\n", - "\n", - "- The number of arrays and other data structures JAX must store in VRAM to fit the model\n", - " to the data in the likelihood function. This is dictated by the model complexity and dataset size.\n", - "\n", - "- The `batch_size` sets how many likelihood evaluations are performed simultaneously.\n", - " Increasing the batch size increases VRAM usage but can reduce overall run time,\n", - " while decreasing it lowers VRAM usage at the cost of slower execution.\n", - "\n", - "Before running an analysis, users should check that the estimated VRAM usage for the\n", - "chosen batch size is comfortably below their GPU\u2019s total VRAM.\n", - "\n", - "The method below prints the VRAM usage estimate for the analysis and model with the specified batch size,\n", - "it takes about 20-30 seconds to run so you may want to comment it out once you are familiar with your GPU's VRAM limits.\n", - "\n", - "For a MGE model with the low visibility dataset fitted in this example VRAM use is relatively low (~0.3GB) For other \n", - "models (e.g. pixelized sources) and datasets with more visibilities it can be much higher (> 1GB going beyond 10GB)." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "analysis.print_vram_use(model=model, batch_size=search.batch_size)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Run Times__\n", - "\n", - "Modeling can be a computationally expensive process. When fitting complex models to high resolution datasets \n", - "run times can be of order hours, days, weeks or even months.\n", - "\n", - "Run times are dictated by two factors:\n", - "\n", - " - The log likelihood evaluation time: the time it takes for a single `instance` of the model to be fitted to \n", - " the dataset such that a log likelihood is returned.\n", - " \n", - " - The number of iterations (e.g. log likelihood evaluations) performed by the non-linear search: more complex lens\n", - " models require more iterations to converge to a solution.\n", - " \n", - "For this analysis, the log likelihood evaluation time is ~0.01 seconds on CPU, < 0.001 seconds on GPU, which is \n", - "extremely fast for modeling. \n", - "\n", - "To estimate the expected overall run time of the model-fit we multiply the log likelihood evaluation time by an \n", - "estimate of the number of iterations the non-linear search will perform. For this model, this is typically around\n", - "? iterations, meaning that this script takes ? on CPU and ? on GPU.\n", - "\n", - "__Model-Fit__\n", - "\n", - "We can now begin the model-fit by passing the model and analysis object to the search, which performs the \n", - "Nautilus non-linear search in order to find which models fit the data with the highest likelihood.\n", - "\n", - "**Run Time Error:** On certain operating systems (e.g. Windows, Linux) and Python versions, the code below may produce \n", - "an error. If this occurs, see the `autolens_workspace/guides/modeling/bug_fix` example for a fix." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "result = search.fit(model=model, analysis=analysis)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Output Folder__\n", - "\n", - "Now this is running you should checkout the `autogalaxy_workspace/output` folder. This is where the results of the \n", - "search are written to hard-disk (in the `start_here` folder), where all outputs are human readable (e.g. as .json,\n", - ".csv or text files).\n", - "\n", - "As the fit progresses, results are written to the `output` folder on the fly using the highest likelihood model found\n", - "by the non-linear search so far. This means you can inspect the results of the model-fit as it runs, without having to\n", - "wait for the non-linear search to terminate.\n", - " \n", - "The `output` folder includes:\n", - "\n", - " - `model.info`: Summarizes the model, its parameters and their priors discussed in the next tutorial.\n", - " \n", - " - `model.results`: Summarizes the highest likelihood model inferred so far including errors.\n", - " \n", - " - `images`: Visualization of the highest likelihood model-fit to the dataset, (e.g. a fit subplot showing the \n", - " galaxies, model data and residuals).\n", - " \n", - " - `files`: A folder containing .fits files of the dataset, the model as a human-readable .json file, \n", - " a `.csv` table of every non-linear search sample and other files containing information about the model-fit.\n", - " \n", - " - search.summary: A file providing summary statistics on the performance of the non-linear search.\n", - " \n", - " - `search_internal`: Internal files of the non-linear search (in this case Nautilus) used for resuming the fit and\n", - " visualizing the search.\n", - "\n", - "__Result__\n", - "\n", - "The search returns a result object, which whose `info` attribute shows the result in a readable format.\n", - "\n", - "[Above, we discussed that the `info_whitespace_length` parameter in the config files could b changed to make \n", - "the `model.info` attribute display optimally on your computer. This attribute also controls the whitespace of the\n", - "`result.info` attribute.]" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(result.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `Result` object also contains:\n", - "\n", - " - The model corresponding to the maximum log likelihood solution in parameter space.\n", - " - The corresponding maximum log likelihood `Galaxies` and `FitInterferometer` objects." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(result.max_log_likelihood_instance)\n", - "\n", - "aplt.subplot_galaxies(\n", - " galaxies=result.max_log_likelihood_galaxies,\n", - " grid=real_space_mask.derive_grid.unmasked,\n", - ")\n", - "aplt.subplot_fit_dirty_images(fit=result.max_log_likelihood_fit)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The result contains the full posterior information of our non-linear search, including all parameter samples, \n", - "log likelihood values and tools to compute the errors on the model. \n", - "\n", - "There are built in visualization tools for plotting this.\n", - "\n", - "The plot is labeled with short hand parameter names (e.g. `sersic_index` is mapped to the short hand \n", - "parameter `n`). These mappings ate specified in the `config/notation.yaml` file and can be customized by users.\n", - "\n", - "The superscripts of labels correspond to the name each component was given in the model (e.g. for the `Isothermal`\n", - "mass its name `mass` defined when making the `Model` above is used)." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "# %%\n", - "'''\n", - "This script gives a concise overview of the basic modeling API, fitting one of the simplest galaxy models possible.\n", - "\n", - "Let\u2019s now consider what features you should read about to improve your galaxy modeling, especially if you are aiming\n", - "to fit more complex models to your data.\n", - "\n", - "__Features__\n", - "\n", - "The examples in the `autogalaxy_workspace/*/interferometer/features` package illustrate other galaxy modeling\n", - "features.\n", - "\n", - "We recommend you check out just one feature next, because it makes galaxy modeling of interferometer datasets\n", - "more reliable and efficient, and will then allow you to model high-resolution datasets with many visibilities:\n", - "\n", - "- ``pixelization``: The galaxy\u2019s light is reconstructed using an adaptive Rectangular mesh or Delaunay mesh.\n", - "\n", - "The files `autogalaxy_workspace/*/guides/modeling/searches` and\n", - "`autogalaxy_workspace/*/guides/modeling/customize` provide guides on how to customize many other aspects of the\n", - "model-fit. Check them out to see if anything sounds useful, but for most users you can get by without using these\n", - "forms of customization!\n", - "\n", - "__Data Preparation__\n", - "\n", - "If you are looking to fit your own interferometer data of a galaxy, check out\n", - "the `autogalaxy_workspace/*/interferometer/data_preparation/start_here.ipynb` script for an overview of how data\n", - "should be prepared before being modeled.\n", - "\n", - "__HowToGalaxy__\n", - "\n", - "This `start_here.py` script, and the features examples above, do not explain many details of how galaxy modeling is\n", - "performed, for example:\n", - "\n", - "- How does PyAutoGalaxy evaluate galaxy light profiles and perform image-plane calculations?\n", - "- How is a galaxy model fitted to data? What quantifies the goodness of fit (e.g. how is a log likelihood computed)?\n", - "- How does Nautilus find the highest likelihood galaxy models? What exactly is a \u201cnon-linear search\u201d?\n", - "\n", - "You do not need to be able to answer these questions in order to fit galaxy models with PyAutoGalaxy and do science.\n", - "However, having a deeper understanding of how it all works is both interesting and will benefit you as a scientist.\n", - "\n", - "This deeper insight is offered by the **HowToGalaxy** Jupyter notebook lectures, found\n", - "at `autogalaxy_workspace/*/howtogalaxy`.\n", - "\n", - "I recommend that you check them out if you are interested in more details!\n", - "\n", - "__Modeling Customization__\n", - "\n", - "The folders `autogalaxy_workspace/*/guides/modeling/searches` give an overview of alternative non-linear searches,\n", - "other than Nautilus, that can be used to fit galaxy models.\n", - "\n", - "They also provide details on how to customize the model-fit, for example the priors.\n", - "'''" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Modeling: Start Here\n", + "====================\n", + "\n", + "This script is the starting point for modeling of interferometer datasets with less than 1000\n", + "visibilities (e.g. SMA, ALMA) and it provides an overview of the modeling API.\n", + "\n", + "__Number of Visibilities__\n", + "\n", + "This example fits a **low-resolution interferometric dataset** with a small number of visibilities (273). The\n", + "dataset is intentionally minimal so that the example runs quickly and allows you to become familiar with the API\n", + "and modeling workflow. The code demonstrated in this example can feasible fit datasets with up to around 10000\n", + "visibilities, above which computational time and VRAM use become significant for this modeling approach.\n", + "\n", + "High-resolution datasets with many visibilities (e.g. high-quality ALMA observations\n", + "with **millions hundreds of millions of visibilities**) can be modeled efficiently. However, this requires\n", + "using the more advanced **pixelized source reconstructions** modeling approach. These large datasets fully\n", + "exploit **JAX acceleration**, enable modeling to run in **hours on a modern GPU**.\n", + "\n", + "If your dataset contains many visibilities, you should start by working through this example and the other examples\n", + "in the `interferometer` folder. Once you are comfortable with the API, the `feature/pixelization` package provides a\n", + "guided path toward efficiently modeling large interferometric datasets.\n", + "\n", + "The threshold between a dataset having many visibilities and therefore requiring pixelized source reconstructions, or\n", + "being small enough to be modeled with light profiles, is around **10,000 visibilities**.\n", + "\n", + "__Contents__\n", + "\n", + "**Number of Visibilities:** Discussion of dataset size and when to use pixelized source reconstructions.\n", + "**Model:** Description of the galaxy model fitted in this example.\n", + "**Mask:** Defining the real-space mask for the interferometer grid.\n", + "**Dataset:** Loading the interferometer dataset from FITS files.\n", + "**Dataset Auto-Simulation:** Automatically simulating data if it does not exist.\n", + "**Over Sampling:** Why over sampling is not needed for interferometer data.\n", + "**Model:** Composing a Sersic bulge and Exponential disk galaxy model using linear light profiles.\n", + "**Search:** Configuring the Nautilus nested sampling non-linear search.\n", + "**Analysis:** Setting up the AnalysisInterferometer object with JAX acceleration.\n", + "**VRAM Use:** Estimating GPU VRAM requirements for the model fit.\n", + "**Run Times:** Estimating the computational cost of the model fit.\n", + "**Model-Fit:** Running the non-linear search to fit the model to data.\n", + "**Output Folder:** Description of the results written to hard-disk during and after the fit.\n", + "**Result:** Inspecting the result object and maximum likelihood model.\n", + "**Features:** Overview of advanced interferometer modeling features like pixelizations.\n", + "**Data Preparation:** Pointers to data preparation scripts for your own data.\n", + "**HowToGalaxy:** Pointers to the HowToGalaxy lecture series for deeper understanding.\n", + "**Modeling Customization:** Overview of alternative non-linear searches and model customization.\n", + "\n", + "__Model__\n", + "\n", + "This script fits `Interferometer` dataset of a galaxy with a model where:\n", + "\n", + " - The galaxy's light is a linear parametric `Sersic` bulge and `Exponential` disk." + ] }, - "nbformat": 4, - "nbformat_minor": 4 + { + "cell_type": "code", + "metadata": {}, + "source": [ + "\n", + "from autoconf import setup_notebook; setup_notebook()\n", + "\n", + "from pathlib import Path\n", + "import autofit as af\n", + "import autogalaxy as ag\n", + "import autogalaxy.plot as aplt\n", + "import numpy as np" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Mask__\n", + "\n", + "We define the \u2018real_space_mask\u2019 which defines the grid the image the galaxy is evaluated using." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "mask_radius = 4.0\n", + "\n", + "real_space_mask = ag.Mask2D.circular(\n", + " shape_native=(256, 256),\n", + " pixel_scales=0.1,\n", + " radius=mask_radius,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Dataset__\n", + "\n", + "Load and plot the galaxy `Interferometer` dataset `simple__sersic` from .fits files, which we will fit \n", + "with the model.\n", + "\n", + "This includes the method used to Fourier transform the real-space image of the galaxy to the uv-plane and compare \n", + "directly to the visiblities. We use a non-uniform fast Fourier transform, which is the most efficient method for \n", + "interferometer datasets containing ~1-10 million visibilities." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset_name = \"simple\"\n", + "dataset_path = Path(\"dataset\") / \"interferometer\" / dataset_name" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Dataset Auto-Simulation__\n", + "\n", + "If the dataset does not already exist on your system, it will be created by running the corresponding\n", + "simulator script. This ensures that all example scripts can be run without manually simulating data first." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "if not dataset_path.exists():\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.run(\n", + " [sys.executable, \"scripts/interferometer/simulator.py\"],\n", + " check=True,\n", + " )\n", + "\n", + "\n", + "dataset = ag.Interferometer.from_fits(\n", + " data_path=dataset_path / \"data.fits\",\n", + " noise_map_path=dataset_path / \"noise_map.fits\",\n", + " uv_wavelengths_path=dataset_path / \"uv_wavelengths.fits\",\n", + " real_space_mask=real_space_mask,\n", + " transformer_class=ag.TransformerDFT,\n", + ")\n", + "\n", + "aplt.subplot_interferometer_dirty_images(dataset=dataset)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Over Sampling__\n", + "\n", + "If you are familiar with using imaging data, you may have seen that a numerical technique called over sampling is used, \n", + "which evaluates light profiles on a higher resolution grid than the image data to ensure the calculation is accurate.\n", + "\n", + "Interferometer does not observe galaxies in a way where over sampling is necessary, therefore all interferometer\n", + "calculations are performed without over sampling.\n", + "\n", + "__Model__\n", + "\n", + "We compose our model using `Model` objects, which represent the galaxies we fit to our data. In this \n", + "example we fit a model where:\n", + "\n", + " - The galaxy's light is a linear parametric `Sersic` bulge and `Exponential` disk, the centres of \n", + " which are aligned [10 parameters].\n", + " \n", + "The number of free parameters and therefore the dimensionality of non-linear parameter space is N=10.\n", + "\n", + "__Linear Light Profiles__\n", + "\n", + "The model below uses a `linear light profile` for the bulge and disk, via the API `lp_linear`. This is a specific type \n", + "of light profile that solves for the `intensity` of each profile that best fits the data via a linear inversion. \n", + "This means it is not a free parameter, reducing the dimensionality of non-linear parameter space. \n", + "\n", + "Linear light profiles significantly improve the speed, accuracy and reliability of modeling and they are used\n", + "by default in every modeling example. A full description of linear light profiles is provided in the\n", + "`autogalaxy_workspace/*/modeling/imaging/features/linear_light_profiles.py` example.\n", + "\n", + "A standard light profile can be used if you change the `lp_linear` to `lp`, but it is not recommended.\n", + "\n", + "__Coordinates__\n", + "\n", + "The model fitting default settings assume that the galaxy centre is near the coordinates (0.0\", 0.0\"). \n", + "\n", + "If for your dataset the galaxy is not centred at (0.0\", 0.0\"), we recommend that you either: \n", + "\n", + " - Reduce your data so that the centre is (`autogalaxy_workspace/*/preprocess`). \n", + " - Manually override the model priors (`autogalaxy_workspace/*/modeling/imaging/customize/priors.py`)." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "bulge = af.Model(ag.lp_linear.Sersic)\n", + "disk = af.Model(ag.lp_linear.Exponential)\n", + "bulge.centre = disk.centre\n", + "\n", + "galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=bulge, disk=disk)\n", + "\n", + "model = af.Collection(galaxies=af.Collection(galaxy=galaxy))" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The `info` attribute shows the model in a readable format.\n", + "\n", + "[The `info` below may not display optimally on your computer screen, for example the whitespace between parameter\n", + "names on the left and parameter priors on the right may lead them to appear across multiple lines. This is a\n", + "common issue in Jupyter notebooks.\n", + "\n", + "The`info_whitespace_length` parameter in the file `config/generag.yaml` in the [output] section can be changed to \n", + "increase or decrease the amount of whitespace (The Jupyter notebook kernel will need to be reset for this change to \n", + "appear in a notebook).]" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(model.info)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Search__\n", + "\n", + "The model is fitted to the data using a non-linear search. In this example, we use the nested sampling algorithm \n", + "Nautilus (https://nautilus.readthedocs.io/en/latest/).\n", + "\n", + "The folders: \n", + "\n", + " - `autogalaxy_workspace/*/modeling/imaging/searches`.\n", + " - `autogalaxy_workspace/*/modeling/imaging/customize`\n", + " \n", + "Give overviews of the non-linear searches **PyAutoGalaxy** supports and more details on how to customize the\n", + "model-fit, including the priors on the model. \n", + "\n", + "The `name` and `path_prefix` below specify the path where results are stored in the output folder: \n", + "\n", + " `/autogalaxy_workspace/output/imaging/simple__sersic/mass[sie]/unique_identifier`.\n", + "\n", + "__Unique Identifier__\n", + "\n", + "In the path above, the `unique_identifier` appears as a collection of characters, where this identifier is generated \n", + "based on the model, search and dataset that are used in the fit.\n", + " \n", + "An identical combination of model, search and dataset generates the same identifier, meaning that rerunning the\n", + "script will use the existing results to resume the model-fit. In contrast, if you change the model, search or dataset,\n", + "a new unique identifier will be generated, ensuring that the model-fit results are output into a separate folder." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "search = af.Nautilus(\n", + " path_prefix=Path(\"interferometer\", \"modeling\"),\n", + " name=\"start_here\",\n", + " unique_tag=dataset_name,\n", + " n_live=100,\n", + " n_batch=50, # GPU model fits are batched and run simultaneously, see VRAM section below.\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Analysis__\n", + "\n", + "The `AnalysisInterferometer` object defines the `log_likelihood_function` used by the non-linear search to fit the \n", + "model to the `Interferometer`dataset.\n", + "\n", + "__JAX__\n", + "\n", + "PyAutouses JAX under the hood for fast GPU/CPU acceleration. If JAX is installed with GPU\n", + "support, your fits will run much faster (around 10 minutes instead of an hour). If only a CPU is available,\n", + "JAX will still provide a speed up via multithreading, with fits taking around 20-30 minutes.\n", + "\n", + "If you don\u2019t have a GPU locally, consider Google Colab which provides free GPUs, so your modeling runs are much faster." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "analysis = ag.AnalysisInterferometer(dataset=dataset, use_jax=True)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__VRAM Use__\n", + "\n", + "When running Autowith JAX on a GPU, the analysis must fit within the GPU\u2019s available VRAM. If insufficient \n", + "VRAM is available, the analysis will fail with an out-of-memory error, typically during JIT compilation or the \n", + "first likelihood call.\n", + "\n", + "Two factors dictate the VRAM usage of an analysis:\n", + "\n", + "- The number of arrays and other data structures JAX must store in VRAM to fit the model\n", + " to the data in the likelihood function. This is dictated by the model complexity and dataset size.\n", + "\n", + "- The `batch_size` sets how many likelihood evaluations are performed simultaneously.\n", + " Increasing the batch size increases VRAM usage but can reduce overall run time,\n", + " while decreasing it lowers VRAM usage at the cost of slower execution.\n", + "\n", + "Before running an analysis, users should check that the estimated VRAM usage for the\n", + "chosen batch size is comfortably below their GPU\u2019s total VRAM.\n", + "\n", + "The method below prints the VRAM usage estimate for the analysis and model with the specified batch size,\n", + "it takes about 20-30 seconds to run so you may want to comment it out once you are familiar with your GPU's VRAM limits.\n", + "\n", + "For a MGE model with the low visibility dataset fitted in this example VRAM use is relatively low (~0.3GB) For other \n", + "models (e.g. pixelized sources) and datasets with more visibilities it can be much higher (> 1GB going beyond 10GB)." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "analysis.print_vram_use(model=model, batch_size=search.batch_size)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Run Times__\n", + "\n", + "Modeling can be a computationally expensive process. When fitting complex models to high resolution datasets \n", + "run times can be of order hours, days, weeks or even months.\n", + "\n", + "Run times are dictated by two factors:\n", + "\n", + " - The log likelihood evaluation time: the time it takes for a single `instance` of the model to be fitted to \n", + " the dataset such that a log likelihood is returned.\n", + " \n", + " - The number of iterations (e.g. log likelihood evaluations) performed by the non-linear search: more complex lens\n", + " models require more iterations to converge to a solution.\n", + " \n", + "For this analysis, the log likelihood evaluation time is ~0.01 seconds on CPU, < 0.001 seconds on GPU, which is \n", + "extremely fast for modeling. \n", + "\n", + "To estimate the expected overall run time of the model-fit we multiply the log likelihood evaluation time by an \n", + "estimate of the number of iterations the non-linear search will perform. For this model, this is typically around\n", + "? iterations, meaning that this script takes ? on CPU and ? on GPU.\n", + "\n", + "__Model-Fit__\n", + "\n", + "We can now begin the model-fit by passing the model and analysis object to the search, which performs the \n", + "Nautilus non-linear search in order to find which models fit the data with the highest likelihood.\n", + "\n", + "**Run Time Error:** On certain operating systems (e.g. Windows, Linux) and Python versions, the code below may produce \n", + "an error. If this occurs, see the `autolens_workspace/guides/modeling/bug_fix` example for a fix." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "result = search.fit(model=model, analysis=analysis)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Output Folder__\n", + "\n", + "Now this is running you should checkout the `autogalaxy_workspace/output` folder. This is where the results of the \n", + "search are written to hard-disk (in the `start_here` folder), where all outputs are human readable (e.g. as .json,\n", + ".csv or text files).\n", + "\n", + "As the fit progresses, results are written to the `output` folder on the fly using the highest likelihood model found\n", + "by the non-linear search so far. This means you can inspect the results of the model-fit as it runs, without having to\n", + "wait for the non-linear search to terminate.\n", + " \n", + "The `output` folder includes:\n", + "\n", + " - `model.info`: Summarizes the model, its parameters and their priors discussed in the next tutorial.\n", + " \n", + " - `model.results`: Summarizes the highest likelihood model inferred so far including errors.\n", + " \n", + " - `images`: Visualization of the highest likelihood model-fit to the dataset, (e.g. a fit subplot showing the \n", + " galaxies, model data and residuals).\n", + " \n", + " - `files`: A folder containing .fits files of the dataset, the model as a human-readable .json file, \n", + " a `.csv` table of every non-linear search sample and other files containing information about the model-fit.\n", + " \n", + " - search.summary: A file providing summary statistics on the performance of the non-linear search.\n", + " \n", + " - `search_internal`: Internal files of the non-linear search (in this case Nautilus) used for resuming the fit and\n", + " visualizing the search.\n", + "\n", + "__Result__\n", + "\n", + "The search returns a result object, which whose `info` attribute shows the result in a readable format.\n", + "\n", + "[Above, we discussed that the `info_whitespace_length` parameter in the config files could b changed to make \n", + "the `model.info` attribute display optimally on your computer. This attribute also controls the whitespace of the\n", + "`result.info` attribute.]" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(result.info)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The `Result` object also contains:\n", + "\n", + " - The model corresponding to the maximum log likelihood solution in parameter space.\n", + " - The corresponding maximum log likelihood `Galaxies` and `FitInterferometer` objects." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(result.max_log_likelihood_instance)\n", + "\n", + "aplt.subplot_galaxies(\n", + " galaxies=result.max_log_likelihood_galaxies,\n", + " grid=real_space_mask.derive_grid.unmasked,\n", + ")\n", + "aplt.subplot_fit_dirty_images(fit=result.max_log_likelihood_fit)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The result contains the full posterior information of our non-linear search, including all parameter samples, \n", + "log likelihood values and tools to compute the errors on the model. \n", + "\n", + "There are built in visualization tools for plotting this.\n", + "\n", + "The plot is labeled with short hand parameter names (e.g. `sersic_index` is mapped to the short hand \n", + "parameter `n`). These mappings ate specified in the `config/notation.yaml` file and can be customized by users.\n", + "\n", + "The superscripts of labels correspond to the name each component was given in the model (e.g. for the `Isothermal`\n", + "mass its name `mass` defined when making the `Model` above is used)." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "\n", + "# %%\n", + "'''\n", + "This script gives a concise overview of the basic modeling API, fitting one of the simplest galaxy models possible.\n", + "\n", + "Let\u2019s now consider what features you should read about to improve your galaxy modeling, especially if you are aiming\n", + "to fit more complex models to your data.\n", + "\n", + "__Features__\n", + "\n", + "The examples in the `autogalaxy_workspace/*/interferometer/features` package illustrate other galaxy modeling\n", + "features.\n", + "\n", + "We recommend you check out just one feature next, because it makes galaxy modeling of interferometer datasets\n", + "more reliable and efficient, and will then allow you to model high-resolution datasets with many visibilities:\n", + "\n", + "- ``pixelization``: The galaxy\u2019s light is reconstructed using an adaptive Rectangular mesh or Delaunay mesh.\n", + "\n", + "The files `autogalaxy_workspace/*/guides/modeling/searches` and\n", + "`autogalaxy_workspace/*/guides/modeling/customize` provide guides on how to customize many other aspects of the\n", + "model-fit. Check them out to see if anything sounds useful, but for most users you can get by without using these\n", + "forms of customization!\n", + "\n", + "__Data Preparation__\n", + "\n", + "If you are looking to fit your own interferometer data of a galaxy, check out\n", + "the `autogalaxy_workspace/*/interferometer/data_preparation/start_here.ipynb` script for an overview of how data\n", + "should be prepared before being modeled.\n", + "\n", + "__HowToGalaxy__\n", + "\n", + "This `start_here.py` script, and the features examples above, do not explain many details of how galaxy modeling is\n", + "performed, for example:\n", + "\n", + "- How does PyAutoGalaxy evaluate galaxy light profiles and perform image-plane calculations?\n", + "- How is a galaxy model fitted to data? What quantifies the goodness of fit (e.g. how is a log likelihood computed)?\n", + "- How does Nautilus find the highest likelihood galaxy models? What exactly is a \u201cnon-linear search\u201d?\n", + "\n", + "You do not need to be able to answer these questions in order to fit galaxy models with PyAutoGalaxy and do science.\n", + "However, having a deeper understanding of how it all works is both interesting and will benefit you as a scientist.\n", + "\n", + "This deeper insight is offered by the **HowToGalaxy** Jupyter notebook lectures, found\n", + "at https://github.com/PyAutoLabs/HowToGalaxy.\n", + "\n", + "I recommend that you check them out if you are interested in more details!\n", + "\n", + "__Modeling Customization__\n", + "\n", + "The folders `autogalaxy_workspace/*/guides/modeling/searches` give an overview of alternative non-linear searches,\n", + "other than Nautilus, that can be used to fit galaxy models.\n", + "\n", + "They also provide details on how to customize the model-fit, for example the priors.\n", + "'''" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "code", + "metadata": {}, + "source": [], + "outputs": [], + "execution_count": null + } + ], + "metadata": { + "anaconda-cloud": {}, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.1" + } + }, + "nbformat": 4, + "nbformat_minor": 4 } \ No newline at end of file diff --git a/notebooks/multi/plot.ipynb b/notebooks/multi/plot.ipynb index 8e3e4a49..78298ca2 100644 --- a/notebooks/multi/plot.ipynb +++ b/notebooks/multi/plot.ipynb @@ -79,7 +79,7 @@ " import sys\n", "\n", " subprocess.run(\n", - " [sys.executable, \"scripts/howtogalaxy/simulators/sersic.py\"],\n", + " [sys.executable, \"scripts/imaging/simulator_sersic.py\"],\n", " check=True,\n", " )\n", "\n", diff --git a/scripts/README.rst b/scripts/README.rst index 61bd965c..35be7dd6 100644 --- a/scripts/README.rst +++ b/scripts/README.rst @@ -15,4 +15,6 @@ Folders - ``interferometer``: Examples for galaxies observed with an interferometer (e.g. ALMA, JVLA). - ``ellipse``: Examples for perform ellipse isophote fitting on galaxy images. - ``multi``: Examples for multiple datasets simultaneously (E.g. multi-wavelength imaging, imaging and interferometry). -- ``howtogalaxy``: The **HowToGalaxy** lectures which teach inexperienced scientists what galaxy fitting is and how to use **PyAutoGalaxy**. \ No newline at end of file + +The **HowToGalaxy** lecture series, which teaches inexperienced scientists what galaxy fitting is and how to use +**PyAutoGalaxy**, now lives in its own repository: https://github.com/PyAutoLabs/HowToGalaxy \ No newline at end of file diff --git a/scripts/ellipse/modeling.py b/scripts/ellipse/modeling.py index f86fd5d4..3f1fc8b0 100644 --- a/scripts/ellipse/modeling.py +++ b/scripts/ellipse/modeling.py @@ -553,7 +553,7 @@ This example script above explains ellipse fitting, but there are many other ways to model a galaxy, using light profiles which represent its surface brightness. -This is explained in the **HowToGalaxy** Jupyter notebook lectures, found at `autogalaxy_workspace/*/howtogalaxy`. +This is explained in the **HowToGalaxy** Jupyter notebook lectures at https://github.com/PyAutoLabs/HowToGalaxy. I recommend that you check them out if you are interested in more details! """ diff --git a/scripts/guides/advanced/over_sampling.py b/scripts/guides/advanced/over_sampling.py index fa4273bd..72e1210c 100644 --- a/scripts/guides/advanced/over_sampling.py +++ b/scripts/guides/advanced/over_sampling.py @@ -368,7 +368,7 @@ import sys subprocess.run( - [sys.executable, "scripts/howtogalaxy/simulators/sersic.py"], + [sys.executable, "scripts/imaging/simulator_sersic.py"], check=True, ) diff --git a/scripts/guides/modeling/customize.py b/scripts/guides/modeling/customize.py index 8ffefe17..344caf81 100644 --- a/scripts/guides/modeling/customize.py +++ b/scripts/guides/modeling/customize.py @@ -44,7 +44,7 @@ import sys subprocess.run( - [sys.executable, "scripts/howtogalaxy/simulators/sersic.py"], + [sys.executable, "scripts/imaging/simulator_sersic.py"], check=True, ) diff --git a/scripts/guides/plot/examples/searches.py b/scripts/guides/plot/examples/searches.py index 0de6b755..dfaaffa0 100644 --- a/scripts/guides/plot/examples/searches.py +++ b/scripts/guides/plot/examples/searches.py @@ -47,7 +47,7 @@ import sys subprocess.run( - [sys.executable, "scripts/howtogalaxy/simulators/sersic.py"], + [sys.executable, "scripts/imaging/simulator_sersic.py"], check=True, ) diff --git a/scripts/howtogalaxy/README.rst b/scripts/howtogalaxy/README.rst deleted file mode 100644 index dad49ba8..00000000 --- a/scripts/howtogalaxy/README.rst +++ /dev/null @@ -1,125 +0,0 @@ -The ``HowToGalaxy`` folder contains **HowToGalaxy** lectures, which teach a new user how to model a galaxy. - -Folders -------- - -- ``chapter_1_introduction``: An introduction to galaxy morphology and structure using **PyAutoGalaxy**. -- ``chapter_2_modeling``: How to model galaxies, including a primer on Bayesian non-linear analysis. -- ``chapter_3_search_chaining``: How to fit complex models using non-linear search chaining. -- ``chapter_4_pixelizations``: How to perform pixelized reconstructions of a galaxy. -- ``chapter_optional``: Optional tutorials. - -Full Explanation ----------------- - -Welcome to **HowToGalaxy** - The **PyAutoGalaxy** tutorial! - -HOW TO TACKLE HowToGalaxy ------------------------ - -The **HowToGalaxy** lecture series current sits at 4 chapters, and each will take a day or so to go through -properly. You probably want to be modeling galaxies faster than that! Furthermore, the concepts -in the later chapters are pretty challenging, and familiarity and modeling is desirable before -you tackle them. - -Therefore, we recommend that you complete chapters 1 & 2 and then apply what you've learnt to the modeling of simulated -and real galaxy data, using the scripts found in the 'autogalaxy_workspace'. Once you're happy -with the results and confident with your use of **PyAutoGalaxy**, you can then begin to cover the advanced functionality -covered in chapters 3 & 4. - -JUYPTER NOTEBOOKS ------------------ - -All tutorials are supplied as Jupyter Notebooks, which come with a '.ipynb' suffix. For those new to Python, Jupyter -Notebooks are a different way to write, view and use Python code. Compared to the traditional Python scripts, -they allow: - -- Small blocks of code to be viewed and run at a time. -- Images and visualization from a code to be displayed directly underneath it. -- Text script to appear between the blocks of code. - -This makes them an ideal way for us to present the **HowToGalaxy** lecture series, therefore I recommend you get yourself -a Jupyter notebook viewer (https://jupyter.org/) if you have not done so already. - -If you *really* want to use Python scripts, all tutorials are supplied a ```` python files in the 'scripts' folder of -each chapter. - -For actual **PyAutoGalaxy** use, I recommend you use Python scripts. Therefore, as you go through the lecture series -you will notice that we will transition you to Python scripts in the third chapter. - -VISUALIZATION -------------- - -Before beginning the **HowToGalaxy** lecture series, in chapter 1 you should do 'tutorial_0_visualization'. This will -take you through how **PyAutoGalaxy** interfaces with matplotlib to perform visualization and will get you setup such that -images and figures display correctly in your Jupyter notebooks. - -CODE STYLE AND FORMATTING -------------------------- - -When you begin the notebooks, you may notice the style and formatting of our Python code looks different to what you -are used to. For example, it is common for brackets to be placed on their own line at the end of function calls, -the inputs of a function or class may be listed over many separate lines and the code in general takes up a lot more -space then you are used to. - -This is intentional, because we believe it makes the cleanest, most readable code possible. In fact - lots of people do, -which is why we use an auto-formatter to produce the code in a standardized format. If you're interested in the style -and would like to adapt it to your own code, check out the Python auto-code formatter 'black'. - -https://github.com/python/black - -OVERVIEW OF CHAPTER 1 (Beginner) --------------------------------- - -**Galaxy Structure with PyAutoGalaxy** - -In chapter 1, we'll learn about galaxy structure and **PyAutoGalaxy**. At the end, you'll be able to: - -1) Create uniform grid's of (x,y) Cartesian coordinates. -2) Combine these grid's with light profiles to make images. -3) Combine these light profiles to make galaxies. -4) Simulate telescope CCD imaging data of a galaxy. -5) Fit imaging data with model images generated via galaxy objects. - -OVERVIEW OF CHAPTER 2 (Beginner) --------------------------------- - -**Bayesian Inference and Non-linear Searches** - -In chapter 2, we'll cover Bayesian inference and model-fitting via a non-linear search. We will use these tools to -fit CCD imaging data of a galaxy with a model. At the end, you'll understand: - -1) The concept of a non-linear search and non-linear parameter space. -2) How to fit a model to galaxy CCD imaging via a non-linear search. -3) The trade-off between realism and complexity when choosing a model. -4) Why an incorrect model may be inferred and how to prevent this from happening. -5) The challenges that are involved in inferred a robust model in a computationally reasonable run-time. - -**Once completed, you'll be ready to model your own galaxies with PyAutoGalaxy!** - -OVERVIEW OF CHAPTER 3 (Intermediate) ------------------------------------- - -**Automated Modeling with non-linear search chaining** - -In chapter 3, we'll learn how to chain multiple non-linear searches together to build automated modeling pipelines -which can: - -1) Break-down the fitting of a model using multiple non-linear searches and prior passing. -2) Use a custom pipeline to fit data containing multiple galaxy where each galaxy is fitted one at a time. -3) Fit the global structure of a galaxy, followed by faint morphological features like a bar. - -OVERVIEW OF CHAPTER 4 (Intermediate) ------------------------------------- - -**Using an inverison to perform a pixelized morphology reconstructions** - -In chapter 4, we'll learn how to reconstruct morphology features of a galaxy using a pixel-grid, ensuring that we can -fit an accurate model to sources with complex and irregular morphologies. You'll learn how to: - -1) Pixelize a galaxy reconstruction into pixels. -2) Perform a linear inversion using this pixelization to reconstruct the galaxy's light. -3) Apply a smoothness prior on the galaxy reconstruction, called regularization. -4) Apply smoothing within a Bayesian framework to objectively quantify the reconstruction's complexity. -5) Use alternative pixelizations, for example a Voronoi mesh. -6) Use these features to fit a model via non-linear searches. \ No newline at end of file diff --git a/scripts/howtogalaxy/__init__.py b/scripts/howtogalaxy/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/scripts/howtogalaxy/chapter_1_introduction/HubbleTuningFork.jpg b/scripts/howtogalaxy/chapter_1_introduction/HubbleTuningFork.jpg deleted file mode 100644 index f0f49bd5..00000000 Binary files a/scripts/howtogalaxy/chapter_1_introduction/HubbleTuningFork.jpg and /dev/null differ diff --git a/scripts/howtogalaxy/chapter_1_introduction/README.rst b/scripts/howtogalaxy/chapter_1_introduction/README.rst deleted file mode 100644 index 2be28012..00000000 --- a/scripts/howtogalaxy/chapter_1_introduction/README.rst +++ /dev/null @@ -1,24 +0,0 @@ -In chapter 1, we introduce you to strong gravitational lensing and the core **PyAutoGalaxy** API. - -**Colab** links to every tutorial are included. - -Files ------ - -`Tutorial 0: Visualization `_ -- Setting up **PyAutoGalaxy**'s visualization library. - -`Tutorial 1: Grids And Galaxies `_ -- How grids of (y,x) coordinates are used to create images of galaxies that ultimately quantify their morphology. - -`Tutorial 2: Data `_ -- Simulating and inspecting telescope imaging data of a galaxy, for example from the Hubble Space Telescope. - -`Tutorial 3: Fitting `_ -- How to fit imaging data of a galaxy and quantify whether a fit is good or bad. - -`Tutorial 4: Methods `_ -- An overview of the different methods used to fit galaxies with. - -`Tutorial 5: Summary `_ -- A summary of the chapter. \ No newline at end of file diff --git a/scripts/howtogalaxy/chapter_1_introduction/__init__.py b/scripts/howtogalaxy/chapter_1_introduction/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/scripts/howtogalaxy/chapter_1_introduction/tutorial_0_visualization.py b/scripts/howtogalaxy/chapter_1_introduction/tutorial_0_visualization.py deleted file mode 100644 index 5bc0eefc..00000000 --- a/scripts/howtogalaxy/chapter_1_introduction/tutorial_0_visualization.py +++ /dev/null @@ -1,150 +0,0 @@ -""" -Tutorial 0: Visualization -========================= - -In this tutorial, we quickly cover visualization in **PyAutoGalaxy** and make sure images display clearly in your -Jupyter notebook and on your computer screen. - -__Contents__ - -**Directories:** Set the working directory so PyAutoGalaxy can find configs, data and output folders. -**Dataset:** Load an example imaging dataset of a galaxy. -**Plot Customization:** Customize matplotlib options like title, figure size and colormap. -**Subplots:** Plot all components of a dataset simultaneously using subplots. -**Visuals:** Add visual overlays like masks and grids to figures. -**Wrap Up:** Summary of visualization in PyAutoGalaxy. -""" - -# from autoconf import setup_notebook; setup_notebook() - -""" -If the printed working directory does not match the workspace path on your computer, you can manually set it -as follows (the example below shows the path I would use on my laptop. The code is commented out so you do not -use this path in this tutorial! -""" -# workspace_path = "/Users/Jammy/Code/PyAuto/autogalaxy_workspace" -# #%cd $workspace_path -# print(f"Working Directory has been set to `{workspace_path}`") - -""" -__Dataset__ - -The `dataset_path` specifies where the dataset is located, which is the -directory `autogalaxy_workspace/dataset/imaging/simple__sersic`. - -There are many example simulated images of galaxies in this directory that will be used throughout the -**HowToGalaxy** lectures. -""" -dataset_path = Path("dataset", "imaging", "simple__sersic") - -""" -__Dataset Auto-Simulation__ - -If the dataset does not already exist on your system, it will be created by running the corresponding -simulator script. This ensures that all example scripts can be run without manually simulating data first. -""" -if not dataset_path.exists(): - import subprocess - import sys - - subprocess.run( - [sys.executable, "scripts/howtogalaxy/simulators/sersic.py"], - check=True, - ) - - -""" -We now load this dataset from .fits files and create an instance of an `imaging` object. -""" -dataset = ag.Imaging.from_fits( - data_path=dataset_path / "data.fits", - noise_map_path=dataset_path / "noise_map.fits", - psf_path=dataset_path / "psf.fits", - pixel_scales=0.1, -) - -""" -We can plot the data as follows: -""" -aplt.plot_array(array=dataset.data, title="Data") - -""" -__Plot Customization__ - -Does the figure display correctly on your computer screen? - -If not, you can customize common matplotlib options by passing them directly to `plot_array`: - - - `title=`: Set the figure title. - - `figsize=`: Control the figure size as a `(width, height)` tuple. - - `colormap=`: Set the matplotlib colormap name (e.g. `"jet"`, `"gray"`). - - `xlabel=`, `ylabel=`: Override the default axis labels. -""" -aplt.plot_array( - array=dataset.data, - title="Data", -) - -""" -Many matplotlib options can be customized, but for now we're only concerned with making sure figures display clear in -your Jupyter Notebooks. Nevertheless, a comprehensive API reference guide of all available plot arguments can -be found in the `autogalaxy_workspace/*/guides/plot` package. You should check this out once you are more familiar with -**PyAutoGalaxy**. - -Ideally, we would not specify a `figsize` every time we plot an image. Fortunately, default values can be fully -customized via the config files. - -Checkout the `mat_wrap.yaml` file in `autogalaxy_workspace/config/visualize/mat_wrap`. - -All default matplotlib values are here. There are a lot of entries, so lets focus on whats important for displaying -figures: - - - mat_wrap.yaml -> Figure -> figure: -> figsize - - mat_wrap.yaml -> YLabel -> figure: -> fontsize - - mat_wrap.yaml -> XLabel -> figure: -> fontsize - - mat_wrap.yaml -> TickParams -> figure: -> labelsize - - mat_wrap.yaml -> YTicks -> figure: -> labelsize - - mat_wrap.yaml -> XTicks -> figure: -> labelsize - -Don't worry about all the other files or options listed for now, as they'll make a lot more sense once you are familiar -with **PyAutoGalaxy**. - -If you had to change any of the above settings to get the figures to display clearly, you should update their values -in the corresponding config files above (you will need to reset your Jupyter notebook server for these changes to -take effect, so make sure you have the right values using the `figsize` argument in the cell above beforehand!). - -__Subplots__ - -In addition to plotting individual figures, **PyAutoGalaxy** can also plot subplots showing all components of a -dataset simultaneously. - -Lets plot a subplot of our `Imaging` data: -""" -aplt.subplot_imaging_dataset(dataset=dataset) - -""" -__Visuals__ - -Visuals can be added to any figure by passing them as keyword arguments directly to `plot_array`. - -For example, we can plot a mask on the image above by passing `mask=mask`. - -The `visuals` example illustrates every overlay argument, for example `mask=`, `grid=`, `positions=`, `lines=`, etc. -""" -mask = ag.Mask2D.circular_annular( - shape_native=dataset.shape_native, - pixel_scales=dataset.pixel_scales, - inner_radius=0.3, - outer_radius=3.0, -) - -aplt.plot_array(array=dataset.data, title="Data") - -""" -__Wrap Up__ - -Throughout lectures you'll see lots more visuals that are plotted on figures and subplots. - -Great! Hopefully, visualization in **PyAutoGalaxy** is displaying nicely for us to get on with the **HowToGalaxy** -lecture series. -""" diff --git a/scripts/howtogalaxy/chapter_1_introduction/tutorial_1_grids_and_galaxies.py b/scripts/howtogalaxy/chapter_1_introduction/tutorial_1_grids_and_galaxies.py deleted file mode 100644 index 6df1fc60..00000000 --- a/scripts/howtogalaxy/chapter_1_introduction/tutorial_1_grids_and_galaxies.py +++ /dev/null @@ -1,654 +0,0 @@ -""" -HowToGalaxy: Introduction -========================= - -Nearly a century ago, Edwin Hubble famously classified galaxies into three distinct groups: -ellipticals, spirals and irregulars. He produced a diagram of these galaxies, called the Hubble Tuning Fork, which -is shown below and still discussed by astronomers in the modern day: - -![HubbleTuning](https://github.com/Jammy2211/autogalaxy_workspace/blob/main/scripts/howtogalaxy/chapter_1_introduction/HubbleTuningFork.jpg) - -To make his diagram, Hubble looked at images of each galaxy in his sample, and subjectively judged by eye how -to classify it. Today, Astronomers use computer software, statistical algorithms and image processing techniques to -perform this task in a more quantifiable and objective way. - -The **HowToGalaxy** series of tutorials will teach you how to perform this analysis yourself, using the open-source -software package **PyAutoGalaxy**. By the end of the **HowToGalaxy** series, you'll be able to take an image of a -galaxy and study its morphology and structure using the same techniques that professional astronomers use today. - -Tutorial 1: Grids And Galaxies -============================== - -In this tutorial, we will introduce the fundamental concepts and quantities used to study galaxy morphology. -These concepts will enable us to create images of galaxies and analyze how their light is distributed across space. -Additionally, we will explore how adjusting various properties of galaxies can alter their appearance. For instance, -we can change the size of a galaxy, rotate it, or modify its brightness. - -To create these images, we first need to define 2D grids of \((y, x)\) coordinates. We will shift and rotate these -grids to manipulate the appearance of the galaxy in the generated images. The grid will serve as the input for light -profiles, which are analytic functions that describe the distribution of a galaxy's light. By evaluating these light -profiles on the grid, we can effectively generate images that represent the structure and characteristics of galaxies. - -__Contents__ - -**Grids:** Create a uniform grid of (y,x) coordinates and show how it can be used to measure the light of a galaxy. -**Geometry:** How to shift and rotate a grid, and convert it to elliptical coordinates. -**Light Profiles:** Using light profiles, analytic functions that describe how a galaxy's light is distributed. -**Galaxies:** Creating galaxies containing light profiles and computing the image of a galaxy. -**One Dimension Projection:** Create projected 2D radial grids for 1D profile calculations. -**Unit Conversion:** Converting angular distances to physical distances using cosmology. -**Wrap Up:** Summary of the key concepts covered in this tutorial. - -The imports below are required to run the howtogalaxy tutorials in a Jupiter notebook. They also import the -`autogalaxy` package and the `autogalaxy.plot` module which are used throughout the tutorials. -""" - -# from autoconf import setup_notebook; setup_notebook() - -import matplotlib.pyplot as plt -import numpy as np - -import autogalaxy as ag -import autogalaxy.plot as aplt - -""" -__Grids__ - -A `Grid2D` is a set of two-dimensional $(y,x)$ coordinates that represent points in space where we evaluate the -light emitted by a galaxy. - -Each coordinate on the grid is referred to as a 'pixel'. This is because we use the grid to create the image of a -galaxy at each of these coordinates, meaning that each coordinate maps to the centre of each pixel in this image. - -Grids are defined in units of 'arc-seconds' ("). An arc-second is a unit of angular measurement used by astronomers to -describe the apparent size of objects in the sky. - -The `pixel_scales` parameter sets how many arc-seconds each pixel represents. For example, if `pixel_scales=0.1`, -then each pixel covers 0.1" of the sky. - -We create a uniform 2D grid of 101 x 101 pixels with a pixel scale of 0.1", corresponding to an area -of 10.1" x 10.1", spanning from -5.05" to 5.05" in both the y and x directions. -""" -grid = ag.Grid2D.uniform( - shape_native=( - 101, - 101, - ), # The dimensions of the grid, which here is 101 x 101 pixels. - pixel_scales=0.1, # The conversion factor between pixel units and arc-seconds. -) - -""" -We can visualize this grid as a uniform grid of dots, each representing a coordinate where the light is measured. -""" -aplt.plot_grid(grid=grid, title="Uniform Grid of Coordinates") - -""" -Each coordinate in the grid corresponds to an arc-second position. Below, we print a few of these coordinates to see -the values. -""" -print("(y,x) pixel 0:") -print(grid.native[0, 0]) # The coordinate of the first pixel. -print("(y,x) pixel 1:") -print(grid.native[0, 1]) # The coordinate of the second pixel. -print("(y,x) pixel 2:") -print(grid.native[0, 2]) # The coordinate of the third pixel. -print("(y,x) pixel 100:") -print(grid.native[1, 0]) # The coordinate of the 100th pixel. -print("...") - -""" -Grids have two internal representations, `native` and `slim`: - -- `native`: A 2D array with shape [total_y_pixels, total_x_pixels, 2], where the 2 corresponds to the (y,x) coordinates. -- `slim`: A 1D array with shape [total_y_pixels * total_x_pixels, 2], where the coordinates are 'flattened' into a single list. - -These formats are useful for different calculations and plotting. Here, we show the same coordinate using both formats. -""" -print("(y,x) pixel 0 (accessed via native):") -print(grid.native[0, 0]) -print("(y,x) pixel 0 (accessed via slim 1D):") -print(grid.slim[0]) - -""" -We can also check the shapes of the `Grid2D` object in both `native` and `slim` formats. For this grid, -the `native` shape is (101, 101, 2) and the `slim` shape is (10201, 2). -""" -print(grid.native.shape) -print(grid.slim.shape) - -""" -For the HowToGalaxy tutorials, you don't need to fully understand why grids have both native and slim representations. -Just note that both are used for calculations and plotting. - -*Exercise*: Try creating grids with different `shape_native` and `pixel_scales` using the `ag.Grid2D.uniform()` function -above. Observe how the grid coordinates change when you adjust `shape_native` and `pixel_scales`. - -__Geometry__ - -The above grid is centered on the origin (0.0", 0.0"). Sometimes, we need to shift the grid to be centered on a -specific point, like the center of a galaxy. - -We can shift the grid to a new center, (y_c, x_c), by subtracting this center from each coordinate. -""" -centre = (0.3, 0.5) # Shifting the grid to be centered at y=1.0", x=2.0". - -grid_shifted = grid -grid_shifted[:, 0] = grid_shifted[:, 0] - centre[0] # Shift in y-direction. -grid_shifted[:, 1] = grid_shifted[:, 1] - centre[1] # Shift in x-direction. - -print("(y,x) pixel 0 After Shift:") -print(grid_shifted.native[0, 0]) # The coordinate of the first pixel after shifting. - -""" -The grid is now centered around (0.3", 0.5"). We can plot the shifted grid to see this change. - -*Exercise*: Try shifting the grid to a different center, for example (0.0", 0.0") or (2.0", 3.0"). Observe how the -center of the grid changes when you adjust the `centre` variable. -""" -aplt.plot_grid(grid=grid_shifted, title="Grid Centered Around (0.3, 0.5)") - -""" -Next, we can rotate the grid by an angle `phi` (in degrees). The rotation is counter-clockwise from the positive x-axis. - -To rotate the grid: - -1. Calculate the distance `radius` of each coordinate from the origin using $r = \sqrt{y^2 + x^2}$. -2. Determine the angle `theta` counter clockwise from the positive x-axis using $\theta = \arctan(y / x)$. -3. Adjust `theta` by the rotation angle and convert back to Cartesian coordinates via $y = r \sin(\theta)$ and $x = r \cos(\theta)$. -""" -angle_degrees = 60.0 - -y = grid_shifted[:, 0] -x = grid_shifted[:, 1] - -radius = np.sqrt(y**2 + x**2) -theta = np.arctan2(y, x) - np.radians(angle_degrees) - -grid_rotated = grid_shifted -grid_rotated[:, 0] = radius * np.sin(theta) -grid_rotated[:, 1] = radius * np.cos(theta) - -print("(y,x) pixel 0 After Rotation:") -print(grid_rotated.native[0, 0]) # The coordinate of the first pixel after rotation. - -""" -The grid has now been rotated 60 degrees counter-clockwise. We can plot it to see the change. - -*Exercise*: Try rotating the grid by a different angle, for example 30 degrees or 90 degrees. Observe how the grid -changes when you adjust the `angle_degrees` variable. -""" -aplt.plot_grid(grid=grid_rotated, title="Grid Rotated 60 Degrees") - -""" -Next, we convert the rotated grid to elliptical coordinates using: - -$\eta = \sqrt{(x_r)^2 + (y_r)^2/q^2}$ - -Where `q` is the axis-ratio of the ellipse and `(x_r, y_r)` are the rotated coordinates. - -Elliptical coordinates are a system used to describe positions in relation to an ellipse rather than a circle. They -are particularly useful in astronomy when dealing with objects like galaxies, which often have elliptical shapes -due to their inclination or intrinsic shape. - -*Exercise*: Try converting the grid to elliptical coordinates using a different axis-ratio, for example 0.3 or 0.8. -What happens to the grid when you adjust the `axis_ratio` variable? -""" -axis_ratio = 0.5 -eta = np.sqrt((grid_rotated[:, 0]) ** 2 + (grid_rotated[:, 1]) ** 2 / axis_ratio**2) - -print("First Ten Elliptical Coordinates:") -print(eta[:10]) - -""" -Above, the angle $\phi$ (in degrees) was used to rotate the grid, and the axis-ratio $q$ was used to convert the grid -to elliptical coordinates. - -From now on, we'll describe ellipticity using "elliptical components" $\epsilon_{1}$ and $\epsilon_{2}$, calculated -from $\phi$ and $q$: - -$\epsilon_{1} = \frac{1 - q}{1 + q} \sin(2\phi)$ -$\epsilon_{2} = \frac{1 - q}{1 + q} \cos(2\phi)$ - -We'll refer to these as `ell_comps` in the code for brevity. - -Future tutorials will explain why $\epsilon_{1}$ and $\epsilon_{2}$ are preferred over $q$ and $\phi$. - -*Exercise*: Try computing the elliptical components from the axis-ratio and angle above. What happens to the elliptical -components when you adjust the `axis_ratio` and `angle_degrees` variables? -""" -fac = (1 - axis_ratio) / (1 + axis_ratio) -epsilon_y = fac * np.sin(2 * np.radians(angle_degrees)) -epsilon_x = fac * np.cos(2 * np.radians(angle_degrees)) - -ell_comps = (epsilon_y, epsilon_x) - -print("Elliptical Components:") -print(ell_comps) - - -""" -__Light Profiles__ - -Galaxies are collections of stars, gas, dust, and other astronomical objects that emit light. Astronomers study this -light to understand various properties of galaxies. - -To model the light of a galaxy, we use light profiles, which are mathematical functions that describe how a galaxy's -light is distributed across space. By applying these light profiles to 2D grids of $(y, x)$ coordinates, we can -create images that represent a galaxy's luminous emission. - -A commonly used light profile is the `Sersic` profile, which is widely adopted in astronomy for representing galaxy -light. The `Sersic` profile is defined by the equation: - -$I_{\rm Ser} (\eta_{\rm l}) = I \exp \left\{ -k \left[ \left( \frac{\eta}{R} \right)^{\frac{1}{n}} - 1 \right] \right\}$ - -In this equation: - - - $\eta$ represents the elliptical coordinates of the profile in arc-seconds (refer to earlier sections for elliptical coordinates). - - $I$ is the intensity normalization of the profile, given in arbitrary units, which controls the overall brightness of the Sersic profile. - - $R$ is the effective radius in arc-seconds, which determines the size of the profile. - - $n$ is the Sersic index, which defines how 'steep' the profile is, influencing the concentration of light. - - $k$ is a constant that ensures half the light of the profile lies within the radius $R$, where $k = 2n - \frac{1}{3}$. - -We can evaluate this function using values for $(\eta, I, R, n)$ to calculate the intensity of the profile at -a particular elliptical coordinate. -""" -elliptical_coordinate = ( - 0.5 # The elliptical coordinate where we compute the intensity, in arc-seconds. -) -intensity = 1.0 # Intensity normalization of the profile in arbitrary units. -effective_radius = 2.0 # Effective radius of the profile in arc-seconds. -sersic_index = 1.0 # Sersic index of the profile. -k = 2 * sersic_index - ( - 1.0 / 3.0 -) # Calculating the constant k, note that this is an approximation. - -# Calculate the intensity of the Sersic profile at a specific elliptical coordinate. -sersic_value = np.exp( - -k * ((elliptical_coordinate / effective_radius) ** (1.0 / sersic_index) - 1.0) -) - -print("Intensity of Sersic Light Profile at Elliptical Coordinate 0.5:") -print(sersic_value) - -""" -The calculation above gives the intensity of the Sersic profile at an elliptical coordinate of 0.5. - -To create a complete image of the Sersic profile, we can evaluate the intensity at every point in our grid of -elliptical coordinates. -""" -sersic_image = np.exp(-k * ((eta / effective_radius) ** (1.0 / sersic_index) - 1.0)) - -""" -When we plot the resulting image, we can see how the properties of the grid affect its appearance: - - - The peak intensity is at the position (0.3", 0.5"), where we shifted the grid. - - The image is elongated along a 60° counter-clockwise angle, corresponding to the rotation of the grid. - - The image has an elliptical shape, consistent with the axis ratio of 0.5. - -This demonstrates how the geometry of the grid directly influences the appearance of the light profile. - -*Exercise*: Try changing the values of `centre`, `ell_comps`, `effective_radius`, and `sersic_index` above. -Observe how these adjustments change the Sersic profile image. -""" -aplt.plot_array( - array=ag.Array2D( - values=sersic_image, mask=grid.mask - ), # The `Array2D` object is discussed below. - title="Sersic Image", -) - -""" -Instead of manually handling these transformations, we can use `LightProfile` objects from the `light_profile` -module (`lp`) for faster and more efficient calculations. - -Below, we define a `Sersic` light profile using the `Sersic` object. We can print the profile to display its parameters. -""" -sersic_light_profile = ag.lp.Sersic( - centre=(0.0, 0.0), - ell_comps=(0.0, 0.1), - intensity=1.0, - effective_radius=2.0, - sersic_index=1.0, -) - -print(sersic_light_profile) - -""" -With this `Sersic` light profile, we can create an image by passing a grid to its `image_2d_from` method. - -The calculation will internally handle all the coordinate transformations and intensity evaluations we performed -manually earlier, making it much simpler. - -The `Sersic` profile we created just above is different from the one we used to manually compute the image, -so the image will look different. However, the process is the same. -""" -image = sersic_light_profile.image_2d_from(grid=grid) - -aplt.plot_array(array=image, title="Sersic Image via Light Profile") - -""" -The `image` is returned as an `Array2D` object. Similar to a `Grid2D`, it has two forms: - - - `native`: A 2D array with shape [total_y_image_pixels, total_x_image_pixels]. - - `slim`: A 1D array that flattens this data into shape [total_y_image_pixels * total_x_image_pixels]. - -The `native` form is often used for visualizations, while the `slim` form can be useful for certain calculations. -""" -print("Intensity of pixel 0:") -print(image.native[0, 0]) -print("Intensity of pixel 1:") -print(image.slim[1]) - -""" -We can visualize the light profile's image. - -We provide it with the light profile and the grid, which are used to create and plot the image. -""" -aplt.plot_array( - array=sersic_light_profile.image_2d_from(grid=grid), title="Image via LightProfile" -) - -""" -__One Dimension Projection__ - -We often want to calculative 1D quantities of a light profile, for example to plot how its light changes as -a function of radius. - -To do this, we must still input a 2D grid into the `image_2d_from` method, therefore we create a project 2D -radial grid as follows which has shape [Number_of_1d_coordinates, 2] and where all [:,0] entries are the same. - -A simple example of such a grid is as follows with 4 1D coordinates is: -""" -grid_2d_projected = ag.Grid2DIrregular( - [ - [1.000000e-06, 1.000000e-06], - [1.000000e-06, 1.000001e00], - [1.000000e-06, 2.000001e00], - [1.000000e-06, 3.000001e00], - ] -) - -""" -As in this example, we often already have a 2D grid we are using to calculate images of a light profile -and it would be convenient to simply create `grid_2d_projected` from that. - -For example, we may want the project grid which traces it major axis in uniform radial steps. - -This is easily computed using the `grid_2d_radial_project_from` function and passing the `centre` and `angle` -of a light profile we can make it align with the light profile itself. - -Note how in this example the two galaxy bulges are not rotationally aligned but we aligned the projected -grid with the first galaxy. The centres are aligned, but if they were not that would cause similar -issues. -""" -grid_2d_projected = grid.grid_2d_radial_projected_from( - centre=sersic_light_profile.centre, angle=sersic_light_profile.angle() -) - -image_1d = sersic_light_profile.image_2d_from(grid=grid_2d_projected) - -""" -We can now plot the 1D radial profile of the light profile. This profile shows how the intensity of the light -changes as a function of distance from the profile's center. This is a more informative way to visualize the light p -rofile's distribution. - -When we plot 1D quantities, we do not use built-in plotting functions as in 2D, but instead use standard -matplotlib functionality. - -The reason is partly that 1D plotting is simple, but also because 1D plots have many different decisions -about what is plotted and how they are computed, meaning its better to give the user full control. - -**Exercise**: Try plotting the 1D radial profile of Sersic profiles with different effective radii and Sersic indices. -Does the 1D representation show more clearly how the light distribution changes with these parameters? -""" -plt.plot(grid_2d_projected[:, 1], image_1d) -plt.xlabel("Radius (arcseconds)") -plt.ylabel("Luminosity") -plt.show() -plt.close() - -""" -Since galaxy light distributions often cover a wide range of values, they are typically better visualized on a log10 -scale. This approach helps highlight details in the faint outskirts of a light profile. - -The `MatPlot2D` object has a `use_log10` option that applies this transformation automatically. Below, you can see -that the image plotted in log10 space reveals more details. -""" -aplt.plot_array( - array=sersic_light_profile.image_2d_from(grid=grid), - title="Sersic Image", - use_log10=True, -) - -""" -__Galaxies__ - -Now, let's introduce `Galaxy` objects, which are key components in **PyAutoGalaxy**. - -A light profile represents a single feature of a galaxy, such as its bulge or disk. To model a complete galaxy, -we combine multiple `LightProfiles` into a `Galaxy` object. This allows us to create images that include different -components of a galaxy. - -In addition to light profiles, a `Galaxy` has a `redshift`, which indicates how far away it is from Earth. The redshift -is essential for performing unit conversions using cosmological calculations, such as converting arc-seconds into -kiloparsecs. (A kiloparsec is a distance unit in astronomy, equal to about 3.26 million light-years.) - -Let's start by creating a galaxy with two `Sersic` light profiles, which notationally we will consider to represent -a bulge and disk component of the galaxy, the two most important structures seen in galaxies which drive the -Hubble tuning fork classification shown at the beginning of this tutorial. -""" -bulge = ag.lp.Sersic( - centre=(0.0, 0.0), - ell_comps=(0.0, 0.111111), - intensity=1.0, - effective_radius=1.0, - sersic_index=2.5, -) - -disk = ag.lp.Sersic( - centre=(0.0, 0.0), - ell_comps=(0.0, 0.3), - intensity=0.3, - effective_radius=3.0, - sersic_index=1.0, -) - -galaxy = ag.Galaxy(redshift=0.5, bulge=bulge, disk=disk) - -print(galaxy) - -""" -We can pass a 2D grid to a light profile to compute its image using the `image_2d_from` method. - -The same approach works for a `Galaxy` object: -""" -image = galaxy.image_2d_from(grid=grid) - -print("Intensity of `Grid2D` pixel 0:") -print(image.native[0, 0]) -print("Intensity of `Grid2D` pixel 1:") -print(image.native[0, 1]) -print("Intensity of `Grid2D` pixel 2:") -print(image.native[0, 2]) -print("...") - -aplt.plot_array(array=image, title="Bulge+Disk Image via Galaxy") - -""" -We can plot the galaxy's image, just like how we did for a light profile. -""" -aplt.plot_array(array=galaxy.image_2d_from(grid=grid), title="Galaxy Bulge+Disk Image") - -""" -The bulge dominates the center of the image, and is pretty much the only luminous emission we see can see on a linear -scale. The disk's emission is present, but it is much fainter and spread over a larger area. - -We can confirm this using the `subplot_of_light_profiles` method, which plots each individual light profile separately. -""" -aplt.subplot_galaxy_light_profiles(galaxy=galaxy, grid=grid) - -""" -Because galaxy light distributions often follow a log10 pattern, plotting in log10 space helps reveal details in the -outskirts of the light profile, in this case the emission of the disk. - -This is especially helpful to separate the bulge and disk profiles, which have different intensities and sizes. -""" -aplt.plot_array( - array=galaxy.image_2d_from(grid=grid), - title="Galaxy Bulge+Disk Image", - use_log10=True, -) - -""" -Using the tools above, we can visualize each light profile's contribution in 1D. - -1D plots show the intensity of the light profile as a function of distance from the profile’s center. The bulge -and disk profiles in this example share the same `centre`, meaning that plotting them together on the same 1D plot -shows how they vary relative to one another. - -If the `centre` of the profiles were different, when you make the 1D plot you would need to decide if you should the -profiles offset from one another or plot them both from zero. -""" -grid_2d_projected = grid.grid_2d_radial_projected_from( - centre=galaxy.bulge.centre, angle=galaxy.bulge.angle() -) -bulge_image_1d = galaxy.bulge.image_2d_from(grid=grid_2d_projected) - -grid_2d_projected = grid.grid_2d_radial_projected_from( - centre=galaxy.disk.centre, angle=galaxy.disk.angle() -) -disk_image_1d = galaxy.disk.image_2d_from(grid=grid_2d_projected) - -plt.plot(grid_2d_projected[:, 1], bulge_image_1d, label="Bulge") -plt.plot(grid_2d_projected[:, 1], disk_image_1d, label="Disk") -plt.xlabel("Radius (arcseconds)") -plt.ylabel("Luminosity") -plt.legend() -plt.show() -plt.close() - -""" -We can group multiple galaxies at the same redshift into a `Galaxies` object, which is created from a list of -individual galaxies. - -We create an additional galaxy and combine it with the original galaxy into a `Galaxies` object. This could -represent two galaxies merging or interacting with each other, which is commonly seen in studies of galaxy evolution. -""" -extra_galaxy = ag.Galaxy( - redshift=0.5, - bulge=ag.lp.Sersic( - centre=(0.2, 0.3), - ell_comps=(0.0, 0.111111), - intensity=1.0, - effective_radius=1.0, - sersic_index=2.5, - ), -) - -galaxies = ag.Galaxies(galaxies=[galaxy, extra_galaxy]) - -""" -The `Galaxies` object has similar methods as those for light profiles and individual galaxies. - -For example, `image_2d_from` sums the images of all the galaxies. -""" -image = galaxies.image_2d_from(grid=grid) - -""" -We can plot the combined image using a `Galaxies`, just like with other plotters. -""" -aplt.plot_array(array=galaxies.image_2d_from(grid=grid), title="Image") - -""" -A subplot of each individual galaxy image can also be created. -""" -aplt.subplot_galaxies(galaxies=galaxies, grid=grid) - -""" -Because galaxy light distributions often follow a log10 pattern, plotting in log10 space helps reveal details in the -outskirts of the light profile. - -This is especially helpful when visualizing how multiple galaxies overlap. -""" -aplt.plot_array(array=galaxies.image_2d_from(grid=grid), title="Image", use_log10=True) - -""" -__Unit Conversion__ - -Earlier, we mentioned that a galaxy’s `redshift` allows us to convert between arcseconds and kiloparsecs. - -A redshift measures how much a galaxy's light is stretched by the Universe's expansion. A higher redshift means the -galaxy is further away, and its light has been stretched more. By knowing a galaxy’s redshift, we can convert angular -distances (like arcseconds) to physical distances (like kiloparsecs). - -To perform this conversion, we use a cosmological model that describes the Universe's expansion. Below, we use -the `Planck15` cosmology, which is based on observations from the Planck satellite. -""" -cosmology = ag.cosmo.Planck15() - -kpc_per_arcsec = cosmology.kpc_per_arcsec_from(redshift=galaxy.redshift) - -print("Kiloparsecs per Arcsecond:") -print(kpc_per_arcsec) - - -""" -This `kpc_per_arcsec` can be used as a conversion factor between arcseconds and kiloparsecs when plotting images of -galaxies. - -We compute this value and plot the image in converted units of kiloparsecs. - -This passes the plotting modules `Units` object a `ticks_convert_factor` and manually specified the new units of the -plot ticks. -""" -aplt.plot_array(array=galaxy.image_2d_from(grid=grid), title="Image") - -""" -__Wrap Up__ - -You've learnt the basic quantities used to study galaxy morphology. - -Lets summarise what we've learnt: - -- **Grids**: We've learnt that a grid is a set of 2D coordinates that represent the positions where we measure the -light of a galaxy. - -- **Geometry**: We've learnt how to shift, rotate, and convert grids to elliptical coordinates. - -- **Light Profiles**: We've learnt that light profiles are mathematical functions that describe how a galaxy's light is -distributed in space. We've used the `Sersic` profile to create images of galaxies. - -- **Galaxies**: We've learnt that galaxies are collections of light profiles that represent a galaxy's light. We've -created galaxies with multiple light profiles and visualized their images. - -- **Unit Conversion**: We've learnt that by assuming redshifts for galaxies we can convert their quantities from -arcseconds to kiloparsecs. - -__Advanced Topics__ - -The following advanced topics are not important for a new user learning the software for the first time. However, -once you are an expert user, the following guides and concepts are important for doing accurate galaxy morphology -analysis, and thus may be things you want to commit to memory as future references. - -__Other Unit Conversion__ - -Above, we used a redshift to convert between arcseconds and kiloparsecs. This is just one example of a unit conversion -that can be performed using a galaxy's redshift. - -There are many other unit conversions that can be performed, such as converting the units of a galaxy's image from -to what Astronomers call an AB magnitude system, which is a system used to measure the brightness of galaxies. - -The `autogalaxy_workspace/*/guides/units` module contains many examples of unit conversions and how to use them, -but they will not be covered in the *HowToGalaxy* tutorials. - -__Over Sampling__ - -Over sampling is a numerical technique where the images of light profiles and galaxies are evaluated -on a higher resolution grid than the image data to ensure the calculation is accurate. - -For a new user, the details of over-sampling are not important, therefore just be aware that all calculations use an -adaptive over sampling scheme which high accuracy across all use cases. - -Once you are more experienced, you should read up on over-sampling in more detail via -the `autogalaxy_workspace/*/guides/over_sampling.ipynb` notebook. -""" diff --git a/scripts/howtogalaxy/chapter_1_introduction/tutorial_2_data.py b/scripts/howtogalaxy/chapter_1_introduction/tutorial_2_data.py deleted file mode 100644 index f691eb68..00000000 --- a/scripts/howtogalaxy/chapter_1_introduction/tutorial_2_data.py +++ /dev/null @@ -1,374 +0,0 @@ -""" -Tutorial 2: Data -================ - -In the previous tutorial, we used light profiles to create images of galaxies. However, those images don't accurately -represent what we would observe through a telescope. - -Real telescope images, like those taken with the Charge Coupled Device (CCD) imaging detectors on the Hubble Space -Telescope (HST), include several factors that affect what we see: - -**Telescope Optics:** The optical components of the telescope can blur the light, influencing the image's sharpness. - -**Exposure Time:** The time the detector collects light, affecting the clarity of the image. Longer exposure times -gather more light, improving the signal-to-noise ratio and creating a clearer image. - -**Background Sky:** Light from a background sky, such as distant stars or zodiacal light, adds noise to the image. - -In this tutorial, we'll simulate a galaxy image by applying these real-world effects to the light profiles and images -we created earlier. - -__Contents__ - -**Initial Setup:** Create a 2D grid and define galaxy light profiles for simulation. -**Optics Blurring:** Simulate how the telescope optics blur the galaxy's light using PSF convolution. -**Poisson Noise:** Add Poisson noise to the image, simulating CCD photon-to-electron randomness. -**Background Sky:** Add background sky light that introduces noise across the entire image. -**Simulator:** Use the SimulatorImaging object to simulate imaging data with all effects combined. -**Output:** Save the simulated data to .fits files for use in future tutorials. -**Wrap Up:** Summary of how CCD imaging data is simulated. -""" - -# from autoconf import setup_notebook; setup_notebook() - -import numpy as np -from pathlib import Path -import autogalaxy as ag -import autogalaxy.plot as aplt - -""" -__Initial Setup__ - -To create our simulated galaxy image, we first need a 2D grid. This grid will represent the coordinate space over -which we will simulate the galaxy's light distribution. -""" -grid = ag.Grid2D.uniform( - shape_native=( - 101, - 101, - ), # The dimensions of the grid, which here is 101 x 101 pixels. - pixel_scales=0.1, # The conversion factor between pixel units and arc-seconds. -) - -""" -Next, we define the properties of our galaxy. In this tutorial, we’ll represent the galaxy with a bulge using a -Sersic light profile. - -In the previous tutorial, the units of `intensity` were arbitrary. However, for this tutorial, where we simulate -realistic imaging data, the intensity must have specific units. We’ll use units of electrons per second per pixel -($e- pix^-1 s^-1$), which is standard for CCD imaging data. -""" -galaxy = ag.Galaxy( - redshift=0.5, - bulge=ag.lp.Sersic( - centre=(0.0, 0.0), - ell_comps=(0.0, 0.111111), - intensity=1.0, # in units of e- pix^-1 s^-1 - effective_radius=1.0, - sersic_index=2.5, - ), -) - -galaxies = ag.Galaxies(galaxies=[galaxy]) - -""" -To visualize the galaxy’s image, which we will use as the starting point for the simulations, we use the following code: -""" -aplt.plot_array( - array=galaxies.image_2d_from(grid=grid), title="Galaxy Image Before Simulating" -) - -""" -__Optics Blurring__ - -All images captured using CCDs (like those on the Hubble Space Telescope) experience some level of blurring -due to the optics of the telescope. This blurring occurs because the optical system spreads out the light from each -point source (e.g., a star or a part of a galaxy). - -The Point Spread Function (PSF) describes how the telescope blurs the image. It can be thought of as a 2D representation -of how a single point of light would appear in the image, spread out by the optics. In practice, the PSF is a 2D -convolution kernel that we apply to the image to simulate this blurring effect. -""" -psf = ag.Convolver.from_gaussian( - shape_native=(11, 11), # The size of the PSF kernel, represented as an 11x11 grid. - sigma=0.1, # Controls the width of the Gaussian PSF, which determines the level of blurring. - pixel_scales=grid.pixel_scales, # Maintains consistency with the scale of the image grid. - normalize=True, # Normalizes the PSF kernel so that its values sum to 1. -) - -""" -We can visualize the PSF to better understand how it will blur the galaxy's image. The PSF is essentially a small -image that represents the spreading out of light from a single point source. This kernel will be used to blur the -entire galaxy image when we perform the convolution. -""" -aplt.plot_array(array=psf.kernel, title="PSF 2D Kernel") - -""" -The PSF is often more informative when plotted on a log10 scale. This approach allows us to clearly observe values -in its tail, which are much smaller than the central peak yet critical for many scientific analyses. The tail -values may significantly affect the spread and detail captured in the data. -""" -aplt.plot_array(array=psf.kernel, title="PSF 2D Kernel", use_log10=True) - -""" -Next, we'll manually perform a 2D convolution of the PSF with the image of the galaxy. This convolution simulates the -blurring that occurs when the telescope optics spread out the galaxy's light. - -1. **Padding the Image**: Before convolution, we add padding (extra space with zero values) around the edges of the - image. This prevents unwanted edge effects when we perform the convolution, ensuring that the image's edges don't - become artificially altered by the process. - -2. **Convolution**: Using the `Convolver` object's `convolve` method, we apply the 2D PSF convolution to the padded - image. This step combines the PSF with the galaxy's light, simulating how the telescope spreads out the light. - -3. **Trimming the Image**: After convolution, we trim the padded areas back to their original size, obtaining a - convolved (blurred) image that matches the dimensions of the initial galaxy image. -""" -image = galaxies.image_2d_from(grid=grid) # The original unblurred image of the galaxy. -padded_image = galaxies.padded_image_2d_from( - grid=grid, - psf_shape_2d=psf.kernel.shape_native, # Adding padding based on the PSF size. -) -convolved_image = psf.convolved_image_from( - image=padded_image, blurring_image=None -) # Applying the PSF convolution. -blurred_image = convolved_image.trimmed_after_convolution_from( - kernel_shape=psf.kernel.shape_native -) # Trimming back to the original size. - -""" -We can now plot the original and the blurred images side by side. This allows us to clearly see how the PSF -convolution affects the appearance of the galaxy, making the image appear softer and less sharp. -""" -aplt.plot_array(array=image, title="Galaxy Image Before PSF") -aplt.plot_array(array=blurred_image, title="Galaxy Image After PSF") - - -""" -__Poisson Noise__ - -In addition to the blurring caused by telescope optics, we also need to consider Poisson noise when simulating imaging -data. - -When a telescope captures an image of a galaxy, photons from the galaxy are collected by the telescope's mirror and -directed onto a CCD (Charge-Coupled Device). The CCD is made up of a silicon lattice (or another material) that -converts incoming photons into electrons. These electrons are then gathered into discrete squares, which form the -pixels of the final image. - -The process of converting photons into electrons is inherently random, following a Poisson distribution. This randomness -means that the number of electrons in each pixel can vary, even if the same number of photons hits the CCD. Therefore, -the electron count per pixel becomes a Poisson random variable. For our simulation, this means that the recorded -number of photons in each pixel will differ slightly from the true number due to this randomness. - -To replicate this effect in our simulation, we can add Poisson noise to the galaxy image using NumPy’s random module, -which generates values from a Poisson distribution. - -It's important to note that the blurring caused by the telescope optics occurs before the photons reach the CCD. -Therefore, we need to add the Poisson noise after blurring the galaxy image. - -We also need to consider the units of our image data. Let’s assume that the galaxy image is measured in units of -electrons per second ($e^- s^{-1}$), which is standard for CCD imaging data. To simulate the number of electrons -actually detected in each pixel, we multiply the image by the observation’s exposure time. This conversion changes t -he units to the total number of electrons collected per pixel over the entire exposure time. - -Once the image is converted, we add Poisson noise, simulating the randomness in the photon-to-electron conversion -process. After adding the noise, we convert the image back to units of electrons per second for analysis, as -this is the preferred unit for astronomers when studying their data. -""" -exposure_time = 300.0 # Units of seconds -blurred_image_counts = ( - blurred_image * exposure_time -) # Convert to total electrons detected over the exposure time. -blurred_image_with_poisson_noise = ( - np.random.poisson(blurred_image_counts, blurred_image_counts.shape) / exposure_time -) # Add Poisson noise and convert back to electrons per second. - -""" -Here is what the blurred image with Poisson noise looks like. -""" -aplt.plot_array( - array=ag.Array2D(values=blurred_image_with_poisson_noise, mask=grid.mask), - title="Image With Poisson Noise", -) - -""" -It is challenging to see the Poisson noise directly in the image above, as it is often subtle. To make the noise more -visible, we can subtract the blurred image without Poisson noise from the one with noise. - -This subtraction yields the "Poisson noise realization" which highlights the variation in each pixel due to the Poisson -distribution of photons hitting the CCD. It represents the noise values that were added to each pixel. We call -it the realization because it is one possible outcome of the Poisson process, and the noise will be different each time -we simulate the image. -""" -poisson_noise_realization = blurred_image_with_poisson_noise - blurred_image - -aplt.plot_array( - array=ag.Array2D(values=poisson_noise_realization, mask=grid.mask), - title="Poisson Noise Realization", -) - -""" -__Background Sky__ - -The final effect we will consider when simulating imaging data is the background sky. - -In addition to light from the galaxy, the telescope also picks up light from the sky. This background sky light is -primarily due to two sources: zodiacal light, which is light scattered by interplanetary dust in the solar system, -and the unresolved emission from distant stars and galaxies. - -For our simulation, we'll assume that the background sky has a uniform brightness across the image, measured at -0.1 electrons per second per pixel. The background sky is added to the image before applying the PSF convolution -and adding Poisson noise. This is important because it means that the background contributes additional noise to the -image. - -The background sky introduces noise throughout the entire image, including areas where the galaxy is not present. -This is why CCD images often appear noisy, especially in regions far from where the galaxy signal is detected. -The sky noise can make it more challenging to observe faint details of the galaxy. - -To simulate this, we add a constant background sky to the galaxy image and then apply Poisson noise to create the -final simulated image as it would appear through a telescope. -""" -background_sky_level = 0.1 - -# Add background sky to the blurred galaxy image. -blurred_image_with_sky = blurred_image + background_sky_level -blurred_image_with_sky_counts = blurred_image_with_sky * exposure_time - -# Apply Poisson noise to the image with the background sky. -blurred_image_with_sky_poisson_noise = ( - np.random.poisson( - blurred_image_with_sky_counts, blurred_image_with_sky_counts.shape - ) - / exposure_time -) - -# Visualize the image with background sky and Poisson noise. -aplt.plot_array( - array=ag.Array2D(values=blurred_image_with_sky_poisson_noise, mask=grid.mask), - title="Image With Background Sky", -) - -# Create a noise map showing the differences between the blurred image with and without noise. -poisson_noise_realization = ( - blurred_image_with_sky_poisson_noise - blurred_image_with_sky -) - -aplt.plot_array( - array=ag.Array2D(values=poisson_noise_realization, mask=grid.mask), - title="Poisson Noise Realization", -) - -""" -__Simulator__ - -The `SimulatorImaging` object lets us create simulated imaging data while including the effects of PSF blurring, -Poisson noise, and background sky all at once: -""" -simulator = ag.SimulatorImaging( - exposure_time=300.0, - psf=psf, - background_sky_level=0.1, - add_poisson_noise_to_data=True, -) - -dataset = simulator.via_galaxies_from(galaxies=galaxies, grid=grid) - -""" -By plotting the `data` from the dataset, we can see that it matches the image we simulated earlier. It includes -the effects of PSF blurring, Poisson noise, and noise from the background sky. This image is a realistic -approximation of what a telescope like the Hubble Space Telescope would capture. -""" -aplt.plot_array(array=dataset.data, title="Simulated Imaging Data") - -""" -The dataset also includes the `psf` (Point Spread Function) used to blur the galaxy image. - -For actual telescope data, the PSF is determined during data processing and is provided along with the observations. -It's crucial for accurately deconvolving the PSF from the galaxy image, allowing us to recover the true properties -of the galaxy. We'll explore this further in the next tutorial. -""" -aplt.plot_array(array=dataset.psf.kernel, title="Simulated PSF", use_log10=True) - -""" -The dataset includes a `noise_map`, which represents the Root Mean Square (RMS) standard deviation of the noise -estimated for each pixel in the image. Higher noise values mean that the measurements in those pixels are -less certain, so those pixels are given less weight when analyzing the data. - -This `noise_map` is different from the Poisson noise arrays we plotted earlier. The Poisson noise arrays show the -actual noise added to the image due to the random nature of photon-to-electron conversion on the CCD, as calculated -using the numpy random module. These noise values are theoretical and cannot be directly measured in real telescope data. - -In contrast, the `noise_map` is our best estimate of the noise present in the image, derived from the data itself -and used in the fitting process. -""" -aplt.plot_array(array=dataset.noise_map, title="Simulated Noise Map") - -""" -The `signal-to-noise_map` shows the ratio of the signal in each pixel to the noise level in that pixel. It is -calculated by dividing the `data` by the `noise_map`. - -This ratio helps us understand how much of the observed signal is reliable compared to the noise, allowing us to -see where we can trust the detected signal from the galaxy and where the noise is more significant. - -In general, a signal-to-noise ratio greater than 3 indicates that the signal is likely real and not overwhelmed by -noise. For our datasets, the signal-to-noise ratio peaks at ~70, meaning we can trust the signal detected in the -image. -""" -aplt.plot_array(array=dataset.signal_to_noise_map, title="Signal-To-Noise Map") - -""" -The `Imaging` object can display all of these components together, making it a powerful tool for visualizing -simulated imaging data. - -It also shows the Data and PSF on a logarithmic (log10) scale, which helps highlight the faint details in these -components. - -The "Over Sampling" plots on the bottom of the figures display advanced features that can be ignored for now. -""" -aplt.subplot_imaging_dataset(dataset=dataset) - -""" -__Output__ - -We will now save these simulated data to `.fits` files, the standard format used by astronomers for storing images. -Most imaging data from telescopes like the Hubble Space Telescope (HST) are stored in this format. - -The `dataset_path` specifies where the data will be saved, in this case, in the directory -`autogalaxy_workspace/dataset/imaging/howtogalaxy/`, which contains many example images distributed with -the `autogalaxy_workspace`. - -The files are named `data.fits`, `noise_map.fits`, and `psf.fits`, and will be used in the next tutorial. -""" -dataset_path = Path("dataset", "imaging", "howtogalaxy") -print("Dataset Path: ", dataset_path) - -aplt.fits_imaging( - dataset=dataset, - data_path=dataset_path / "data.fits", - noise_map_path=dataset_path / "noise_map.fits", - psf_path=dataset_path / "psf.fits", - overwrite=True, -) - -""" -__Wrap Up__ - -In this tutorial, you learned how CCD imaging data of a galaxy is collected using real telescopes like the -Hubble Space Telescope, and how to simulate this data using the `SimulatorImaging` object. - -Let's summarise what we've covered: - -- **Optics Blurring**: The optics of a telescope blur the light from galaxies, reducing the clarity and sharpness of -the images. - -- **Poisson Noise**: The process of converting photons to electrons on a CCD introduces Poisson noise, which is random -variability in the number of electrons collected in each pixel. - -- **Background Sky**: Light from the sky is captured along with light from the galaxy, adding a layer of noise across -the entire image. - -- **Simulator**: The `SimulatorImaging` object enables us to simulate realistic imaging data by including all of -these effects together and contains the `data`, `psf`, and `noise_map` components. - -- **Output**: We saved the simulated data to `.fits` files, the standard format used by astronomers for storing images. -""" diff --git a/scripts/howtogalaxy/chapter_1_introduction/tutorial_3_fitting.py b/scripts/howtogalaxy/chapter_1_introduction/tutorial_3_fitting.py deleted file mode 100644 index 4bcc61ed..00000000 --- a/scripts/howtogalaxy/chapter_1_introduction/tutorial_3_fitting.py +++ /dev/null @@ -1,665 +0,0 @@ -""" -Tutorial 3: Fitting -=================== - -In previous tutorials, we used light profiles to create simulated images of galaxies and visualized how these images -would appear when captured by a CCD detector on a telescope like the Hubble Space Telescope. - -However, this simulation process is the reverse of what astronomers typically do when analyzing real data. Usually, -astronomers start with an observation—an actual image of a galaxy—and aim to infer detailed information about the -galaxy’s properties, such as its shape, structure, formation, and evolutionary history. - -To achieve this, we must fit the observed image data with a model, identifying the combination of light profiles that -best matches the galaxy's appearance in the image. In this tutorial, we'll illustrate this process using the imaging -data simulated in the previous tutorial. Our goal is to demonstrate how we can recover the parameters of the light -profiles that we used to create the original simulation, as a proof of concept for the fitting procedure. - -The process of fitting data introduces essential statistical concepts like the `model`, `residual_map`, `chi-squared`, -`likelihood`, and `noise_map`. These terms are crucial for understanding how fitting works, not only in astronomy but -also in any scientific field that involves data modeling. This tutorial will provide a detailed introduction to these -concepts and show how they are applied in practice to analyze astronomical data. - -__Contents__ - -**Dataset:** Load the imaging dataset previously simulated, consisting of the image, noise map, and PSF. -**Mask:** Apply a mask to the data, excluding regions with low signal-to-noise ratios from the analysis. -**Masked Grid:** Create a masked grid containing only coordinates of unmasked pixels. -**Fitting:** Fit the data with a galaxy model, computing the model image, residuals, chi-squared, and log likelihood. -**Incorrect Fit:** Demonstrate how small deviations from true parameters impact fit quality. -**Model Fitting:** Perform a basic model fit, adjusting parameters to improve the fit. -**Wrap Up:** Summary of the fitting process and key statistical concepts. -""" - -import numpy as np -from pathlib import Path -import autogalaxy as ag -import autogalaxy.plot as aplt - -""" -__Dataset__ - -We begin by loading the imaging dataset that we will use for fitting in this tutorial. This dataset is identical to the -one we simulated in the previous tutorial, representing how a galaxy would appear if captured by a CCD camera. - -In the previous tutorial, we saved this dataset as .fits files in the `autogalaxy_workspace/dataset/imaging/howtogalaxy` -folder. The `.fits` format is commonly used in astronomy for storing image data along with metadata, making it a -standard for CCD imaging. - -The `dataset_path` below specifies where these files are located: `autogalaxy_workspace/dataset/imaging/howtogalaxy/`. -""" -dataset_path = Path("dataset", "imaging", "howtogalaxy") - -""" -__Dataset Auto-Simulation__ - -If the dataset does not already exist on your system, it will be created by running the corresponding -simulator script. This ensures that all example scripts can be run without manually simulating data first. -""" -if not dataset_path.exists(): - import subprocess - import sys - - subprocess.run( - [sys.executable, "scripts/imaging/simulator.py"], - check=True, - ) - - -dataset = ag.Imaging.from_fits( - data_path=dataset_path / "data.fits", - noise_map_path=dataset_path / "noise_map.fits", - psf_path=dataset_path / "psf.fits", - pixel_scales=0.1, -) - -""" -The `Imaging` object contains three key components: `data`, `noise_map`, and `psf`: - -- `data`: The actual image of the galaxy, which we will analyze. - -- `noise_map`: A map indicating the uncertainty or noise level in each pixel of the image, reflecting how much the - observed signal in each pixel might fluctuate due to instrumental or background noise. - -- `psf`: The Point Spread Function, which describes how a point source of light is spread out in the image by the - telescope's optics. It characterizes the blurring effect introduced by the instrument. - -Let's print some values from these components and plot a summary of the dataset to refresh our understanding of the -imaging data. -""" -print("Value of first pixel in imaging data:") -print(dataset.data.native[0, 0]) -print("Value of first pixel in noise map:") -print(dataset.noise_map.native[0, 0]) -print("Value of first pixel in PSF:") -print(dataset.psf.kernel.native[0, 0]) - -aplt.subplot_imaging_dataset(dataset=dataset) - -""" -__Mask__ - -The signal-to-noise map of the image highlights areas where the signal (light from the galaxy) is detected above the -background noise. Values above 3.0 indicate regions where the galaxy's light is detected with a signal-to-noise ratio -of at least 3, while values below 3.0 are dominated by noise, where the galaxy's light is not clearly distinguishable. - -To ensure the fitting process focuses only on meaningful data, we typically mask out regions with low signal-to-noise -ratios, removing areas dominated by noise from the analysis. This allows the fitting process to concentrate on the -regions where the galaxy is clearly detected. - -Here, we create a `Mask2D` to exclude certain regions of the image from the analysis. The mask defines which parts of -the image will be used during the fitting process. - -For our simulated image, a circular 3" mask centered at the center of the image is appropriate, since the simulated -galaxy was positioned at the center. -""" -mask = ag.Mask2D.circular( - shape_native=dataset.shape_native, - pixel_scales=dataset.pixel_scales, - radius=3.0, # The circular mask's radius in arc-seconds - centre=(0.0, 0.0), # center of the image which is also the center of the galaxy -) - -print(mask) # 1 = True, meaning the pixel is masked. Edge pixels are indeed masked. -print(mask[48:53, 48:53]) # Central pixels are `False` and therefore unmasked. - -""" -We can visualize the mask over the galaxy image using an `Imaging`, which helps us adjust the mask as needed. -This is useful to ensure that the mask appropriately covers the galaxy's light and does not exclude important regions. - -To overlay objects like a mask onto a figure, we use the `Visuals2D` object. This tool allows us to add custom -visuals to any plot, providing flexibility in creating tailored visual representations. -""" -aplt.plot_array(array=dataset.data, title="Imaging Data With Mask") - -""" -Once we are satisfied with the mask, we apply it to the imaging data using the `apply_mask()` method. This ensures -that only the unmasked regions are considered during the analysis. -""" -dataset = dataset.apply_mask(mask=mask) - -""" -When we plot the masked imaging data again, the mask is now automatically included in the plot, even though we did -not explicitly pass it using the `Visuals2D` object. The plot also zooms into the unmasked area, showing only the -region where we will focus our analysis. This is particularly helpful when working with large images, as it centers -the view on the regions where the galaxy's signal is detected. -""" -aplt.plot_array(array=dataset.data, title="Masked Imaging Data") - -""" -The mask is now stored as an additional attribute of the `Imaging` object, meaning it remains attached to the -dataset. This makes it readily available when we pass the dataset to a `FitImaging` object for the fitting process. -""" -print("Mask2D:") -print(dataset.mask) - -""" -In earlier tutorials, we discussed how grids and arrays have `native` and `slim` representations: - -- `native`: Represents the original 2D shape of the data, maintaining the full pixel array of the image. -- `slim`: Represents a 1D array containing only the values from unmasked pixels, allowing for more efficient - processing when working with large images. - -After applying the mask, the `native` and `slim` representations change as follows: - -- `native`: The 2D array keeps its original shape, [total_y_pixels, total_x_pixels], but masked pixels (those where - the mask is True) are set to 0.0. -- `slim`: This now only contains the unmasked pixel values, reducing the array size - from [total_y_pixels * total_x_pixels] to just the number of unmasked pixels. - -Let's verify this by checking the shape of the data in its `slim` representation. -""" -print("Number of unmasked pixels:") -print(dataset.data.native.shape) -print( - dataset.data.slim.shape -) # This should be lower than the total number of pixels, e.g., 100 x 100 = 10,000 - -""" -The `mask` object also has a `pixels_in_mask` attribute, which gives the number of unmasked pixels. This should -match the size of the `slim` data structure. -""" -print(dataset.data.mask.pixels_in_mask) - -""" -We can use the `slim` attribute to print the first unmasked values from the image and noise map: -""" -print("First unmasked image value:") -print(dataset.data.slim[0]) -print("First unmasked noise map value:") -print(dataset.noise_map.slim[0]) - -""" -Additionally, we can verify that the `native` data structure has zeros at the edges where the mask is applied and -retains non-zero values in the central unmasked regions. -""" -print("Example masked pixel in the image's native representation at its edge:") -print(dataset.data.native[0, 0]) -print("Example unmasked pixel in the image's native representation at its center:") -centre = tuple(s // 2 for s in dataset.data.shape_native) -print(dataset.data.native[centre]) - -""" -__Masked Grid__ - -In tutorial 1, we emphasized that the `Grid2D` object is crucial for evaluating a galaxy's light profile. This grid -contains (y, x) coordinates for each pixel in the image and is used to map out the positions where the galaxy's -light is calculated. - -From a `Mask2D`, we derive a `masked_grid`, which consists only of the coordinates of unmasked pixels. This ensures -that light profile calculations focus exclusively on regions where the galaxy's light is detected, saving computational -time and improving efficiency. - -Below, we plot the masked grid: -""" -masked_grid = mask.derive_grid.unmasked - -aplt.plot_grid(grid=masked_grid, title="Masked Grid2D") - -""" -By plotting this masked grid over the galaxy image, we can see that the grid aligns with the unmasked pixels of the -image. - -This alignment **is crucial** for accurate fitting because it ensures that when we evaluate a galaxy's light profile, -the calculations occur only at positions where we have real data from. -""" -aplt.plot_array(array=dataset.data, title="Image Data With 2D Grid Overlaid") - -""" -__Fitting__ - -Now that our data is masked, we are ready to proceed with the fitting process. - -Fitting the data is done using the `Galaxy` and `Galaxies objects that we introduced in tutorial 2. We will start by -setting up a `Galaxies`` object, using the same galaxy configuration that we previously used to simulate the -imaging data. This setup will give us what is known as a 'perfect' fit, as the simulated and fitted models are identical. -""" -galaxy = ag.Galaxy( - redshift=0.5, - bulge=ag.lp.Sersic( - centre=(0.0, 0.0), - ell_comps=(0.0, 0.111111), - intensity=1.0, # in units of e- pix^-1 s^-1 - effective_radius=1.0, - sersic_index=2.5, - ), -) - -galaxies = ag.Galaxies(galaxies=[galaxy]) - -""" -Next, let's plot the image of the galaxies. This should look familiar, as it is the same image we saw in -previous tutorials. The difference now is that we use the dataset's `grid`, which corresponds to the `masked_grid` -we defined earlier. This means that the galaxy image is only evaluated in the unmasked region, skipping calculations -in masked regions. -""" -aplt.plot_array( - array=galaxies.image_2d_from(grid=dataset.grid), title="Galaxy Image To Be Fitted" -) - -""" -Now, we proceed to fit the image by passing both the `Imaging` and `Galaxies` objects to a `FitImaging` object. -This object will compute key quantities that describe the fit’s quality: - -`image`: Creates an image of the galaxies using their image_2d_from() method. -`model_data`: Convolves the galaxy image with the data's PSF to account for the effects of telescope optics. -`residual_map`: The difference between the model data and observed data. -`normalized_residual_map`: Residuals divided by noise values, giving units of noise. -`chi_squared_map`: Squares the normalized residuals. -`chi_squared` and `log_likelihood`: Sums the chi-squared values to compute chi_squared, and converts this into -a log_likelihood, which measures how well the model fits the data (higher values indicate a better fit). - -Let's create the fit and inspect each of these attributes: -""" -fit = ag.FitImaging(dataset=dataset, galaxies=galaxies) - -""" -The `model_data` represents the galaxy's image after accounting for effects like PSF convolution. - -An important technical note is that when we mask data, we discussed above how the image of the galaxy is not evaluated -outside the mask and is set to zero. This is a problem for PSF convolution, as the PSF blurs light from these regions -outside the mask but at its edge into the mask. They must be correctly evaluated to ensure the model image accurately -represents the image data. - -The `FitImaging` object handles this internally, but evaluating the model image in the additional regions outside the mask -that are close enough to the mask edge to be blurred into the mask. -""" -print("First model image pixel:") -print(fit.model_data.slim[0]) -aplt.plot_array(array=fit.model_data, title="Model Image") - -""" -Even before computing other fit quantities, we can normally assess if the fit is going to be good by visually comparing -the `data` and `model_data` and assessing if they look similar. - -In this example, the galaxies used to fit the data are the same as the galaxies used to simulate it, so the two -look very similar (the only difference is the noise in the image). -""" -aplt.plot_array(array=dataset.data, title="Data") -aplt.plot_array(array=fit.model_data, title="Model Image") - -""" -The `residual_map` is the different between the observed image and model image, showing where in the image the fit is -good (e.g. low residuals) and where it is bad (e.g. high residuals). - -The expression for the residual map is simply: - -\[ \text{residual} = \text{data} - \text{model\_data} \] - -The residual-map is plotted below, noting that all values are very close to zero because the fit is near perfect. -The only non-zero residuals are due to noise in the image. -""" -residual_map = dataset.data - fit.model_data -print("First residual-map pixel:") -print(residual_map.slim[0]) - -print("First residual-map pixel via fit:") -print(fit.residual_map.slim[0]) - -aplt.plot_array(array=fit.residual_map, title="Residual Map") - -""" -Are these residuals indicative of a good fit to the data? Without considering the noise in the data, it's difficult -to ascertain. That is, its hard to ascenrtain if a residual value is large or small because this depends on the -amount of noise in that pixel. - -The `normalized_residual_map` divides the residual-map by the noise-map, giving the residual in units of the noise. -Its expression is: - -\[ \text{normalized\_residual} = \frac{\text{residual\_map}}{\text{noise\_map}} = \frac{\text{data} - \text{model\_data}}{\text{noise\_map}} \] - -If you're familiar with the concept of standard deviations (sigma) in statistics, the normalized residual map represents -how many standard deviations the residual is from zero. For instance, a normalized residual of 2.0 (corresponding -to a 95% confidence interval) means that the probability of the model underestimating the data by that amount is only 5%. -""" -normalized_residual_map = residual_map / dataset.noise_map - -print("First normalized residual-map pixel:") -print(normalized_residual_map.slim[0]) - -print("First normalized residual-map pixel via fit:") -print(fit.normalized_residual_map.slim[0]) - -aplt.plot_array(array=fit.normalized_residual_map, title="Normalized Residual Map") - -""" -Next, we define the `chi_squared_map`, which is obtained by squaring the `normalized_residual_map` and serves as a -measure of goodness of fit. - -The chi-squared map is calculated as: - -\[ \chi^2 = \left(\frac{\text{data} - \text{model\_data}}{\text{noise\_map}}\right)^2 \] - -Squaring the normalized residual map ensures all values are positive. For instance, both a normalized residual of -0.2 -and 0.2 would square to 0.04, indicating the same quality of fit in terms of `chi_squared`. - -As seen from the normalized residual map, it's evident that the model provides a good fit to the data, in this -case because the chi-squared values are close to zero. -""" -chi_squared_map = (normalized_residual_map) ** 2 -print("First chi-squared pixel:") -print(chi_squared_map.slim[0]) - -print("First chi-squared pixel via fit:") -print(fit.chi_squared_map.slim[0]) - -aplt.plot_array(array=fit.chi_squared_map, title="Chi-Squared Map") - -""" -Now, we consolidate all the information in our `chi_squared_map` into a single measure of goodness-of-fit -called `chi_squared`. - -It is defined as the sum of all values in the `chi_squared_map` and is computed as: - -\[ \chi^2 = \sum \left(\frac{\text{data} - \text{model\_data}}{\text{noise\_map}}\right)^2 \] - -This summing process highlights why ensuring all values in the chi-squared map are positive is crucial. If we -didn't square the values (making them positive), positive and negative residuals would cancel each other out, -leading to an inaccurate assessment of the model's fit to the data. - -The lower the `chi_squared`, the fewer residuals exist between the model's fit and the data, indicating a better -overall fit! -""" -chi_squared = np.sum(chi_squared_map) -print("Chi-squared = ", chi_squared) -print("Chi-squared via fit = ", fit.chi_squared) - -""" -The reduced chi-squared is the `chi_squared` value divided by the number of data points (e.g., the number of pixels -in the mask). - -This quantity offers an intuitive measure of the goodness-of-fit, as it normalizes the `chi_squared` value by the -number of data points. That is, a reduced chi-squared of 1.0 indicates that the model provides a good fit to the data, -because every data point is fitted with a chi-squared value of 1.0. - -A reduced chi-squared value significantly greater than 1.0 indicates that the model is not a good fit to the data, -whereas a value significantly less than 1.0 suggests that the model is overfitting the data. -""" -reduced_chi_squared = chi_squared / dataset.mask.pixels_in_mask -print("Reduced Chi-squared = ", reduced_chi_squared) - -""" -Another quantity that contributes to our final assessment of the goodness-of-fit is the `noise_normalization`. - -The `noise_normalization` is computed as the logarithm of the sum of squared noise values in our data: - -\[ -\text{{noise\_normalization}} = \sum \log(2 \pi \text{{noise\_map}}^2) -\] - -This quantity is fixed because the noise-map remains constant throughout the fitting process. Despite this, -including the `noise_normalization` is considered good practice due to its statistical significance. - -Understanding the exact meaning of `noise_normalization` isn't critical for our primary goal of successfully -fitting a model to a dataset. Essentially, it provides a measure of how well the noise properties of our data align -with a Gaussian distribution. -""" -noise_normalization = np.sum(np.log(2 * np.pi * dataset.noise_map**2)) -print("Noise Normalization = ", noise_normalization) -print("Noise Normalization via fit = ", fit.noise_normalization) - -""" -From the `chi_squared` and `noise_normalization`, we can define a final goodness-of-fit measure known as -the `log_likelihood`. - -This measure is calculated by taking the sum of the `chi_squared` and `noise_normalization`, and then multiplying the -result by -0.5: - -\[ \text{log\_likelihood} = -0.5 \times \left( \chi^2 + \text{noise\_normalization} \right) \] - -Don't worry about why we multiply by -0.5; it's a standard practice in statistics to ensure the log likelihood is -defined correctly. -""" -log_likelihood = -0.5 * (chi_squared + noise_normalization) -print("Log Likelihood = ", log_likelihood) -print("Log Likelihood via fit = ", fit.log_likelihood) - -""" -In the previous discussion, we noted that a lower \(\chi^2\) value indicates a better fit of the model to the -observed data. - -When we calculate the log likelihood, we take the \(\chi^2\) value and multiply it by -0.5. This means that a -higher log likelihood corresponds to a better model fit. Our goal when fitting models to data is to maximize the -log likelihood. - -The **reduced \(\chi^2\)** value provides an intuitive measure of goodness-of-fit. Values close to 1.0 suggest a -good fit, while values below or above 1.0 indicate potential underfitting or overfitting of the data, respectively. -In contrast, the log likelihood values can be less intuitive. For instance, a log likelihood value printed above -might be around 5300. - -However, log likelihoods become more meaningful when we compare them. For example, if we have two models, one with -a log likelihood of 5300 and the other with 5310 we can conclude that the first model fits the data better -because it has a higher log likelihood by 10.0. - -In fact, the difference in log likelihood between models can often be associated with a probability indicating how -much better one model fits the data compared to another. This can be expressed in terms of standard deviations (sigma). - -As a rule of thumb: - -- A difference in log likelihood of **2.5** suggests that one model is preferred at the **2.0 sigma** level. -- A difference in log likelihood of **5.0** indicates a preference at the **3.0 sigma** level. -- A difference in log likelihood of **10.0** suggests a preference at the **5.0 sigma** level. - -All these metrics can be visualized together using the `FitImaging` object, which offers a comprehensive -overview of the fit quality. -""" -fit = ag.FitImaging(dataset=dataset, galaxies=galaxies) - -aplt.subplot_fit_imaging(fit=fit) - -""" -If you're familiar with model-fitting, you've likely encountered terms like 'residuals', 'chi-squared', -and 'log_likelihood' before. - -These metrics are standard ways to quantify the quality of a model fit. They are applicable not only to 1D data but -also to more complex data structures like 2D images, 3D data cubes, or any other multidimensional datasets. - -__Incorrect Fit___ - -In the previous section, we successfully created and fitted a galaxy model to the image data, resulting in an -excellent fit. The residual map and chi-squared map showed no significant discrepancies, indicating that the -galaxy's light was accurately captured by our model. This optimal solution translates to one of the highest log -likelihood values possible, reflecting a good match between the model and the observed data. - -Now, let's modify our galaxy model to create a fit that is close to the correct solution but slightly off. -Specifically, we will slightly offset the center of the galaxy by half a pixel (0.05") in both the x and y directions. -This change will allow us to observe how even small deviations from the true parameters can impact the quality of the fit. -""" -galaxy = ag.Galaxy( - redshift=0.5, - bulge=ag.lp.Sersic( - centre=(0.05, 0.05), # This is different from the previous center. - ell_comps=(0.0, 0.111111), - intensity=1.0, - effective_radius=1.0, - sersic_index=2.5, - ), -) - -galaxies = ag.Galaxies(galaxies=[galaxy]) - -""" -After implementing this slight adjustment, we can now plot the fit. In doing so, we observe that residuals have -emerged at the center of the galaxy, which indicates a mismatch between our model and the data. Consequently, -this discrepancy results in increased chi-squared values, which in turn affects our log likelihood. -""" -fit_bad = ag.FitImaging(dataset=dataset, galaxies=galaxies) - -aplt.subplot_fit_imaging(fit=fit_bad) - -""" -Next, we can compare the log likelihood of our current model to the log likelihood value we computed previously. -""" -print("Previous Likelihood:") -print(fit.log_likelihood) -print("New Likelihood:") -print(fit_bad.log_likelihood) - -""" -As expected, we observe that the log likelihood has decreased! This decline confirms that our new model is indeed a -worse fit to the data compared to the original model. - -Now, let’s change our galaxy model once more, this time setting it to a position that is far from the true parameters. -We will offset the galaxy's center significantly to see how this extreme deviation affects the fit quality. -""" -galaxy = ag.Galaxy( - redshift=0.5, - bulge=ag.lp.Sersic( - centre=( - 0.65, - 0.65, - ), # This position is significantly different from the previous one. - ell_comps=(0.0, 0.111111), - intensity=1.0, - effective_radius=1.0, - sersic_index=2.5, - ), -) - -galaxies = ag.Galaxies(galaxies=[galaxy]) - -fit_very_bad = ag.FitImaging(dataset=dataset, galaxies=galaxies) - -aplt.subplot_fit_imaging(fit=fit_very_bad) - -""" -It is now evident that this model provides a terrible fit to the data. The galaxies do not resemble a plausible -representation of our simulated galaxy dataset, which we already anticipated given that we generated the data ourselves! - -As expected, the log likelihood has dropped dramatically with this poorly fitting model. -""" -print("Previous Likelihoods:") -print(fit.log_likelihood) -print(fit_bad.log_likelihood) -print("New Likelihood:") -print(fit_very_bad.log_likelihood) - -""" -__Model Fitting__ - -In the previous sections, we used the true model to fit the data, which resulted in a high log likelihood and minimal -residuals. We also demonstrated how even small deviations from the true parameters can significantly degrade the fit -quality, reducing the log likelihood. - -In practice, however, we don't know the "true" model. For example, we might have an image of a galaxy observed with -the Hubble Space Telescope, but the values for parameters like its `effective_radius`, `sersic_index`, and others are -unknown. The process of determining the best-fit model is called model fitting, and it is the main topic of -Chapter 2 of *HowToGalaxy*. - -To conclude this section, let's perform a basic, hands-on model fit to develop some intuition about how we can find -the best-fit model. We'll start by loading a simple dataset that was simulated using a `Sersic` profile, where the -true parameters of this profile are unknown. -""" -dataset_name = "simple" -dataset_path = Path("dataset") / "imaging" / dataset_name - -dataset = ag.Imaging.from_fits( - data_path=dataset_path / "data.fits", - psf_path=dataset_path / "psf.fits", - noise_map_path=dataset_path / "noise_map.fits", - pixel_scales=0.1, -) - -mask = ag.Mask2D.circular( - shape_native=dataset.shape_native, - pixel_scales=dataset.pixel_scales, - radius=3.0, -) - -dataset = dataset.apply_mask(mask=mask) - -aplt.subplot_imaging_dataset(dataset=dataset) - - -""" -Now, you'll try to determine the best-fit model for this image, corresponding to the parameters used to simulate the -dataset. - -We'll use the simplest possible approach: try different combinations of light profile parameters and adjust them -based on how well each model fits the data. You’ll quickly find that certain parameters produce a much better fit -than others. For example, determining the correct values of the `centre` should not take too long. - -Pay attention to the `log_likelihood` and the `residual_map` as you adjust the parameters. These will guide you in -determining if your model is providing a good fit to the data. Aim to increase the log likelihood and reduce the -residuals. - -Keep experimenting with different values for a while, seeing how small you can make the residuals and how high you -can push the log likelihood. Eventually, you’ll likely reach a point where further improvements become difficult, -even after trying many different parameter values. This is a good point to stop and reflect on your first experience -with model fitting, and then to scroll to the next cell to see a discussion of the exercise. -""" - -galaxy = ag.Galaxy( - redshift=0.5, - bulge=ag.lp.Sersic( - centre=(1.0, 10), # These are the parameters - ell_comps=(0.5, 0.9), # you need to adjust - intensity=1.0, # to try and improve - effective_radius=1.0, # the model's fit - sersic_index=1.0, # to the data! - ), -) -galaxies = ag.Galaxies(galaxies=[galaxy]) - -fit = ag.FitImaging(dataset=dataset, galaxies=galaxies) - -aplt.subplot_fit_imaging(fit=fit) - -print("Log Likelihood:") -print(fit.log_likelihood) - -""" -Manually guessing model parameters repeatedly is a very inefficient and slow way to find the best fit. If the model -were more complex—say, if the `galaxy` had additional light profile components beyond just its `bulge` (like a -second `Sersic` profile representing a `disk`)—the model would become so intricate that this manual approach -would be practically impossible. This is definitely not how model fitting is done in practice. - -However, this exercise has given you a basic intuition for how model fitting works. The statistical inference tools -that are actually used for model fitting will be introduced in Chapter 2. Interestingly, these tools are not entirely -different from the approach you just tried. Essentially, they also involve iteratively testing models until those -with high log likelihoods are found. The key difference is that a computer can perform this process thousands of -times, and it does so in a much more efficient and strategic way. - -__Wrap Up__ - -In this tutorial, you have learned how to fit a galaxy model to imaging data, a fundamental process in astronomy -and statistical inference. - -Let's summarise what we have covered: - -- **Dataset**: We loaded the imaging dataset that we previously simulated, consisting of the galaxy image, noise map, - and PSF. - -- **Mask**: We applied a circular mask to the data, excluding regions with low signal-to-noise ratios from the analysis. - -- **Masked Grid**: We created a masked grid, which contains only the coordinates of unmasked pixels, to evaluate the - galaxy's light profile. - -- **Fitting**: We fitted the data with a galaxy model, computing key quantities like the model image, residuals, - chi-squared, and log likelihood to assess the quality of the fit. - -- **Bad Fits**: We demonstrated how even small deviations from the true parameters can significantly impact the fit - quality, leading to decreased log likelihood values. - -- **Model Fitting**: We performed a basic model fit on a simple dataset, adjusting the model parameters to improve the - fit quality. -""" diff --git a/scripts/howtogalaxy/chapter_1_introduction/tutorial_4_methods.py b/scripts/howtogalaxy/chapter_1_introduction/tutorial_4_methods.py deleted file mode 100644 index e69de29b..00000000 diff --git a/scripts/howtogalaxy/chapter_1_introduction/tutorial_5_summary.py b/scripts/howtogalaxy/chapter_1_introduction/tutorial_5_summary.py deleted file mode 100644 index 272c98ab..00000000 --- a/scripts/howtogalaxy/chapter_1_introduction/tutorial_5_summary.py +++ /dev/null @@ -1,176 +0,0 @@ -""" -Tutorial 9: Summary -=================== - -In this chapter, we have learnt that: - - 1) **PyAutoGalaxy** uses Cartesian `Grid2D`'s of $(y,x)$ coordinates to evaluate galaxy luminous emission. - 2) These grids are combined with light profiles to compute images and other quantities. - 3) Profiles are grouped together to make galaxies. - 4) Collections of galaxies (at the same redshift) can be made.. - 5) The Universe's cosmology can be input into this `Galaxies` to convert its units to kiloparsecs. - 6) The galaxies's image can be used to simulate galaxy `Imaging` like it was observed with a real telescope. - 7) This data can be fitted, so to as quantify how well a model galaxy system represents the observed image. - -In this summary, we'll go over all the different Python objects introduced throughout this chapter and consider how -they come together as one. - -__Contents__ - -**Initial Setup:** Create profiles, galaxies and a Galaxies object for illustration. -**Object Composition:** How Galaxies, Galaxy and Profile objects compose together. -**Visualization:** Customize and visualize any aspect of galaxies using the plotting API. -**Code Design:** Discussion of PyAutoGalaxy's object-oriented design philosophy. -**Source Code:** Links to the source code repositories for PyAutoFit, PyAutoArray and PyAutoGalaxy. -**Wrap Up:** Summary of chapter 1 and preview of the modeling chapter. -""" - -# from autoconf import setup_notebook; setup_notebook() - -from pathlib import Path -import autogalaxy as ag -import autogalaxy.plot as aplt - -""" -__Initial Setup__ - -Below, we do all the steps we have learned this chapter, making profiles, galaxies, etc. - -Note that we use two galaxies, the first of which has a bulge and disk. -""" -grid = ag.Grid2D.uniform(shape_native=(100, 100), pixel_scales=0.05) - -galaxy_0 = ag.Galaxy( - redshift=0.5, - bulge=ag.lp.Sersic( - centre=(0.0, 0.0), - ell_comps=(0.0, 0.111111), - intensity=1.0, - effective_radius=1.0, - sersic_index=2.5, - ), - disk=ag.lp.Exponential( - centre=(0.0, 0.0), - ell_comps=(0.0, 0.111111), - intensity=1.0, - effective_radius=1.0, - ), -) - -galaxy_1 = ag.Galaxy( - redshift=0.5, - bulge=ag.lp.Sersic( - centre=(1.0, 1.0), - ell_comps=(0.0, 0.111111), - intensity=1.0, - effective_radius=1.0, - sersic_index=2.5, - ), -) - -galaxies = ag.Galaxies(galaxies=[galaxy_0, galaxy_1]) - -""" -__Object Composition__ - -Lets now consider how all of the objects we've covered throughout this chapter (`LightProfile`'s, `MassProfile`'s, -`Galaxy`'s, `Galaxies`'s) come together. - -The `Galaxies` contain the `Galaxy`'s which contains the `Profile`'s: -""" -print(galaxies[0]) -print() -print(galaxies[0].bulge) -print() -print(galaxies[0].disk) -print() -print(galaxies[1].bulge) -print() - -""" -Once we have defined the galaxies, we can plot any quantity introduced throughout this chapter for a specific component, -a single galaxy, or multiple galaxies as needed. - -For example, if we want to plot the image of the first galaxy's bulge and disk, we can do this in a variety of -different ways. -""" -aplt.plot_array(array=galaxies.image_2d_from(grid=grid), title="Image") - -aplt.plot_array(array=galaxies[0].image_2d_from(grid=grid), title="Image") - -""" -Understanding how these objects decompose into the different components of a galaxy is important for general -**PyAutoGalaxy** use. - -As the galaxy systems that we analyse become more complex, it is useful to know how to decompose their light -profiles, galaxies and galaxies to extract different pieces of information about the galaxy. - -For example, we made our galaxy above with two light profiles, a `bulge` and `disk`. We can plot the image of -each component individually, now that we know how to break-up the different components of the galaxies. -""" -aplt.plot_array(array=galaxies[0].bulge.image_2d_from(grid=grid), title="Bulge Image") - -aplt.plot_array(array=galaxies[0].disk.image_2d_from(grid=grid), title="Disk Image") - -""" -__Visualization__ - -Furthermore, using the `MatPLot2D` and `Visuals2D` objects we can visualize any aspect we're interested -in and fully customize the figure. - -Before beginning chapter 2 of **HowToGalaxy**, you should checkout the package `autogalaxy_workspace/plot`. -This provides a full API reference of every plotting option in **PyAutoGalaxy**, allowing you to create your own -fully customized figures of galaxies with minimal effort! -""" -aplt.plot_array(array=galaxies[0].bulge.image_2d_from(grid=grid), title="Bulge Image") - -""" -And, we're done, not just with the tutorial, but the chapter! - -__Code Design__ - -To end, I want to quickly talk about the **PyAutoGalaxy** code-design and structure, which was really the main topic of -this tutoriag. - -Throughout this chapter, we never talk about anything like it was code. We didn`t refer to 'variables', 'parameters`' -'functions' or 'dictionaries', did we? Instead, we talked about 'galaxies'. We discussed -the objects that we, as scientists, think about when we consider a galaxy system. - -Software that abstracts the underlying code in this way follows an `object-oriented design`, and it is our hope -with **PyAutoGalaxy** that we've made its interface (often called the API for short) very intuitive, whether you were -previous familiar with galaxy morphology or a complete newcomer! - -__Source Code__ - -If you do enjoy code, variables, functions, and parameters, you may want to dig deeper into the **PyAutoGalaxy** source -code at some point in the future. Firstly, you should note that all of the code we discuss throughout the **HowToGalaxy** -lectures is not contained in just one project (e.g. the **PyAutoGalaxy** GitHub repository) but in fact three repositories: - -**PyAutoFit** - Everything required for modeling (the topic of chapter 2): https://github.com/rhayes777/PyAutoFit - -**PyAutoArray** - Handles all data structures and Astronomy dataset objects: https://github.com/Jammy2211/PyAutoArray - -**PyAutoGalaxy** - Contains the light profiles and galaxies: https://github.com/Jammy2211/PyAutoGalaxy - -Instructions on how to build these projects from source are provided here: - -https://pyautogalaxy.readthedocs.io/en/latest/installation/source.html - -We take a lot of pride in our source code, so I can promise you its well written, well documented and thoroughly -tested (check out the `test` directory if you're curious how to test code well!). - -__Wrap Up__ - -You`ve learn a lot in this chapter, but what you have not learnt is how to 'model' a real galaxy. - -In the real world, we have no idea what the 'correct' combination of light profiles are that will give a good fit to -a galaxy. Modeling is the process of finding the model which provides a good fit and it is the topic of chapter 2 -of **HowToGalaxy**. - -Finally, if you enjoyed doing the **HowToGalaxy** tutorials please git us a star on the **PyAutoGalaxy** GitHub -repository: - - https://github.com/Jammy2211/PyAutoGalaxy - -Even the smallest bit of exposure via a GitHub star can help our project grow! -""" diff --git a/scripts/howtogalaxy/chapter_2_modeling/README.rst b/scripts/howtogalaxy/chapter_2_modeling/README.rst deleted file mode 100644 index 1584168d..00000000 --- a/scripts/howtogalaxy/chapter_2_modeling/README.rst +++ /dev/null @@ -1,30 +0,0 @@ -In chapter 2, we'll take you through how to model galaxies using a non-linear search. - -**Colab** links to every tutorial are included. - -Files ------ - -`Tutorial 1: Non-linear Search `_ -- How a non-linear search is used to fit a model and the concepts of a parameter space and priors. - -`Tutorial 2: Practicalities `_ -- Practicalities of performing model-fitting, like how to inspect the results on your hard-disk. - -`Tutorial 3: Realism and Complexity `_ -- Finding a balance between realism and complexity when composing and fitting a model. - -`Tutorial 4: Dealing with Failure `_ -- What to do when PyAutoGalaxy finds an inaccurate model. - -`Tutorial 5: Linear Profiles `_ -- Light profiles which capture complex morphologies in a reduced number of non-linear parameters. - -`Tutorial 6: Masking `_ -- How to mask your data to improve the model. - -`Tutorial 7: Results `_ -- Overview of the results available after successfully fitting a model. - -`Tutorial 8: Need for Speed `_ -- How to fit complex models whilst balancing efficiency and run-time. diff --git a/scripts/howtogalaxy/chapter_2_modeling/__init__.py b/scripts/howtogalaxy/chapter_2_modeling/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/scripts/howtogalaxy/chapter_2_modeling/tutorial_1_non_linear_search.py b/scripts/howtogalaxy/chapter_2_modeling/tutorial_1_non_linear_search.py deleted file mode 100644 index 3aa77248..00000000 --- a/scripts/howtogalaxy/chapter_2_modeling/tutorial_1_non_linear_search.py +++ /dev/null @@ -1,725 +0,0 @@ -""" -Tutorial 1: Non-linear Search -============================= - -The starting point for most scientific analysis conducted by an Astronomer is that they have observations of a galaxy -using a telescope like the Hubble Space Telescope, and seek to learn about the galaxy and the Universe from these -observations. With **PyAutoGalaxy**, we seek to learn about the galaxy's structure and morphology, asking questions like -how big is the galaxy, is it disky or bulgy, and how is its light distributed? - -To answer these questions, we must therefore fit the dataset with a model of the galaxy, where the model defines the -light profile that make up the galaxy we fit. Our goal is the find the combination of light profile parameters that -best-fit the data, such that the model represents the galaxy, and therefore the Universe, as well as possible. - -This process is called model-fitting, or "modeling" for short, and we actually did our first example of this in the -previous chapter. If you recall, in the fitting tutorial we set up a fit to a simulated dataset where the parameter -used to simulate the data were unknown, and you guessed the values of the parameters that best fit the data. You -iteratively improved the model-fit by guessing new parameters, over and over, finding solutions which produced -higher `log_likelihood` values. - -However, this approach was not optimal: it was manual and slow, we had no certainty that we had found the best -(e.g. maximum likelihood) solution and for more complex models, with more parameters and light profiles, it would -have been completely unfeasible. - -In this chapter, we perform modeling as a scientist would actually do it, and introduce the statistical inference -techniques that will ultimately allow us to fit complex models made of many light profiles to real galaxy data, -and begin learning about real galaxies in the Universe. - -This first tutorial introduces a number of key statistical concepts that are fundamental to understanding how -model-fitting works, both for **PyAutoGalaxy** and in general. - -__Overview__ - -In this tutorial, we will use a non-linear search to fit a single Serisc light profile to simulated imaging of a -galaxy. We will: - -- Introduce concept like a "parameter space", "likelihood surface" and "priors", and relate them to how a non-linear - search works. - -- Introduce the `Analysis` class, which defines the `log_likelihood_function` that quantifies the goodness of fit of a - model instance to the data. - -- Fit datasets with different non-linear searches, including a maximum likelihood estimator (MLE), - Markok Chain Monte Carlo (MCMC) and nested sampling. - -__Contents__ - -This tutorial is split into the following sections: - -- **Parameter Space**: Introduce the concept of a "parameter space" and how it relates to model-fitting. -- **Non-Linear Search**: Introduce the concept of a "non-linear search" and how it fits models to data. -- **Search Types**: Introduce the maximum likelihood estimator (MLE), Markov Chain Monte Carlo (MCMC) and nested sampling search algorithms used in this tutorial. -- **Deeper Background**: Provide links to resources that more thoroughly describe the statistical principles that underpin non-linear searches. -- **Data**: Load and plot the galaxy dataset we'll fit. -- **Model**: Introduce the galaxy model we'll fit to the data. -- **Priors**: Introduce priors and how they are used to define the parameter space and guide the non-linear search. -- **Analysis**: Introduce the `Analysis` class, which contains the `log_likelihood_function` used to fit the model to the data. -- **Searches**: An overview of the searches used in this tutorial. -- **Maximum Likelihood Estimation (MLE)**: Perform a model-fit using the MLE search. -- **Markov Chain Monte Carlo (MCMC)**: Perform a model-fit using the MCMC search. -- **Nested Sampling**: Perform a model-fit using the nested sampling search. -- **Result**: The result of the model-fit, including the maximum likelihood model. -- **Samples**: The samples of the non-linear search, used to compute parameter estimates and uncertainties. -- **Customizing Searches**: How to customize the settings of the non-linear search. -- **Wrap Up**: A summary of the concepts introduced in this tutorial. - -__Parameter Space__ - -In mathematics, a function is defined by its parameters, which map inputs to outputs. For example, consider the simple function: - -\[ -f(x) = x^2 -\] - -Here, \(x\) is the input parameter, and \(f(x)\) returns the output \(x^2\). This relationship defines -the "parameter space" of the function, which in this case forms a parabola. - -Functions can also have multiple parameters, such as: - -\[ -f(x, y, z) = x + y^2 - z^3 -\] - -This defines a parameter space in three dimensions, representing the relationships between \(x\), \(y\), \(z\), -and the output \(f(x, y, z)\). - -This concept of parameter space is closely related to how we approach model-fitting. For instance, in chapter 1, w -e created instances of a `Galaxy` object with -parameters like \( (\text{`centre_0`}, \text{`centre_1`}, \text{`ell_comps_0`}, \text{`ell_comps_1`}, \text{`intensity`}, \text{`effective_radius`}, \text{`sersic_index`}) \). -These parameters were used to fit data and compute a log likelihood. - -We can think of this process as analogous to the function \(f(\( (\text{`centre_0`}, \text{`centre_1`}, \text{`ell_comps_0`}, \text{`ell_comps_1`}, \text{`intensity`}, \text{`effective_radius`}, \text{`sersic_index`}) \))\), -where the output is the log likelihood. This function, which maps parameter values to a log likelihood, is known -as the "likelihood function" in statistical inference. To be explicit, we’ll refer to it as the `log_likelihood_function` -since it deals with the log of the likelihood function. - -By framing the likelihood this way, we can think of our model as having its own parameter space—a multidimensional -surface defined by all possible values of the -parameters \( (\text{`centre_0`}, \text{`centre_1`}, \text{`ell_comps_0`}, \text{`ell_comps_1`}, \text{`intensity`}, \text{`effective_radius`}, \text{`sersic_index`}) \). -This surface, known as the "likelihood surface," represents how the log likelihood changes across different parameter -values. During model-fitting, our goal is to find the peak of this surface, where the fit to the data is optimal. - -This parameter space is "non-linear," meaning the relationship between the model parameters and the log likelihood is -not a simple linear one. Because of this non-linearity, we cannot predict the log likelihood from a given set of model -parameters without actually performing a fit to the data, as we did in tutorial 1. - -__Non-Linear Search__ - -Now that we understand our problem in terms of a non-linear parameter space with a likelihood surface, we can -introduce the method used to fit the model to the data —- the "non-linear search". - -Previously, our approach involved manually guessing models until finding one with a good fit and high log likelihood. -Surprisingly, this random guessing forms the basis of how model-fitting using a non-linear search actually works! - -A non-linear search involves systematically guessing many models while tracking their log likelihoods. As the -algorithm progresses, it tends to favor models with parameter combinations that have previously yielded higher -log likelihoods. This iterative refinement helps to efficiently explore the vast parameter space. - -There are two key differences between guessing random models and using a non-linear search: - -- **Computational Efficiency**: The non-linear search can evaluate the log likelihood of a model parameter - combinations in milliseconds and therefore many thousands of models in minutes. This computational speed enables - it to thoroughly sample potential solutions, which would be impractical for a human. - -- **Effective Sampling**: The search algorithm maintains a robust memory of previously guessed models and their log - likelihoods. This allows it to sample potential solutions more thoroughly and converge on the highest - likelihood solutions more efficiently, which is again impractical for a human. - -Think of the non-linear search as systematically exploring parameter space to pinpoint regions with the highest log -likelihood values. Its primary goal is to identify and converge on the parameter values that best describe the data. - -__Search Types__ - -There are different types of non-linear searches, each of which explores parameter space in a unique way. -In this example, we will use three types of searches, which broadly represent the various approaches to non-linear -searches used in statistical inference. - -These are: - -- **Maximum Likelihood Estimation (MLE)**: This method aims to find the model that maximizes the likelihood function. - It does so by testing nearby models and adjusting parameters in the direction that increases the likelihood. - -- **Markov Chain Monte Carlo (MCMC)**: This approach uses a group of "walkers" that explore parameter space randomly. - The likelihood at each walker's position influences the probability of the walker moving to a new position. - -- **Nested Sampling**: This technique samples points from the parameter space iteratively. Lower likelihood points - are replaced by higher likelihood ones, gradually concentrating the samples in regions of high likelihood. - -We will provide more details on each of these searches below. - -__Deeper Background__ - -**The descriptions of how searches work in this example are simplfied and phoenomenological and do not give a full -description of how they work at a deep statistical level. The goal is to provide you with an intuition for how to use -them and when different searches are appropriate for different problems. Later tutorials will provide a more formal -description of how these searches work.** - -If you're interested in learning more about these principles, you can explore resources such as: - -- [Markov Chain Monte Carlo (MCMC)](https://en.wikipedia.org/wiki/Markov_chain_Monte_Carlo) -- [Introduction to MCMC Sampling](https://twiecki.io/blog/2015/11/10/mcmc-sampling/) -- [Nested Sampling](https://www.imperial.ac.uk/media/imperial-college/research-centres-and-groups/astrophysics/public/icic/data-analysis-workshop/2016/NestedSampling_JRP.pdf) -- [A Zero-Math Introduction to MCMC Methods](https://towardsdatascience.com/a-zero-math-introduction-to-markov-chain-monte-carlo-methods-dcba889e0c50) -""" - -# from autoconf import setup_notebook; setup_notebook() - -from pathlib import Path -import autogalaxy as ag -import autogalaxy.plot as aplt - -""" -__PyAutoFit__ - -Modeling uses the probabilistic programming language -[PyAutoFit](https://github.com/rhayes777/PyAutoFit), an open-source project that allows complex model -fitting techniques to be straightforwardly integrated into scientific modeling software. - -**PyAutoFit** is actually a spin-off project of **PyAutoGalaxy**. whereby we found that the statistic techniques and -methods we applied to model galaxies could be used in a more general setting to many different scientific -problems. Check it out if you are interested in developing your own software to perform advanced model-fitting! - -We import this library separately from **PyAutoGalaxy**. -""" -import autofit as af - -""" -__Initial Setup__ - -Let's first load the `Imaging` dataset, which we will use to fit a model with a non-linear search. - -The galaxy in this image was generated using a `Sersic` light profile, which we'll also use in our model fitting -in this tutorial. This means the model we are going to fit is identical to the one used to simulate the data, -allowing us to assess the fitting process under controlled conditions. - -The dataset, as well as all subsequent datasets used in future tutorials, is stored in -the `autogalaxy_workspace/dataset/imaging` folder. -""" -dataset_name = "simple__sersic" -dataset_path = Path("dataset") / "imaging" / dataset_name - -""" -__Dataset Auto-Simulation__ - -If the dataset does not already exist on your system, it will be created by running the corresponding -simulator script. This ensures that all example scripts can be run without manually simulating data first. -""" -if not dataset_path.exists(): - import subprocess - import sys - - subprocess.run( - [sys.executable, "scripts/howtogalaxy/simulators/sersic.py"], - check=True, - ) - - -dataset = ag.Imaging.from_fits( - data_path=dataset_path / "data.fits", - noise_map_path=dataset_path / "noise_map.fits", - psf_path=dataset_path / "psf.fits", - pixel_scales=0.1, -) - -aplt.subplot_imaging_dataset(dataset=dataset) - -""" -__Mask__ - -The fit requires a mask, which we define as a 3.0" circle. -""" -mask = ag.Mask2D.circular( - shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=3.0 -) - -dataset = dataset.apply_mask(mask=mask) - -aplt.subplot_imaging_dataset(dataset=dataset) - -""" -__Model__ - -To compose a model, we will set up a `Galaxy` using a `af.Model`. Instead of manually specifying every parameter -for the galaxy's light profiles (as we did before), we will now define the galaxy using only the class of each -profile. Using a `Model` object tells **PyAutoGalaxy** that the parameters of the profiles should be fitted for -during the non-linear search. - -In this case, we'll model the galaxy with an elliptical Sersic light profile, which is linear and represents -its bulge component (the same profile used to simulate the galaxy). -""" -galaxy_model = af.Model(ag.Galaxy, redshift=0.5, bulge=ag.lp_linear.Sersic) - -""" -We now input the model component into a `Collection` object, which groups all the model components used to fit the data. - -As with profiles, we give galaxies descriptive names like `bulge`, or `disk`. Since this model has only one -galaxy, we'll simply refer to it as `galaxy` throughout the tutorials. - -It may seem odd that we define two `Collections`, with the `Collection` in the outer loop only having a `galaxies` -attribute. In future tutorials, we'll see that we can add additional model-components to a model other than just -galaxies, and the API below therefore makes it simple to extend the model to include these components. -""" -model = af.Collection(galaxies=af.Collection(galaxy=galaxy_model)) - -""" -The `info` attribute shows the model in a readable format. - -[The `info` below may not display optimally on your computer screen, for example the whitespace between parameter -names on the left and parameter priors on the right may lead them to appear across multiple lines. This is a -common issue in Jupyter notebooks. - -The`info_whitespace_length` parameter in the file `config/generag.yaml` in the [output] section can be changed to -increase or decrease the amount of whitespace (The Jupyter notebook kernel will need to be reset for this change to -appear in a notebook).] -""" -print(model.info) - -""" -__Priors__ - -When we examine the `.info` of our model, we notice that each parameter (like `centre`, `effective_radius`, -and `sersic_index` in our `Sersic` model) is associated with priors, such as `UniformPrior`. Priors define the -range of permissible values that each parameter can assume during the model fitting process, for example a uniform -prior means that a parameter is equally likely to be any value within the given range, but cannot be outside of it. - -The priors displayed above use default values defined in the `config/priors` directory. These default values have -been chosen to be broad, and contain the breadth of plausible solutions one should expect when fitting light -profiles to a real galaxy. - -For instance, consider the `centre` parameter of our `Sersic` light profile. In theory, it could take on any value from -negative to positive infinity. However, imaging datasets are typically reduced such that the galaxy centre is close -to (0.0", 0.0"). Therefore, a `GaussianPrior` with `mean=0.0` and `sigma=0.1` is a good description of where the -galaxy `centre` is. - -If the galaxy had a different centre in the dataset, we would change the mean of the prior to reflect this. -However, in general, we advise all galaxy images are reduced such that the galaxy is at (0.0", 0.0"). - -Priors serve two primary purposes: - -**Defining Valid Parameter Space:** Priors specify the range of parameter values that constitute valid solutions. -This ensures that our model explores only those solutions that are consistent with our observed data and physical -constraints. - -**Incorporating Prior Knowledge:** Priors also encapsulate our prior beliefs or expectations about the model -parameters. For instance, if we have previously fitted a similar model to another dataset and obtained certain -parameter values, we can incorporate this knowledge into our priors for a new dataset. This approach guides the -model fitting process towards parameter values that are more probable based on our prior understanding. - -Below, we manually specify the priors on all parameter in our `Sersic` model. The custom priors below are -close to the default priors drawn via configfiles, with the main purpose of the code below to show you how to -customize priors yourself. -""" -model.galaxies.galaxy.bulge.centre.centre_0 = af.GaussianPrior(mean=0.0, sigma=0.1) -model.galaxies.galaxy.bulge.centre.centre_1 = af.GaussianPrior(mean=0.0, sigma=0.1) -model.galaxies.galaxy.bulge.ell_comps.ell_comps_0 = af.UniformPrior( - lower_limit=-1.0, upper_limit=1.0 -) -model.galaxies.galaxy.bulge.ell_comps.ell_comps_1 = af.UniformPrior( - lower_limit=-1.0, upper_limit=1.0 -) -model.galaxies.galaxy.bulge.effective_radius = af.UniformPrior( - lower_limit=0.0, upper_limit=10.0 -) -model.galaxies.galaxy.bulge.sersic_index = af.UniformPrior( - lower_limit=0.8, upper_limit=8.0 -) - -""" -By reprinting the `model.info`, we can see that the priors have been updated to the values we specified. -""" -print(model.info) - -""" -__Analysis__ - -The `AnalysisImaging` object defines how an instance of a model, consisting of a set of parameters values for the -light profiles, is fitted to the `Imaging` dataset. - -The fit is performed using the analysis class's `log_likelihood_function`, which in model-fitting is a commonly used -term to describe a function that given a model and data, fits the model to the data to return a value of log -likelihood. - -In the fitting tutorial in chapter 1 you essentially performed this likelihood function yourself by hand, when you -entered different models to a `FitImaging` object and used its `log_likelihood` property to quantify how well it -fitted the data. - -A detailed step-by-step visual guide of the likelihood function is provided -at `autogalaxy_workspace/*/imaging/log_likelihood_function/parametric.ipynb`. -""" -analysis = ag.AnalysisImaging(dataset=dataset, use_jax=True) - -""" -__Searches__ - -To perform a non-linear search, we create an instance of a `NonLinearSearch` object. **PyAutoFit** offers many options -for this. A detailed description of each search method and guidance on when to use them can be found in -the [search cookbook](https://pyautofit.readthedocs.io/en/latest/cookbooks/search.html). - -In this tutorial, we’ll focus on three searches that represent different approaches to model fitting: - -1. **Maximum Likelihood Estimation (MLE)** using the `LBFGS` non-linear search. -2. **Markov Chain Monte Carlo (MCMC)** using the `Emcee` non-linear search. -3. **Nested Sampling** using the `Nautilus` non-linear search. - -In this example, non-linear search results are stored in memory rather and not written to hard disk because the fits -are fast and can therefore be easily regenerated. The next tutorial will perform fits which write results to the -hard-disk and discuss the outputs that are generated. - -__Maximum Likelihood Estimation (MLE)__ - -Maximum likelihood estimation (MLE) is the most straightforward type of non-linear search. Here is a simplified -overview of how it works: - -1. Starts at a point in parameter space with a set of initial values for the model parameters. -2. Calculates the likelihood of the model at this starting point. -3. Evaluates the likelihood at nearby points to estimate the gradient, determining the direction in which to move "up" in parameter space. -4. Moves to a new point where, based on the gradient, the likelihood is higher. - -This process repeats until the search finds a point where the likelihood can no longer be improved, indicating that -the maximum likelihood has been reached. - -The `LBFGS` search is an example of an MLE algorithm that follows this iterative procedure. Let’s see how it -performs on our Sersic model. - -In the example below, we do not specify a starting point for the MLE, so it begins at the center of the prior -range for each parameter. -""" -search = af.LBFGS() - -""" -To begin the model-fit via the non-linear search, we pass it our model and analysis and begin the fit. - -The fit will take a minute or so to run. -""" -print( - """ - The non-linear search has begun running. - This Jupyter notebook cell with progress once the search has completed - this could take a few minutes! - """ -) - -model = af.Collection(galaxies=af.Collection(galaxy=galaxy_model)) - -result = search.fit(model=model, analysis=analysis) - -print("The search has finished run - you may now continue the notebook.") - -""" -Upon completion the non-linear search returns a `Result` object, which contains information about the model-fit. - -The `info` attribute shows the result in a readable format. - -[Above, we discussed that the `info_whitespace_length` parameter in the config files could b changed to make -the `model.info` attribute display optimally on your computer. This attribute also controls the whitespace of the -`result.info` attribute.] -""" -print(result.info) - - -""" - -One thing the result contains we'll use now is the `FitImaging` object that corresponds to the set of model -parameters that gave the maximum log likelihood solution. - -We plot this object to inspect how good our fit was. -""" -aplt.subplot_fit_imaging(fit=result.max_log_likelihood_fit) - -""" -The fit quality is poor, with significant residuals, meaning that the MLE failed to identify the correct model. - -This happened because the starting point of the search was a poor match to the data, placing it far from the true -solution in parameter space. As a result, after moving "up" the likelihood gradient several times, the search -settled into a "local maximum", a local peak in the likelihood surface that was better than models that surrounded -it in parameter space, but far from the global maximum solution. - -To achieve a better fit with MLE, the search needs to begin in a region of parameter space where the log likelihood -is higher. This process is known as "initialization" and it involves providing the search with an -appropriate "starting point" in parameter space. - -The code below shows how we can customize the starting point of the search to provide a better initialization, -with the values input below close to the true values used to simulate the dataset. -""" -initializer = af.InitializerParamStartPoints( - { - model.galaxies.galaxy.bulge.centre.centre_0: 0.0, - model.galaxies.galaxy.bulge.centre.centre_1: 0.0, - model.galaxies.galaxy.bulge.ell_comps.ell_comps_0: 0.1, - model.galaxies.galaxy.bulge.ell_comps.ell_comps_1: 0.05, - model.galaxies.galaxy.bulge.effective_radius: 1.0, - model.galaxies.galaxy.bulge.sersic_index: 3.7, - } -) - -search = af.LBFGS(initializer=initializer) - -print( - """ - The non-linear search has begun running. - This Jupyter notebook cell with progress once the search has completed - this could take a few minutes! - """ -) - -model = af.Collection(galaxies=af.Collection(galaxy=galaxy_model)) - -result = search.fit(model=model, analysis=analysis) - -print("The search has finished run - you may now continue the notebook.") - -""" -By printing `result.info` and looking at the maximum log likelihood model, we can confirm the search provided a -good model fit with a much higher likelihood than the incorrect model above. -""" -print(result.info) - -aplt.subplot_fit_imaging(fit=result.max_log_likelihood_fit) - -""" -The MLE is a great starting point for model-fitting because it’s fast, conceptually simple, and often yields -accurate results. It is especially effective if you can provide a good initialization, allowing it to find the -best-fit solution quickly. - -However, MLE has its limitations. As seen above, it can get "stuck" in a local maximum, particularly if the -starting point is poorly chosen. In complex model-fitting problems, providing a suitable starting point can be -challenging. While MLE performed well in the example with just three parameters, it struggles with models that have -many parameters, as the complexity of the likelihood surface makes simply moving "up" the gradient less effective. - -The MLE also does not provide any information on the errors on the parameters, which is a significant limitation. -The next two types of searches "map out" the likelihood surface, such that they not only infer the maximum likelihood -solution but also quantify the errors on the parameters. - -__Markov Chain Monte Carlo (MCMC)__ - -Markov Chain Monte Carlo (MCMC) is a more powerful method for model-fitting, though it is also more computationally -intensive and conceptually complex. Here’s a simplified overview: - -1. Place a set of "walkers" in parameter space, each with random parameter values. -2. Calculate the likelihood of each walker's position. -3. Move the walkers to new positions, guided by the likelihood of their current positions. Walkers in high-likelihood -regions encourage those in lower regions to move closer to them. - -This process repeats, with the walkers converging on the highest-likelihood regions of parameter space. - -Unlike MLE, MCMC thoroughly explores parameter space. While MLE moves a single point up the likelihood gradient, -MCMC uses many walkers to explore high-likelihood regions, making it more effective at finding the global maximum, -though slower. - -There is a reduced risk of getting stuck in local maxima, because whilst some walkers may temporarily get stuck in -one, other walkers will explore other regions of parameter space and encourage the walker in local maxima to move -away from it and head towards higher likelihood regions. - -In the example below, we use the `Emcee` MCMC search to fit the galaxy. The search starts with walkers -initialized in a "ball" around the center of the model’s priors, similar to the MLE search that failed earlier. -""" -search = af.Emcee( - nwalkers=20, # The number of walkers we'll use to sample parameter space. - nsteps=300, # The number of steps each walker takes, after which 20 * 300 = 6000 steps the non-linear search ends. -) - -print( - """ - The non-linear search has begun running. - This Jupyter notebook cell with progress once the search has completed - this could take a few minutes! - """ -) - -model = af.Collection(galaxies=af.Collection(galaxy=galaxy_model)) - -result = search.fit(model=model, analysis=analysis) - -print("The search has finished run - you may now continue the notebook.") - -print(result.info) - -aplt.subplot_fit_imaging(fit=result.max_log_likelihood_fit) - -""" -The MCMC search succeeded, finding the same high-likelihood model that the MLE search with a good starting point -identified, even without a good initialization. Its use of multiple walkers exploring parameter space allowed it to -avoid the local maxima that had trapped the MLE search. - -A major advantage of MCMC is that it provides estimates of parameter uncertainties by "mapping out" the likelihood -surface, unlike MLE, which only finds the maximum likelihood solution. These error estimates are accessible in -the `result.info` string and through the `result.samples` object, which is explained fully in tutorial 5. - -While a good starting point wasn't necessary for this simple model, it becomes essential for efficiently mapping the -likelihood surface in more complex models with many parameters. The code below shows an MCMC fit using a good starting -point, with two key differences from the MLE initialization: - -1. Instead of single starting values, we provide bounds for each parameter. MCMC initializes each walker in a -small "ball" in parameter space, requiring a defined range for each parameter from which values are randomly drawn. - -2. We do not specify a starting point for the `ell_comps` parameters, allowing its initial values to be drawn from its -priors. This illustrates that with MCMC, it’s not necessary to know a good starting point for every parameter. -""" -initializer = af.InitializerParamBounds( - { - model.galaxies.galaxy.bulge.centre.centre_0: (-0.01, 0.01), - model.galaxies.galaxy.bulge.centre.centre_1: (-0.01, 0.01), - model.galaxies.galaxy.bulge.effective_radius: (0.0, 2.0), - model.galaxies.galaxy.bulge.sersic_index: (3.0, 5.0), - } -) - -search = af.Emcee( - nwalkers=20, # The number of walkers we'll use to sample parameter space. - nsteps=300, # The number of steps each walker takes, after which 10 * 200 = 2000 steps the non-linear search ends. - initializer=initializer, -) - -print( - """ - The non-linear search has begun running. - This Jupyter notebook cell with progress once the search has completed - this could take a few minutes! - """ -) - -model = af.Collection(galaxies=af.Collection(galaxy=galaxy_model)) - -result = search.fit(model=model, analysis=analysis) - -print("The search has finished run - you may now continue the notebook.") - -print(result.info) - -""" -MCMC is a powerful tool for model-fitting, providing accurate parameter estimates and uncertainties. For simple models -without a starting point, MCMC can still find the correct solution, and if a good starting point is provided, it can -efficiently scale to more complex models with more parameters. - -The main limitation of MCMC is that one has to supply the number of steps the walkers take (`nsteps`). If this value -is too low, the walkers may not explore the likelihood surface sufficiently. It can be challenging to know the right -number of steps, especially if models of different complexity are being fitted or if datasets of varying quality are -used. One often ends up having to perform "trial and error" to verify a sufficient number of steps are used. - -MCMC can perform badly in parameter spaces with certain types of complexity, for example when there are -are many local maxima "peaks" that many of the walkers can become stuck walking around them, so many walkers that -when they communicate with one another they cannot gain enough information to move away from the local peak. - -__Nested Sampling__ - -**Nested Sampling** is an advanced method for model-fitting that excels in handling complex models with intricate -parameter spaces. Here’s a simplified overview of its process: - -1. Start with a set of "live points" in parameter space, each initialized with random parameter values drawn from their respective priors. - -2. Compute the log likelihood for each live point. - -3. Draw a new point based on the likelihood of the current live points, favoring regions of higher likelihood. - -4. If the new point has a higher likelihood than any existing live point, it becomes a live point, and the lowest likelihood live point is discarded. - -This iterative process continues, gradually focusing the live points around higher likelihood regions of parameter -space until they converge on the highest likelihood solution. - -Like MCMC, Nested Sampling effectively maps out parameter space, providing accurate estimates of parameters and -their uncertainties. -""" -search = af.Nautilus( - n_live=100, - n_batch=50, # Explained in next tutorial -) - -""" -To begin the model-fit via the non-linear search, we pass it our model and analysis and begin the fit. - -The fit will take a minute or so to run. -""" -print( - """ - The non-linear search has begun running. - This Jupyter notebook cell with progress once the search has completed - this could take a few minutes! - """ -) - -model = af.Collection(galaxies=af.Collection(galaxy=galaxy_model)) - -result = search.fit(model=model, analysis=analysis) - -print("The search has finished run - you may now continue the notebook.") - -print(result.info) - -aplt.subplot_fit_imaging(fit=result.max_log_likelihood_fit) - -""" -The **Nested Sampling** search was successful, identifying the same high-likelihood model as the MLE and MCMC searches. -One of the main benefits of Nested Sampling is its ability to provide accurate parameter estimates and uncertainties, -similar to MCMC. Additionally, it features a built-in stopping criterion, which eliminates the need for users to -specify the number of steps the search should take. - -This method also excels in handling complex parameter spaces, particularly those with multiple local peaks. This is because -the live points will identify each peak and converge around them, but then begin to be discarded from a peak if higher -likelihood points are found elsewhere in parameter space. In MCMC, the walkers can get stuck indefinitely in -peaks, causing the method to stall. - -Another significant advantage is that Nested Sampling estimates an important statistical quantity -known as "evidence." This value quantifies how well the model fits the data while considering the model's complexity, -making it essential for Bayesian model comparison, which will be covered in later tutorials. - -Nested sampling cannot use a starting point, as it always samples parameter space from scratch by drawing live points -from the priors. This is both good and bad, depending on if you have access to a good starting point or not. If you do -not, your MCMC / MLE fit will likely struggle with initialization compared to Nested Sampling. Conversely, if you do -possess a robust starting point, it can significantly enhance the performance of MCMC, allowing it to begin closer to -the highest likelihood regions of parameter space. This proximity can lead to faster convergence and more reliable results. - -However, Nested Sampling does have limitations; it often scales poorly with increased model complexity. For example, -once a model has around 50 or more parameters, Nested Sampling can become very slow, whereas MCMC remains efficient -even in such complex parameter spaces. - -__What is The Best Search To Use?__ - -In statistical inference, the choice of the best search method depends on several factors specific to the problem, -such as model complexity, availability of a starting point, and whether error estimates are required. More subtle -aspects, like the shape of the likelihood surface and the presence of multiple peaks, also play a role, as you'll -explore in later tutorials. - -When approaching a model-fitting problem, it's usually advisable to try various search methods to find the most -effective one. - -However, after extensive testing within **PyAutoGalaxy**, a clear recommendation has emerged for fitting galaxy light -profiles and studying galaxy structure. The nested sampling method `Nautilus` consistently proves to be the most -effective. It requires fewer iterations than MCMC or MLE methods (when no starting point is used), provides robust -sampling even for complex models, includes a built-in stopping criterion, and delivers reliable error estimates. - -All examples in the `autogalaxy_workspace` use the `Nautilus` search, and future tutorials in the **HowToGalaxy** -series will also use it. We strongly recommend using `Nautilus` from now on. - -That said, MLE and MCMC searches can still be effective, and you're encouraged to experiment with them. If you have -a reliable starting point, MLE and MCMC can outperform `Nautilus` in terms of speed and efficiency. Additionally, -for very complex models (e.g., with 30+ parameters), `Nautilus` may struggle, while MCMC might handle the complexity -better—though you're unlikely to encounter that level of complexity soon. - -Lastly, MLE, MCMC, and nested sampling are just three categories of non-linear searches, each containing different -algorithms with their own strengths and weaknesses. Exploring these options could help you find the best approach -for your model-fitting needs. For more details on each method, see the [search cookbook](https://pyautofit.readthedocs.io/en/latest/cookbooks/search.html). - -__Wrap Up__ - -This tutorial has laid the foundation with several fundamental concepts in model fitting and statistical inference: - -1. **Parameter Space**: This refers to the range of possible values that each parameter in a model can take. It -defines the dimensions over which the likelihood of different parameter values is evaluated. - -2. **Likelihood Function**: This function receives as input a model with specific parameter values and returns a -log likelihood value that quantifies how well the model describes the data. - -3. **Likelihood Surface**: This surface represents how the likelihood of the model varies across the parameter space. -It helps in identifying the best-fit parameters that maximize the likelihood of the model given the data. - -4. **Non-linear Search**: This is an optimization technique used to explore the parameter space and find the -combination of parameter values that best describe the data. It iteratively adjusts the parameters to maximize the -likelihood. Many different search algorithms exist, each with their own strengths and weaknesses, and this tutorial -used the MLE, MCMC, and nested sampling searches. - -5. **Priors**: Priors are probabilities assigned to different values of parameters before considering the data. -They encapsulate our prior knowledge or assumptions about the parameter values. Priors can constrain the parameter -space, making the search more efficient and realistic. - -6. **Model Fitting**: The process of adjusting model parameters to minimize the difference between model predictions -and observed data, quantified by the likelihood function. - -Understanding these concepts is crucial as they form the backbone of model fitting and parameter estimation in -scientific research and data analysis. In the next tutorials, these concepts will be further expanded upon to -deepen your understanding and provide more advanced techniques for model fitting and analysis. -""" diff --git a/scripts/howtogalaxy/chapter_2_modeling/tutorial_2_practicalities.py b/scripts/howtogalaxy/chapter_2_modeling/tutorial_2_practicalities.py deleted file mode 100644 index cfb55d4e..00000000 --- a/scripts/howtogalaxy/chapter_2_modeling/tutorial_2_practicalities.py +++ /dev/null @@ -1,393 +0,0 @@ -""" -Tutorial 2: Practicalities -========================== - -In the last tutorial, we introduced foundational statistical concepts essential for model-fitting, such as parameter -spaces, likelihoods, priors, and non-linear searches. Understanding these statistical concepts is crucial for -performing model fits effectively. - -However, achieving successful model-fitting also requires practical skills, including how to manage outputs, -review results, interpret model quality, and ensure run-times are efficient enough for your scientific needs. - -This tutorial will focus on these practical aspects of model-fitting, including: - -- How to save results to your hard disk. -- How to navigate the output folder and examine model-fit results to assess quality. -- How to estimate the run-time of a model-fit before initiating it, and change settings to make it faster or run the analysis in parallel. - -__Contents__ - -This tutorial is split into the following sections: - - **PyAutoFit:** The parent package of PyAutoGalaxy, which handles practicalities of model-fitting. - **Initial Setup:** Load the dataset we'll fit a model to using a non-linear search. - **Mask:** Apply a mask to the dataset. - **Model:** Introduce the model we will fit to the data. - **Search:** Setup the non-linear search, Nautilus, used to fit the model to the data. - **Search Settings:** Discuss the settings of the non-linear search, including the number of live points. - **Number Of Cores:** Discuss how to use multiple cores to fit models faster in parallel. - **Parallel Script:** Running the model-fit in parallel if a bug occurs in a Jupiter notebook. - **Iterations Per Update:** How often the non-linear search outputs the current results to hard-disk. - **Analysis:** Create the Analysis object which contains the `log_likelihood_function` that the non-linear search calls. - **Model-Fit:** Fit the model to the data. - **Result:** Print the results of the model-fit to the terminal. - **Output Folder:** Inspect the output folder where results are stored. - **Unique Identifier:** Discussion of the unique identifier of the model-fit which names the folder in the output directory. - **Output Folder Contents:** What is output to the output folder (model results, visualization, etc.). - **Result:** Plot the best-fit model to the data. -""" - -# from autoconf import setup_notebook; setup_notebook() - -from pathlib import Path -import autogalaxy as ag -import autogalaxy.plot as aplt - -""" -__PyAutoFit__ - -Modeling uses the probabilistic programming language -[PyAutoFit](https://github.com/rhayes777/PyAutoFit), an open-source project that allows complex model -fitting techniques to be straightforwardly integrated into scientific modeling software. - -The majority of tools that make model-fitting practical are provided by PyAutoFit, for example it handles -all output of the non-linear search to hard-disk, the visualization of results and the estimation of run-times. -""" -import autofit as af - -""" -__Initial Setup__ - -Lets first load the `Imaging` dataset we'll fit a model with using a non-linear search. - -This is the same dataset we fitted in the previous tutorial, and we'll repeat the same fit, as we simply want -to illustrate the practicalities of model-fitting in this tutorial. -""" -dataset_name = "simple__sersic" -dataset_path = Path("dataset") / "imaging" / dataset_name - -""" -__Dataset Auto-Simulation__ - -If the dataset does not already exist on your system, it will be created by running the corresponding -simulator script. This ensures that all example scripts can be run without manually simulating data first. -""" -if not dataset_path.exists(): - import subprocess - import sys - - subprocess.run( - [sys.executable, "scripts/howtogalaxy/simulators/sersic.py"], - check=True, - ) - - -dataset = ag.Imaging.from_fits( - data_path=dataset_path / "data.fits", - noise_map_path=dataset_path / "noise_map.fits", - psf_path=dataset_path / "psf.fits", - pixel_scales=0.1, -) - -aplt.subplot_imaging_dataset(dataset=dataset) - -""" -__Mask__ - -We again apply a 3.0" mask to the dataset. -""" -mask = ag.Mask2D.circular( - shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=3.0 -) - -dataset = dataset.apply_mask(mask=mask) - -aplt.subplot_imaging_dataset(dataset=dataset) - -""" -__Over Sampling__ - -Over sampling is a numerical technique where the images of light profiles and galaxies are evaluated -on a higher resolution grid than the image data to ensure the calculation is accurate. - -For a new user, the details of over-sampling are not important, therefore just be aware that below we make it so that -all calculations use an adaptive over sampling scheme which ensures high accuracy and precision. - -Once you are more experienced, you should read up on over-sampling in more detail via -the `autogalaxy_workspace/*/guides/over_sampling.ipynb` notebook. -""" -over_sample_size = ag.util.over_sample.over_sample_size_via_radial_bins_from( - grid=dataset.grid, - sub_size_list=[8, 4, 1], - radial_list=[0.3, 0.6], - centre_list=[(0.0, 0.0)], -) - -dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size) - - -""" -__Model__ - -We compose the model using the same API as the previous tutorial. - -This model is the same as the previous tutorial, a single `Galaxy` with a `Sersic` light profile. -""" -galaxy_model = af.Model(ag.Galaxy, redshift=0.5, bulge=ag.lp_linear.Sersic) - -model = af.Collection(galaxies=af.Collection(galaxy=galaxy_model)) - -print(model.info) - -""" -__Search__ - -To fit the model, we now create the non-linear search object, selecting the nested sampling algorithm, Nautilus, -as recommended in the previous tutorial. - -We set up the `Nautilus` object with these parameters: - -- **`path_prefix`**: specifies the output directory, here set to `autogalaxy_workspace/output/howtogalaxy/chapter_2`. - -- **`name`**: gives the search a descriptive name, which creates the full output path -as `autogalaxy_workspace/output/howtogalaxy/chapter_2/tutorial_2_practicalities`. - -- **`n_live`**: controls the number of live points Nautilus uses to sample parameter space. - -__Search Settings__ - -Nautilus samples parameter space by placing "live points" representing different galaxy models. Each point has an -associated `log_likelihood` that reflects how well it fits the data. By mapping where high-likelihood solutions are -located, it can focus on searching those regions. - -The main setting to balance is the **number of live points**. More live points allow Nautilus to map parameter space -more thoroughly, increasing accuracy but also runtime. Fewer live points reduce run-time but may make the search less -reliable, possibly getting stuck in local maxima. - -The ideal number of live points depends on model complexity. More parameters generally require more live points, but -the default of 200 is sufficient for most galaxy models. Lower values can still yield reliable results, particularly -for simpler models. For this example (6 parameters), we reduce the live points to 80 to speed up runtime without -compromising accuracy. - -Tuning non-linear search settings (e.g., the number of live points) to match model complexity is essential. We aim -for enough live points to ensure accurate results (i.e., finding a global maximum) but not so many that runtime -is excessive. - -In practice, the optimal number of live points is often found through trial and error, guided by summary statistics -on how well the search is performing, which we’ll cover below. For this single Sersic model with a linear light -profile, 80 live points is sufficient to achieve reliable results. - -__Iterations Per Update__ - -Every N iterations, the non-linear search outputs the current results to the folder `autogalaxy_workspace/output`, -which includes producing visualization. - -Depending on how long it takes for the model to be fitted to the data (see discussion about run times below), -this can take up a large fraction of the run-time of the non-linear search. - -For this fit, the fit is very fast, thus we set a high value of `iterations_per_quick_update=10000` to ensure these updates -so not slow down the overall speed of the model-fit. - -**If the iteration per update is too low, the model-fit may be significantly slowed down by the time it takes to -output results and visualization frequently to hard-disk. If your fit is consistent displaying a log saying that it -is outputting results, try increasing this value to ensure the model-fit runs efficiently.** -""" -search = af.Nautilus( - path_prefix=Path("howtogalaxy", "chapter_2"), - name="tutorial_2_practicalities", - unique_tag=dataset_name, - n_live=80, - n_batch=50, # GPU lens model fits are batched and run simultaneously, see VRAM section below. - iterations_per_quick_update=2500, -) - -""" -__Analysis__ - -We again create the `AnalysisImaging` object which contains the `log_likelihood_function` that the non-linear search -calls to fit the model to the data. -""" -analysis = ag.AnalysisImaging(dataset=dataset, use_jax=True) - -""" -__Run Times__ - -Another key practicality in model-fitting is the run-time of the model-fit. Modeling can be a computationally -expensive process, whereby the fitting of complex models to high resolution datasets run times may be of order hours, -days or weeks. - -Run times are dictated by two factors: - - - **The log likelihood evaluation time:** the time it takes for a single `instance` of the model to be fitted to - the dataset such that a log likelihood is returned. - - - **The number of iterations (e.g. log likelihood evaluations) performed by the non-linear search:** more complex lens - models require more iterations to converge to a solution (and as discussed above, settings like the number of live - points also control this). - -For this analysis, the log likelihood evaluation time is ~0.05 seconds, which is extremely fast for model fitting. - -The more advanced fitting techniques discussed at the end of chapter 1 (e.g. shapelets, multi Gaussian expansions, -pixelizations) have longer log likelihood evaluation times. However, on GPU, they can be so fast they may not produce -significantly longer overall run-time feasible. - -To estimate the expected overall run time of the model-fit we multiply the log likelihood evaluation time by an -estimate of the number of iterations the non-linear search will perform, which is around 50000 to 100000 for this model. - -Tests thus far reveal CPU run times are around 30 minutes, and GPU run times around 10 minutes. - -__Model-Fit__ - -To begin the model-fit, we pass the model and analysis objects to the search, which performs a non-linear search to -identify models that best fit the data. - -Running model fits via non-linear search can take significant time. While the fit in this tutorial should -complete in a few minutes, more complex models may require longer run times. In Jupyter notebooks, this can be -limiting, as the notebook cell will only complete once the fit finishes, preventing you from advancing through the -tutorial or running additional code. - -To work around this, we recommend running tutorial scripts as standalone Python scripts, found -in `autogalaxy_workspace/scripts/howtogalaxy`. These scripts mirror the notebook tutorials but run independently of -Jupyter notebooks. For example, you can start a script with: - -`python3 scripts/howtogalaxy/chapter_2_modeling/tutorial_2_practicalities.py` - -Using scripts allows results to be saved to the hard drive in the `output` folder, enabling you to inspect results -immediately once the script completes. When rerun, the script loads results directly from disk, so any Jupyter -notebook cells will quickly load and display the complete model-fit results if they’re already saved. - -This approach not only avoids the slowdowns associated with notebook cells during lengthy runs but is also essential -for using super-computers for fitting tasks, as they require separate Python scripts. - -For tasks like loading results, inspecting data, plotting, and interpreting results, Jupyter notebooks remain ideal. -""" -print( - "The non-linear search has begun running - checkout the autogalaxy_workspace/output/" - " folder for live output of the results, images and model." - " This Jupyter notebook cell with progress once search has completed - this could take some time!" -) - -result = search.fit(model=model, analysis=analysis) - -print("Search has finished run - you may now continue the notebook.") - -""" -__Result Info__ - -A concise readable summary of the results is given by printing its `info` attribute. - -[Above, we discussed that the `info_whitespace_length` parameter in the config files could b changed to make -the `model.info` attribute display optimally on your computer. This attribute also controls the whitespace of the -`result.info` attribute.] -""" -print(result.info) - -""" -__Output Folder__ - -Now checkout the `autogalaxy_workspace/output` folder. - -This is where the results of the search are written to hard-disk (in the `tutorial_2_practicalities` folder). - -Once completed images, results and information about the fit appear in this folder, meaning that you don't need -to keep running Python code to see the result. - -__Unique Identifier__ - -In the output folder, you will note that results are in a folder which is a collection of random characters. This acts -as a `unique_identifier` of the model-fit, where this identifier is generated based on the model, search and dataset -that are used in the fit. - -An identical combination of model, search and dataset generates the same identifier, meaning that rerunning the -script will use the existing results to resume the model-fit. In contrast, if you change the model, search or dataset, -a new unique identifier will be generated, ensuring that the model-fit results are output into a separate folder. - -We additionally want the unique identifier to be specific to the dataset fitted, so that if we fit different datasets -with the same model and search results are output to a different folder. We achieved this for the fit above by passing -the `dataset_name` to the search's `unique_tag`. - -__Output Folder Contents__ - -Now this is running you should checkout the `autogalaxy_workspace/output` folder. This is where the results of the -search are written to hard-disk (in the `start_here` folder), where all outputs are human readable (e.g. as .json, -.csv or text files). - -As the fit progresses, results are written to the `output` folder on the fly using the highest likelihood model found -by the non-linear search so far. This means you can inspect the results of the model-fit as it runs, without having to -wait for the non-linear search to terminate. - -The `output` folder includes: - - - `model.info`: Summarizes the model, its parameters and their priors discussed in the next tutorial. - - - `model.results`: Summarizes the highest likelihood model inferred so far including errors. - - - `images`: Visualization of the highest likelihood model-fit to the dataset, (e.g. a fit subplot showing the - galaxies, model data and residuals). - - - `files`: A folder containing .fits files of the dataset, the model as a human-readable .json file, - a `.csv` table of every non-linear search sample and other files containing information about the model-fit. - - - `search.summary`: A file providing summary statistics on the performance of the non-linear search. - - - `search_internal`: Internal files of the non-linear search (in this case Nautilus) used for resuming the fit and - visualizing the search. - -__Result__ - -The `search.fit` method produces a `result` object, packed with useful information about the model fit that we’ll -explore in detail in a later tutorial. - -One component of the `result` object we’ll use now is the `FitImaging` object, which corresponds to the set of model -parameters that yielded the maximum log-likelihood solution. Plotting this object lets us visually inspect how well -the model fits the data. - -In this example, the fit to the data is excellent, with residuals near zero, as expected since the same model was -used both to simulate and fit the data. -""" -aplt.subplot_fit_imaging(fit=result.max_log_likelihood_fit) - -""" -The Probability Density Functions (PDF's) of the results can be plotted using Nautilus's in-built visualization -library, which is wrapped via the `corner_anesthetic` function. - -The PDF shows the 1D and 2D probabilities estimated for every parameter after the model-fit. The two dimensional -figures can show the degeneracies between different parameters, for example how increasing the intensity $I$ of the -source galaxy and decreasing its effective radius $R_{Eff}$ lead to similar likelihoods and probabilities. - -This PDF will be discussed more in the next tutorial. - -The plot is labeled with short hand parameter names (e.g. `sersic_index` is mapped to the short hand -parameter `n`). These mappings ate specified in the `config/notation.yaml` file and can be customized by users. - -The superscripts of labels correspond to the name each component was given in the model (e.g. for the `Isothermal` -mass its name `mass` defined when making the `Model` above is used). -""" - -""" -__Other Practicalities__ - -The following are examples of other practicalities which I will document fully in this example script in the future, -but so far have no found the time: - -- `config`: The files in `autogalaxy_workspace/config` which control many aspects of how PyAutoGalaxy runs, - including visualization, the non-linear search settings. - -- `config/priors`: Folder containing the default priors on all model components. - -- `results`: How to load the results of a model-fit from the output folder to a Python script or Jupyter notebook. - -- `output.yaml`: What files are output to control file size. - -__Wrap Up__ - -This tutorial has illustrated how to handle a number of practicalities that are key to performing model-fitting -effectively. These include: - -- How to save results to your hard disk. - -- How to navigate the output folder and examine model-fit results to assess quality. - -- How to estimate the run-time of a model-fit before initiating it, and change settings to make it faster or run the - analysis in parallel. -""" diff --git a/scripts/howtogalaxy/chapter_2_modeling/tutorial_3_realism_and_complexity.py b/scripts/howtogalaxy/chapter_2_modeling/tutorial_3_realism_and_complexity.py deleted file mode 100644 index ec2f3cbe..00000000 --- a/scripts/howtogalaxy/chapter_2_modeling/tutorial_3_realism_and_complexity.py +++ /dev/null @@ -1,243 +0,0 @@ -""" -Tutorial 3: Realism and Complexity -================================== - -In the previous two tutorials, we fitted a fairly basic model: the galaxy's light was a single bulge component. -In real observations we know that galaxies are observed to have multiple different morphological structures. - -In this tutorial, we'll use a more realistic model, which consists of the following light profiles: - - - An `Sersic` light profile for the galaxy's bulge [7 parameters]. - - An `Exponential` light profile for the galaxy's disk [6 parameters] - -This model has 13 free parameters, meaning that the parameter space and likelihood function it defines has a -dimensionality of N=13. This is over double the number of parameters and dimensions of the models we fitted in the -previous tutorials and in future exercises, we will fit even more complex models with some 13+ parameters. - -Therefore, take note, as we make our model more realistic, we also make its parameter space more complex, this is -an important concept to keep in mind for the remainder of this chapter! - -__Contents__ - -**Initial Setup:** Load the dataset and apply a mask. -**Model + Search + Analysis:** Compose a bulge+disk model and fit it using Nautilus. -**Result:** Inspect the result and compare to the true model. -**Global and Local Maxima:** Demonstrate how the search can infer incorrect local maxima solutions. -**Wrap Up:** Summary of the challenges of fitting more complex and realistic models. -""" - -# from autoconf import setup_notebook; setup_notebook() - -from pathlib import Path -import autogalaxy as ag -import autogalaxy.plot as aplt -import autofit as af - -""" -__Initial Setup__ - -we'll use new galaxying data, where: - - - The galaxy's bulge is an `Sersic`. - - The galaxy's disk is an `Exponential`. -""" -dataset_name = "simple" -dataset_path = Path("dataset") / "imaging" / dataset_name - -""" -__Dataset Auto-Simulation__ - -If the dataset does not already exist on your system, it will be created by running the corresponding -simulator script. This ensures that all example scripts can be run without manually simulating data first. -""" -if not dataset_path.exists(): - import subprocess - import sys - - subprocess.run( - [sys.executable, "scripts/imaging/simulator.py"], - check=True, - ) - - -dataset = ag.Imaging.from_fits( - data_path=dataset_path / "data.fits", - noise_map_path=dataset_path / "noise_map.fits", - psf_path=dataset_path / "psf.fits", - pixel_scales=0.1, -) - -""" -__Mask__ - -We'll create and use a 2.5" `Mask2D`, which is slightly smaller than the masks we used in previous tutorials. -""" -mask = ag.Mask2D.circular( - shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=2.5 -) - -dataset = dataset.apply_mask(mask=mask) - -""" -__Over Sampling__ - -Apply adaptive over sampling to ensure the calculation is accurate, you can read up on over-sampling in more detail via -the `autogalaxy_workspace/*/guides/over_sampling.ipynb` notebook. -""" -over_sample_size = ag.util.over_sample.over_sample_size_via_radial_bins_from( - grid=dataset.grid, - sub_size_list=[8, 4, 1], - radial_list=[0.3, 0.6], - centre_list=[(0.0, 0.0)], -) - -dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size) - -""" -When plotted, the galaxy's bulge and disk are clearly visible in the centre of the image. -""" -aplt.subplot_imaging_dataset(dataset=dataset) - -""" -__Model + Search + Analysis__ - -Now lets fit the dataset using a search. -""" -model = af.Collection( - galaxies=af.Collection( - galaxy=af.Model( - ag.Galaxy, redshift=0.5, bulge=ag.lp.Sersic, disk=ag.lp.Exponential - ) - ) -) - -search = af.Nautilus( - path_prefix=Path("howtogalaxy", "chapter_2"), - name="tutorial_3_realism_and_complexity", - unique_tag=dataset_name, - n_live=100, - n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. -) - -analysis = ag.AnalysisImaging(dataset=dataset, use_jax=True) - -print( - "The non-linear search has begun running - checkout the autogalaxy_workspace/output/howtogalaxy/chapter_2/tutorial_3_realism_and_complexity" - " folder for live output of the results, images and model." - " This Jupyter notebook cell with progress once search has completed - this could take some time!" -) - -result = search.fit(model=model, analysis=analysis) - -print("Search has finished run - you may now continue the notebook.") - -""" -__Result__ - -Inspection of the `info` summary of the result suggests the model has gone to reasonable values. -""" -print(result.info) - -""" -And lets look at how well the model fits the imaging data, which as we are used to fits the data brilliantly! -""" -aplt.subplot_fit_imaging(fit=result.max_log_likelihood_fit) - -""" -__Global and Local Maxima__ - -Up to now, all of our non-linear searches have been successes. They find a model that provides a visibly good fit -to the data, minimizing the residuals and inferring a high log likelihood value. - -These solutions are called 'global maxima', they correspond to the highest likelihood regions over all of parameter -space. There are no other models in parameter space that would give higher likelihoods, this is the model we want -to always infer! - -However, non-linear searches may not always successfully locate the global maxima models. They may instead infer -a 'local maxima', a solution which has a high log likelihood value relative to the models near it in parameter -space, but where the log likelihood is significantly below the global maxima solution located somewhere else in -parameter space. - -Why does a non-linear search infer these local maxima solutions? As discussed previously, the search guesses many -models over and over, guessing more models in regions of parameter space where previous guesses gave the highest -likelihood solutions. The search gradually 'converges' around any solution that gives a higher likelihood than the -models nearby it in parameter space. If the search is not thorough enough, it may converge around a solution that -appears to give a high likelihood (compared to the models around it) but, as discussed, is only a local maxima over -all of parameter space. - -Inferring such solutions is essentially a failure of our non-linear search and it is something we do not want to -happen! Lets infer a local maxima, by reducing the number of live points, `n_live`, nautilus uses to map out -parameter space. We are going to use so few that the initial search over parameter space has an extremely low -probability of getting close the global maxima, meaning it converges on a local maxima. -""" -search = af.Nautilus( - path_prefix=Path("howtogalaxy", "chapter_2"), - name="tutorial_3_realism_and_complexity__local_maxima", - unique_tag=dataset_name, - n_live=50, - n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. -) - -print( - "The non-linear search has begun running - checkout the autogalaxy_workspace/output/3_realism_and_complexity" - " folder for live output of the results, images and model." - " This Jupyter notebook cell with progress once search has completed - this could take some time!" -) - -result_local_maxima = search.fit(model=model, analysis=analysis) - -print("Search has finished run - you may now continue the notebook.") - -""" -__Result__ - -Inspection of the `info` summary of the result suggests certain parameters have gone to different values to the fit -performed above. -""" -print(result_local_maxima.info) - -""" -Lats look at the fit to the `Imaging` data, which is clearly worse than our original fit above. -""" -aplt.subplot_fit_imaging(fit=result_local_maxima.max_log_likelihood_fit) - -""" -Finally, just to be sure we hit a local maxima, lets compare the maximum log likelihood values of the two results - -The local maxima value is significantly lower, confirming that our non-linear search simply failed to locate lens -models which fit the data better when it searched parameter space. -""" -print("Likelihood of Global Model:") -print(result.max_log_likelihood_fit.log_likelihood) -print("Likelihood of Local Model:") -print(result_local_maxima.max_log_likelihood_fit.log_likelihood) - -""" -__Wrap Up__ - -In this example, we intentionally made our non-linear search fail, by using so few live points it had no hope of -sampling parameter space thoroughly. For modeling real galaxies we wouldn't do this intentionally, but the risk of -inferring a local maxima is still very real, especially as we make our model more complex. - -Lets think about *complexity*. As we make our model more realistic, we also made it more complex. For this -tutorial, our non-linear parameter space went from 7 dimensions to 13. This means there was a much larger *volume* of -parameter space to search. As this volume grows, there becomes a higher chance that our non-linear search gets lost -and infers a local maxima, especially if we don't set it up with enough live points! - -At its core, modeling is all about learning how to get a non-linear search to find the global maxima region of -parameter space, even when the model is complex. This will be the main theme throughout the rest of this chapter -and is the main subject of chapter 3. - -In the next exercise, we'll learn how to deal with failure and begin thinking about how we can ensure our non-linear -search finds the global-maximum log likelihood solution. First, think about the following: - - 1) When you look at an image of a galaxy, do you get a sense of roughly what values certain model - parameters are? - - 2) The non-linear search failed because parameter space was too complex. Could we make it less complex, whilst - still keeping our model fairly realistic? - - 3) The galaxy in this example had only 7 non-linear parameters. Real galaxies may have multiple components (e.g. a - disk, bulge, bar, star-forming knot) and there may even be more than 1 galaxy! Do you think there is any hope of - us navigating a parameter space if the galaxies contributes 30+ parameters? -""" diff --git a/scripts/howtogalaxy/chapter_2_modeling/tutorial_4_dealing_with_failure.py b/scripts/howtogalaxy/chapter_2_modeling/tutorial_4_dealing_with_failure.py deleted file mode 100644 index 15f2217e..00000000 --- a/scripts/howtogalaxy/chapter_2_modeling/tutorial_4_dealing_with_failure.py +++ /dev/null @@ -1,442 +0,0 @@ -""" -Tutorial 4: Dealing With Failure -================================ - -In the previous tutorial we intentionally made our non-linear search infer a local maxima solution and therefore return -a physically incorrect model. In this tutorial, we will pretend that we have modeled our galaxy and inferred a local -maxima. We introduce three approaches one can take that changes how we fit the model, all of which have the aim of -ensuring we infer the global maxima: - - 1) Prior Tuning: Tell the non-linear search where to search parameter space. - 2) Reduce Complexity: Fit a model with fewer parameters and therefore a simpler parameter space. - 3) Look Harder: Brute force a global maxima by telling the non-linear search to sample parameter space more thoroughly. - -Each approach has its benefits and disadvantages and we will discuss them in detail. - -In the previous tutorial, when we inferred a local maxima we knew that we had done so. For modeling a real galaxy, -we do not know the true model and it may be unclear if the solution we inferred is a global or local maxima. The -methods we learn in this tutorial are therefore equally important for verifying that a solution that looks like a -global maxima solution is in indeed the global maxima. - -__Contents__ - -**Initial Setup:** Load the dataset and apply a mask. -**Approach 1: Prior Tuning:** Narrow the priors to guide the search to the correct region of parameter space. -**Approach 2: Reducing Complexity:** Simplify the model to reduce the dimensionality of parameter space. -**Approach 3: Look Harder:** Increase the thoroughness of the non-linear search sampling. -""" - -# from autoconf import setup_notebook; setup_notebook() - -import numpy as np -from pathlib import Path -import autogalaxy as ag -import autogalaxy.plot as aplt -import autofit as af - -""" -__Initial Setup__ - -we'll use the same galaxy data as the previous tutorial, where: - - - The galaxy's bulge is an `Sersic`. - - The galaxy's disk is an `Exponential`. -""" -dataset_name = "simple" -dataset_path = Path("dataset") / "imaging" / dataset_name - -""" -__Dataset Auto-Simulation__ - -If the dataset does not already exist on your system, it will be created by running the corresponding -simulator script. This ensures that all example scripts can be run without manually simulating data first. -""" -if not dataset_path.exists(): - import subprocess - import sys - - subprocess.run( - [sys.executable, "scripts/imaging/simulator.py"], - check=True, - ) - - -dataset = ag.Imaging.from_fits( - data_path=dataset_path / "data.fits", - noise_map_path=dataset_path / "noise_map.fits", - psf_path=dataset_path / "psf.fits", - pixel_scales=0.1, -) - -""" -__Mask__ - -we'll create and use a smaller 2.5" `Mask2D` again. -""" -mask = ag.Mask2D.circular( - shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=2.5 -) - -dataset = dataset.apply_mask(mask=mask) - -""" -__Over Sampling__ - -Apply adaptive over sampling to ensure the calculation is accurate, you can read up on over-sampling in more detail via -the `autogalaxy_workspace/*/guides/over_sampling.ipynb` notebook. -""" -over_sample_size = ag.util.over_sample.over_sample_size_via_radial_bins_from( - grid=dataset.grid, - sub_size_list=[8, 4, 1], - radial_list=[0.3, 0.6], - centre_list=[(0.0, 0.0)], -) - -dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size) - -""" -When plotted, the galaxy's bulge and disk are clearly visible in the centre of the image. -""" -aplt.subplot_imaging_dataset(dataset=dataset) - -""" -__Approach 1: Prior Tuning__ - -First, we are going to try giving our non-linear search a helping hand. Our priors tell the non-linear search where -to look in parameter space. If we tell it to look in the right place (that is, 'tune' our priors), this might mean -the search finds the global solution when it previously found a local maxima. - -We saw in a previous tutorial that we can fully customize priors in **PyAutoGalaxy**, so lets give it a go. I've set up -a custom search below and specified priors that give the non-linear search a better chance of inferring the global -maxima solution, alongside discussing how I have changed each prior from the default values specified by the -`config/priors/default` config files. - -In a previous tutorial, we customized the priors of a model by creating a `Galaxy` as a `Model` and customizing each -prior: -""" -galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=ag.lp.Sersic, disk=ag.lp.Exponential) - -galaxy.bulge.centre_0 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1) -galaxy.bulge.centre_1 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1) -galaxy.disk.centre_0 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1) -galaxy.disk.centre_1 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1) - -""" -We can alternatively create the light and bulge profiles as a `Model` and customize their parameters, and then pass them -to the model galaxy and overall model. These two approaches are equivalent, but in this example the style below -provides more concise and readable code. We will therefore switch to this code style in this tutorial, but may swap -back and forth between the two styles throughout **HowToGalaxy** depending on what is more readable. -""" -bulge = af.Model(ag.lp.Sersic) -disk = af.Model(ag.lp.Exponential) - -""" -By default, the prior on the $(y,x)$ coordinates of a `LightProfile` is a GaussianPrior with -`mean=0.0` and `sigma=0.3`. However, visual inspection of our galaxy image tells us that its centre (based on the -galaxy's luminous emission) is at x = 0.0" and y = 0.0", so lets reduce the `sigma` value on this prior so the -non-linear search looks over a very narrow range of `centre` values in parameter space. -""" -bulge.centre_0 = af.UniformPrior(lower_limit=-0.05, upper_limit=0.05) -bulge.centre_1 = af.UniformPrior(lower_limit=-0.05, upper_limit=0.05) -disk.centre_0 = af.UniformPrior(lower_limit=-0.05, upper_limit=0.05) -disk.centre_1 = af.UniformPrior(lower_limit=-0.05, upper_limit=0.05) - -""" -By default, the elliptical components of the of our galaxy's elliptical `LightProfile` are `TruncatedGaussianPrior`'s -with `mean=0.0` and `sigma=0.5`. Note that the solution `ell_comps=(0.0, 0.0)` corresponds to a spherical system -and that all physical solutions (e.g. with axis-ratios running from 0.0 -> 1.0 and position angles 0.0 -> 180.0 degrees) -are encapsulated for solutions where each component runs from -1.0 -> 1.0). - -However, through visual inspection of the image we can often determine the position angle of the galaxy's light, which -for this data is clearly 45.0 degrees counter-clockwise from the x-axis. We can update the priors on our elliptical -components to reflect this. The `lower_limit` and `upper_limit` on a `TruncatedGaussianPrior` ensure the solutions cannot go -outside the physically plausible range -1.0 -> 1.0. -""" -bulge.ell_comps.ell_comps_0 = af.TruncatedGaussianPrior( - mean=0.333333, sigma=0.1, lower_limit=-1.0, upper_limit=1.0 -) -bulge.ell_comps.ell_comps_1 = af.TruncatedGaussianPrior( - mean=0.0, sigma=0.1, lower_limit=-1.0, upper_limit=1.0 -) - -disk.ell_comps.ell_comps_0 = af.TruncatedGaussianPrior( - mean=0.333333, sigma=0.1, lower_limit=-1.0, upper_limit=1.0 -) -disk.ell_comps.ell_comps_1 = af.TruncatedGaussianPrior( - mean=0.0, sigma=0.1, lower_limit=-1.0, upper_limit=1.0 -) - -""" -The `effective_radius` of light profile is its 'half-light' radius, the radius at which 50% of its total luminosity -is internal to a circle defined within that radius. **PyAutoGalaxy** assumes a `UniformPrior` on this quantity between -0.0" and 30.0". This large range of values is required to cover the size of all possible galaxies that can be -observed in the Universe. - -However, inspection of this image shows the galaxy's light does not extend anywhere near 30.0", so lets reduce its -value for both bulge and disk components. - -We retain an `upper_limit=np.inf`, which means that the maximum physical value that the effective radius -can go to is infinity, as it is in the real Universe. Equally, the `lower_limit=0.0` ensures unphysical negative -values are not sampled. - -However, the inputs `mean=1.0` and `sigma=0.8` ensure the majority of samples are drawn around much lower values -between 0.0" -> 2.0". -""" -bulge.effective_radius = af.TruncatedGaussianPrior( - mean=1.0, sigma=0.8, lower_limit=0.0, upper_limit=np.inf -) -disk.effective_radius = af.TruncatedGaussianPrior( - mean=1.0, sigma=0.8, lower_limit=0.0, upper_limit=np.inf -) - -""" -The `sersic_index` defines how concentrated the light profile is. In galaxy structure studies, values of Sersic index -around 1.0 indicate a disk galaxy (which is the value the `Exponential` uses). - -Higher values of 3 or 4 indicate an elliptical galaxy. **PyAutoGalaxy** assumes a `UniformPrior` between 0.8 and 8.0 -by default on this parameter, as a user could model galaxies -where the galaxy is of any morphology. - -We are assuming the `bulge` component is a bulge, thus we can change its prior on the `sersic_index` to a value near 3. -""" -bulge.sersic_index = af.TruncatedGaussianPrior( - mean=3.0, sigma=1.0, lower_limit=0.0, upper_limit=np.inf -) - -""" -We now compose the overall model, where the galaxy model uses the `Model` components above which had their -priors customizes. - -In this exercise, I'm not going to change any priors on the galaxy. Whilst modeling experts can look at a -galaxy and often tell you roughly where the galaxy is located, it is something of art -form. Furthermore, the source's morphology can be pretty complex, making it difficult to come up with a good source -prior! -""" -galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=bulge, disk=disk) - -model = af.Collection(galaxies=af.Collection(galaxy=galaxy)) - -""" -The `info` attribute shows the model, including the priors specified above. -""" -print(model.info) - -""" -We can now create this custom search and run it. Our non-linear search will now start by sampling higher likelihood -regions of parameter space, given our improved and more informed priors. -""" -search = af.Nautilus( - path_prefix=Path("howtogalaxy", "chapter_2"), - name="tutorial_4_custom_priors", - unique_tag=dataset_name, - n_live=100, - n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. -) - -analysis = ag.AnalysisImaging(dataset=dataset, use_jax=True) - -print( - "The non-linear search has begun running - checkout the workspace/output/howtogalaxy/chapter_2/tutorial_4_custom_priors" - " folder for live output of the results, images and model." - " This Jupyter notebook cell with progress once search has completed - this could take some time!" -) - -result_custom_priors = search.fit(model=model, analysis=analysis) - -print("Search has finished run - you may now continue the notebook.") - -print(result_custom_priors.info) - -""" -__Result__ - -Bam! We get a good model, which corresponds to the global maxima. By giving our non-linear search a helping hand and -informing it of where to sample parameter space, we can increase the odds that we find the global maxima solution. -""" -aplt.subplot_fit_imaging(fit=result_custom_priors.max_log_likelihood_fit) - -""" -__Discussion__ - -By tuning our priors to the galaxy we fit we increase the chance of inferring the global maxima model. The search -may also fit the model a lot faster, given it spends less time searches regions of parameter space that do not -correspond to good solutions. - -Before moving onto the next approach, lets think about the advantages and disadvantages of prior tuning: - -Advantages: - - - We have a higher chance of finding the globally maximum log likelihood solutions in parameter space. - - The search took less time to run because the non-linear search explored less of parameter space. - -Disadvantages: - - - If we specified a prior incorrectly the non-linear search will infer an incorrect solution. - - The priors for the search were tailored to the specific galaxy we fitted. If we are fitting multiple galaxies, - we would have customize the priors for every single fit, for large samples of galaxies this would take a lot of time! - -__Approach 2: Reducing Complexity__ - -The non-linear search may fail because the model is too complex, making its parameter space too difficult to -sample accurately. Can we can make the model less complex, whilst keeping it realistic enough to perform our -scientific study? What assumptions can we make to reduce the number of model parameters and therefore -dimensionality of non-linear parameter space? -""" -bulge = af.Model(ag.lp.Sersic) -disk = af.Model(ag.lp.Exponential) - -""" -First, we create a model that assumes that the bulge and disk are geometrically aligned. That is, the bulge and -disk centres and elliptical components are perfectly aligned with one another. This may, or may -not, be a reasonable assumption, but it`ll remove 4 parameters from the model (the centre and elliptical -components of the bulge profile), so it is worth trying! - -To apply our assumption that the bulge and disk are geometrically aligned, we `pair` the `centre` and `ell_comps` -parameters by setting them equal to one another. This removes the parameter on the left-hand side of the pairing from -the galaxy model such that when a model is created it has the same value as the parameter on the right-hand side. -""" -bulge.centre = disk.centre -bulge.ell_comps = disk.ell_comps - -""" -We now compose the model, which will have a non-linear parameter space with 4 less dimensions than the fit performed -previously. -""" -galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=bulge, disk=disk) - -model = af.Collection(galaxies=af.Collection(galaxy=galaxy)) - -""" -The `info` attribute shows the model, including the parameter linking specified above. -""" -print(model.info) - -""" -We now create this search and run it. -""" -search = af.Nautilus( - path_prefix=Path("howtogalaxy", "chapter_2"), - name="tutorial_4_reducing_complexity", - unique_tag=dataset_name, - n_live=100, - n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. -) - -print( - "The non-linear search has begun running - checkout the workspace/output/howtogalaxy/chapter_2/tutorial_4_reducing_complexity" - " folder for live output of the results, images and model." - " This Jupyter notebook cell with progress once search has completed - this could take some time!" -) - -result_bulge_disk_align = search.fit(model=model, analysis=analysis) - -print("Search has finished run - you may now continue the notebook.") - -print(result_bulge_disk_align.info) - -aplt.subplot_fit_imaging(fit=result_bulge_disk_align.max_log_likelihood_fit) - -""" -__Result__ - -The results look pretty good. Our galaxy fits the data pretty well and we've clearly inferred a model that looks -similar to the one above. However, inspection of the residuals shows that the fit was not quite as good as the -first search. - -It turns out that for this simulation, the bulge and disk had different elliptical components. The quality of the fit -suffered and the highest value of log likelihood for the search inferred was lower as a result. - -Herein lies the pitfalls of making assumptions, they may make your model less realistic and your fits worse! - -__Discussion__ - -Again, lets consider the advantages and disadvantages of this approach: - -Advantages: - - - By reducing parameter space`s complexity we again had a higher chance of inferring the global maximum log - likelihood and the time required by the search to do this is reducing. - - Unlike tuned priors, the search was not specific to one galaxy and we could run it on many galaxy images. - -Disadvantages: - - - Our model was less realistic and our fit suffered as a result. - -__Approach 3: Look Harder__ - -In approaches 1 and 2 we extended our non-linear search an olive branch and helped it find the highest log likelihood -regions of parameter space. In approach 3 ,we're going to tell it to just `look harder`. - -Every non-linear search has settings which govern how thoroughly it searches parameter space, with the number of live -points that was passed to `Nautilus` an example of such a setting. The more thoroughly the search looks, the more likely -it is that it`ll find the global maximum model. However, the search will also take longer! - -We create a more thorough `nautilus` search, that uses `n_live=200`. What these settings -are actually changing is discussed in the optional tutorial `HowToGalaxy/chapter_optional/tutorial_searches.ipynb`. - -Due to the long run times of this search, we comment it output below so it does not run. Feel free to undo these -comments so the script runs faster. -""" -galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=ag.lp.Sersic, disk=ag.lp.Exponential) - -model = af.Collection(galaxies=af.Collection(galaxy=galaxy)) - -search = af.Nautilus( - path_prefix=Path("howtogalaxy", "chapter_2"), - name="tutorial_4_look_harder", - unique_tag=dataset_name, - n_live=200, - n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. -) - -print( - "The non-linear search has begun running - checkout the workspace/output/howtogalaxy/chapter_2/tutorial_4_look_harder" - " folder for live output of the results, images and model." - " This Jupyter notebook cell with progress once search has completed - this could take some time!" -) - -# result_look_harder = search.fit(model=model, analysis=analysis) - -print("Search has finished run - you may now continue the notebook.") - -# print(result_look_harder.info) - -# aplt.subplot_fit_imaging(fit=result_look_harder.max_log_likelihood_fit) - -""" -lets list the advantages and disadvantages of simply adjusting the non-linear search: - -Advantages: - - - Its easy to setup, we simply change settings of the non-linear search. - - - It generalizes to any galaxy. - - - We can make our model as complex as we want. - -Disadvantage: - - - Its potentially expensive. Very expensive. For very complex models, the run times can hours, days, weeks or, dare - I say it, months! - -So, we can now fit galaxies. And when it fails, we know how to get it to work. - -In chapter 3 of **HowToGalaxy**, we will introduce a technique called 'non-linear search chaining', which performs a -model fit by chaining together multiple searches back-to-back . This allows us to combine the 3 different approaches -discussed and exploit the advantages of each, whilst not being hindered by their disadvantages. - -With search chaining, we can: - - - Fit simpler models with lower dimensionality parameter spaces in the earlier searches and gradually increase the - model complexity search-by-search, guiding the model-fit to a sufficiently realistic model. - - - In these earlier searches (with easier to sample parameter spaces), use fast non-linear search settings to compute - the results quickly and switch to slower settings in later searches when we fit more complex models. - - - Use 'prior passing' to setup the priors of each parameter in the later searches, based on the models inferred - by the earlier searches. We can therefore guide each search on how to sample a complex model's parameter space - in a way that can be fully generalized to any galaxy. - -To wrap up chapter 2, we have a few more tutorials, where we will discuss masking in more detail, the `Result` object -and how to make **PyAutoGalaxy** run faster. -""" diff --git a/scripts/howtogalaxy/chapter_2_modeling/tutorial_5_linear_profiles.py b/scripts/howtogalaxy/chapter_2_modeling/tutorial_5_linear_profiles.py deleted file mode 100644 index a70370e7..00000000 --- a/scripts/howtogalaxy/chapter_2_modeling/tutorial_5_linear_profiles.py +++ /dev/null @@ -1,544 +0,0 @@ -""" -Tutorial 5: Linear Profiles -=========================== - -In the previous tutorial we learned how to balance model complexity with our non-linear search in order to infer -accurate model solutions and avoid failure. We saw how in order to fit a model accurately one may have to -parameterize and fit a simpler model with fewer non-linear parameters, at the expense of fitting the data less -accurately. - -It would be desirable if we could make our model have more flexibility enabling it to fit more complex galaxy -structures, but in a way that does not increase (or perhaps even decreases) the number of non-linear parameters. -This would keep the `nautilus` model-fit efficient and accurate. - -This is possible using linear light profiles, which solve for their `intensity` parameter via efficient linear -algebra, using a process called an inversion. The inversion always computes `intensity` values that give the best -fit to the data (e.g. they minimize the chi-squared and therefore maximize the likelihood). - -This tutorial will first fit a model using two linear light profiles. Because their `intensity` values are solved for -implicitly, this means they are not a dimension of the non-linear parameter space fitted by `nautilus`, therefore -reducing the complexity of parameter space and making the fit faster and more accurate. - -This tutorial will then show how many linear light profiles can be combined into a `Basis`, which comes from the term -'basis function'. By combining many linear light profiles models can be composed which are able to fit complex galaxy -structures (e.g. asymmetries, twists) with just N=6-8 non-linear parameters. - -__Contents__ - -**Initial Setup:** Load the dataset and apply a mask. -**Linear Light Profiles:** Fit a model using linear light profiles that solve for intensity via linear algebra. -**Run Time:** Estimate the run time of the model-fit before starting. -**Result:** Inspect the result and the solved intensity values. -**Intensities:** Access the intensity values of linear light profiles after the fit. -**Visualization:** Visualize the fit using linear light profiles. -**Basis:** Combine many linear light profiles into a basis for fitting complex structures. -**Model Fit:** Fit the basis model to the data. -**Disk MGE:** Use a Multi-Gaussian Expansion for the disk component. -**Multi Gaussian Expansion Benefits:** Discussion of the advantages of MGE models. -**Positive Only Solver:** Ensure linear algebra solutions have positive intensity values. -**Other Basis Functions:** Overview of other basis functions like shapelets. -**Wrap Up:** Summary of linear light profiles and basis functions. -""" - -# from autoconf import setup_notebook; setup_notebook() - -import numpy as np -from pathlib import Path -import autogalaxy as ag -import autogalaxy.plot as aplt -import autofit as af - -""" -__Initial Setup__ - -we'll use the same galaxy data as the previous tutorial, where: - - - The galaxy's bulge is an `Sersic`. - - The galaxy's disk is an `Exponential`. -""" -dataset_name = "simple" -dataset_path = Path("dataset") / "imaging" / dataset_name - -""" -__Dataset Auto-Simulation__ - -If the dataset does not already exist on your system, it will be created by running the corresponding -simulator script. This ensures that all example scripts can be run without manually simulating data first. -""" -if not dataset_path.exists(): - import subprocess - import sys - - subprocess.run( - [sys.executable, "scripts/imaging/simulator.py"], - check=True, - ) - - -dataset = ag.Imaging.from_fits( - data_path=dataset_path / "data.fits", - noise_map_path=dataset_path / "noise_map.fits", - psf_path=dataset_path / "psf.fits", - pixel_scales=0.1, -) - -""" -__Mask__ - -we'll create and use a smaller 2.5" `Mask2D` again. -""" -mask = ag.Mask2D.circular( - shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=2.5 -) - -dataset = dataset.apply_mask(mask=mask) - -""" -__Over Sampling__ - -Apply adaptive over sampling to ensure the calculation is accurate, you can read up on over-sampling in more detail via -the `autogalaxy_workspace/*/guides/over_sampling.ipynb` notebook. -""" -over_sample_size = ag.util.over_sample.over_sample_size_via_radial_bins_from( - grid=dataset.grid, - sub_size_list=[8, 4, 1], - radial_list=[0.3, 0.6], - centre_list=[(0.0, 0.0)], -) - -dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size) - -""" -When plotted, the galaxy's bulge and disk are clearly visible in the centre of the image. -""" -aplt.subplot_imaging_dataset(dataset=dataset) - -""" -__Linear Light Profiles__ - -First, we use a variant of a light profile discussed called a "linear light profile", which is accessed via the -command `ag.lp_linear`. - -The `intensity` values of linear light profiles are solved for via linear algebra. We use the `Sersic` -and `Exponential` linear light profiles, which are identical to the ordinary `Sersic` and `Exponential` -profiles fitted in previous tutorials, except for their `intensity` parameter now being solved for implicitly. - -Because the `intensity` parameter of each light profile is not a free parameter in the model-fit, the dimensionality of -non-linear parameter space is reduced by 1 for each light profile (in this example, 2). This also removes the -degeneracies between the `intensity` and other light profile parameters (e.g. `effective_radius`, `sersic_index`), -making the model-fit more robust. - -This is a rare example where we are able to reduce the complexity of parameter space without making the model itself -any simpler. There is really no downside to using linear light profiles, so I would recommend you adopt them as -standard for your own model-fits from here on! -""" -bulge = af.Model(ag.lp_linear.Sersic) -disk = af.Model(ag.lp_linear.Exponential) - -bulge.centre = disk.centre - -galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=bulge, disk=disk) - -model = af.Collection(galaxies=af.Collection(galaxy=galaxy)) - -""" -The `info` attribute shows the model, including the linear light profiles. - -Note how the `intensity` is no longer listed and does not have a prior associated with it. -""" -print(model.info) - -""" -We now create this search and run it. -""" -search = af.Nautilus( - path_prefix=Path("howtogalaxy", "chapter_2"), - name="tutorial_5_linear_light_profile", - unique_tag=dataset_name, - n_live=100, - n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. -) - -analysis = ag.AnalysisImaging(dataset=dataset, use_jax=True) - -""" -__Run Time__ - -For standard light profiles, the log likelihood evaluation time is of order ~0.01 seconds for this dataset. - -For linear light profiles, the log likelihood evaluation increases to around ~0.05 seconds per likelihood evaluation. -This is still fast, but it does mean that the fit may take around five times longer to run. - -However, because two free parameters have been removed from the model (the `intensity` of the lens bulge and -source bulge), the total number of likelihood evaluations will reduce. Furthermore, the simpler parameter space -likely means that the fit will take less than 10000 per free parameter to converge. This is aided further -by the reduction in `n_live` to 100. - -Fits using standard light profiles and linear light profiles therefore take roughly the same time to run. However, -the simpler parameter space of linear light profiles means that the model-fit is more reliable, less susceptible to -converging to an incorrect solution and scales better if even more light profiles are included in the model. - -Run the non-linear search. -""" -print( - "The non-linear search has begun running - checkout the workspace/output/howtogalaxy/chapter_2/tutorial_5_linear_light_profile" - " folder for live output of the results, images and model." - " This Jupyter notebook cell with progress once search has completed - this could take some time!" -) - -result_linear_light_profile = search.fit(model=model, analysis=analysis) - -""" -__Result__ - -The `info` attribute shows the resulting model, which does not display the `intensity` values for each light profile. -""" -print(result_linear_light_profile.info) - -""" -__Intensities__ - -The intensities of linear light profiles are not a part of the model parameterization and therefore are not displayed -in the `model.results` file. - -To extract the `intensity` values of a specific component in the model, we use the `max_log_likelihood_galaxies`, -which has already performed the inversion and therefore the galaxy light profiles have their solved for -`intensity`'s associated with them. -""" -galaxies = result_linear_light_profile.max_log_likelihood_galaxies - -print(galaxies[0].bulge.intensity) - -""" -The `Galaxies` contained in the `max_log_likelihood_fit` also has the solved for `intensity` values: -""" -fit = result_linear_light_profile.max_log_likelihood_fit - -galaxies = fit.galaxies - -print(galaxies[0].bulge.intensity) - -""" -__Visualization__ - -Linear light profiles and objects containing them (e.g. galaxies) cannot be plotted because they do not -have an `intensity` value. - -Therefore, the objects created above which replaces all linear light profiles with ordinary light profiles must be -used for visualization: -""" -galaxies = result_linear_light_profile.max_log_likelihood_galaxies - -aplt.plot_array(array=galaxies.image_2d_from(grid=dataset.grid), title="Image") - -aplt.plot_array(array=galaxies[0].image_2d_from(grid=dataset.grid), title="Image") - -""" -__Basis__ - -We can use many linear light profiles to build a `Basis`. - -For example, below, we make a `Basis` out of 30 elliptical Gaussian linear light profiles which: - - - All share the same centre and elliptical components. - - The `sigma` size of the Gaussians increases in log10 increments. - -Because `log10(1.0) = 0.0` the first Gaussian `sigma` value is therefore 0.0001, whereas because `log10(10) = 1.0` -the size of the final Gaussian is 1.0. - -The equation below has therefore been chosen to provide intuition on the scale of the Gaussians. -""" -total_gaussians = 30 - -# The sigma values of the Gaussians will be fixed to values spanning 0.01 to the mask radius, 3.0". -mask_radius = 3.0 -log10_sigma_list = np.linspace(-2, np.log10(mask_radius), total_gaussians) - -# By defining the centre here, it creates two free parameters that are assigned below to all Gaussians. - -centre_0 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1) -centre_1 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1) - -bulge_gaussian_list = [] - -# A list of Gaussian model components whose parameters are customized belows. - -gaussian_list = af.Collection( - af.Model(ag.lp_linear.Gaussian) for _ in range(total_gaussians) -) - -# Iterate over every Gaussian and customize its parameters. - -for i, gaussian in enumerate(gaussian_list): - gaussian.centre.centre_0 = centre_0 # All Gaussians have same y centre. - gaussian.centre.centre_1 = centre_1 # All Gaussians have same x centre. - gaussian.ell_comps = gaussian_list[ - 0 - ].ell_comps # All Gaussians have same elliptical components. - gaussian.sigma = ( - 10 ** log10_sigma_list[i] - ) # All Gaussian sigmas are fixed to values above. - -bulge_gaussian_list += gaussian_list - -# The Basis object groups many light profiles together into a single model component. - -bulge = af.Model( - ag.lp_basis.Basis, - profile_list=bulge_gaussian_list, -) - -""" -One we have a `Basis`, we can treat it like any other light profile in order to create a `Galaxy` and `Galaxies` and -use it to fit data. -""" -galaxy = ag.Galaxy(redshift=0.5, bulge=bulge) - -galaxies = ag.Galaxies(galaxies=[galaxy]) - -fit = ag.FitImaging(dataset=dataset, galaxies=galaxies) - -""" -By plotting the fit, we see that the `Basis` does a reasonable job at capturing the appearance of the lens galaxy. - -There are imperfections, but this is because we did not fit the model via a non-linear search in order to determine -the optimal values of the Gaussians in the basis. In particular, the Gaussians above were all spherical, when the -lens galaxy is elliptical. - -We rectify this below, where we use a non-linear search to determine the optimal values of the Gaussians! -""" -aplt.subplot_fit_imaging(fit=fit) - -""" -__Model Fit__ - -To fit a model using `Basis` functions, the API is very similar to that shown throughout this chapter, using both -the `af.Model()` and `af.Collection()` objects. - -In this example we fit a `Basis` model for the bulge where: - - - The bulge is a superposition of 30 parametric linear `Gaussian` profiles [4 parameters]. - - The centres and elliptical components of each family of Gaussians are all linked together. - - The `sigma` size of the Gaussians increases in log10 increments. - -The number of free parameters and therefore the dimensionality of the MGe is just N=4. -""" -total_gaussians = 30 - -# The sigma values of the Gaussians will be fixed to values spanning 0.01 to the mask radius, 3.0". -mask_radius = 3.0 -log10_sigma_list = np.linspace(-2, np.log10(mask_radius), total_gaussians) - -# By defining the centre here, it creates two free parameters that are assigned below to all Gaussians. - -centre_0 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1) -centre_1 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1) - -bulge_gaussian_list = [] - -# A list of Gaussian model components whose parameters are customized belows. - -gaussian_list = af.Collection( - af.Model(ag.lp_linear.Gaussian) for _ in range(total_gaussians) -) - -# Iterate over every Gaussian and customize its parameters. - -for i, gaussian in enumerate(gaussian_list): - gaussian.centre.centre_0 = centre_0 # All Gaussians have same y centre. - gaussian.centre.centre_1 = centre_1 # All Gaussians have same x centre. - gaussian.ell_comps = gaussian_list[ - 0 - ].ell_comps # All Gaussians have same elliptical components. - gaussian.sigma = ( - 10 ** log10_sigma_list[i] - ) # All Gaussian sigmas are fixed to values above. - -bulge_gaussian_list += gaussian_list - -# The Basis object groups many light profiles together into a single model component. - -bulge = af.Model( - ag.lp_basis.Basis, - profile_list=bulge_gaussian_list, -) - -""" -__Disk MGE__ - -The residuals of the fit above showed us that the galaxy in the data is composed of multiple structures (e.g. a bulge -and disk) which have distinct elliptical coordinates. - -We therefore compose a second `Basis` of 10 Gaussians to represent the `disk`. This is parameterized the same as -the `bulge` (e.g. all Gaussians share the same `centre` and `ell_comps`) but is treated as a completely -independent set of parameters. -""" -total_gaussians = 10 - -# The sigma values of the Gaussians will be fixed to values spanning 0.01 to the mask radius, 3.0". -mask_radius = 3.0 -log10_sigma_list = np.linspace(-2, np.log10(mask_radius), total_gaussians) - -disk_gaussian_list = [] - -# A list of Gaussian model components whose parameters are customized belows. - -gaussian_list = af.Collection( - af.Model(ag.lp_linear.Gaussian) for _ in range(total_gaussians) -) - -# Iterate over every Gaussian and customize its parameters. - -for i, gaussian in enumerate(gaussian_list): - gaussian.centre.centre_0 = bulge_gaussian_list[ - 0 - ].centre_0 # All Gaussians have same y centre as bulge. - gaussian.centre.centre_1 = bulge_gaussian_list[ - 0 - ].centre_1 # All Gaussians have same x centre as bulge. - gaussian.ell_comps = gaussian_list[ - 0 - ].ell_comps # All Gaussians have same elliptical components. - gaussian.sigma = ( - 10 ** log10_sigma_list[i] - ) # All Gaussian sigmas are fixed to values above. - -disk_gaussian_list += gaussian_list - -# The Basis object groups many light profiles together into a single model component. - -disk = af.Model( - ag.lp_basis.Basis, - profile_list=disk_gaussian_list, -) - -""" -We now compose the overall model which uses both sets of Gaussians to represent separately the bulge and disk. - -The overall dimensionality of non-linear parameter space is just N=6, which is fairly remarkable if you -think about just how complex the structures are that these two `Basis` of Gaussians can capture! -""" -galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=bulge, disk=disk) - -model = af.Collection(galaxies=af.Collection(galaxy=galaxy)) - - -""" -The `info` attribute shows the model, which is a lot longer than we have seen previously, given that is -composed of many Gaussians! -""" -print(model.info) - -""" -We now fit the model, with just `n_live=50` given the simiplicity of parameter space. -""" -search = af.Nautilus( - path_prefix=Path("howtogalaxy", "chapter_2"), - name="tutorial_5_basis", - unique_tag=dataset_name, - n_live=50, - n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. -) - -print( - "The non-linear search has begun running - checkout the workspace/output/howtogalaxy/chapter_2/tutorial_5_basis" - " folder for live output of the results, images and model." - " This Jupyter notebook cell with progress once search has completed - this could take some time!" -) - -result_basis = search.fit(model=model, analysis=analysis) - -""" -__Result__ - -The result `info` attribute shows the result, which is again longer than usual given the large number of Gaussians -used in the fit. -""" -print(result_basis.info) - -""" -Visualizing the fit shows that we successfully fit the data to the noise level. - -Note that the result objects `max_log_likelihood_galaxies` and `max_log_likelihood_fit` automatically convert -all linear light profiles to ordinary light profiles, including every single one of the 20 Gaussians fitted -above. - -This means we can use them directly to perform the visualization below. -""" -print(result_basis.max_log_likelihood_instance) - -aplt.subplot_galaxies( - galaxies=result_basis.max_log_likelihood_galaxies, grid=result_basis.grids.lp -) - -aplt.subplot_fit_imaging(fit=result_basis.max_log_likelihood_fit) - -""" -__Multi Gaussian Expansion Benefits__ - -Symmetric light profiles (e.g. elliptical Sersics) may leave significant residuals, because they fail to capture -irregular and asymmetric morphological of galaxies (e.g. isophotal twists, an ellipticity which varies radially). -An MGE fully captures these features and can therefore much better represent the emission of complex galaxies. - -The MGE model can be composed in a way that has fewer non-linear parameters than an elliptical Sersic. In this example, -a groups of Gaussians is used to represent the `bulge` of the galaxy, which in total correspond to just N=4 non-linear -parameters (a `bulge` and `disk` comprising two linear Sersics has N=10 parameters). - -The MGE model parameterization is also composed such that neither the `intensity` parameters or any of the -parameters controlling the size of the Gaussians (their `sigma` values) are non-linear parameters sampled by Nautilus. -This removes the most significant degeneracies in parameter space, making the model much more reliable and efficient -to fit. - -Therefore, not only does an MGE fit more complex galaxy morphologies, it does so using fewer non-linear parameters -in a much simpler non-linear parameter space which has far less significant parameter degeneracies! - -__Positive Only Solver__ - -Many codes which use linear algebra typically rely on a linear algabra solver which allows for positive and negative -values of the solution (e.g. `np.linalg.solve`), because they are computationally fast. - -This is problematic, as it means that negative surface brightnesses values can be computed to represent a galaxy's -light, which is clearly unphysical. For an MGE, this produces a positive-negative "ringing", where the -Gaussians alternate between large positive and negative values. This is clearly undesirable and unphysical. - -**PyAutoGalaxy** (and therefore all examples above) uses a positive only linear algebra solver which has been -extensively optimized to ensure it is as fast as positive-negative solvers. This ensures that all light profile -intensities are positive and therefore physical. - -__Other Basis Functions__ - -In addition to the Gaussians used in this example, there is another basis function implemented in PyAutoGalaxy -that is commonly used to represent the light of galaxies, called a `Shapelet`. - -Shapelets are basis functions with analytic properties that are appropriate for capturing the exponential / disk-like -features of a galaxy. They do so over a wide range of scales, and can often represent features in source galaxies -that a single Sersic function or MGE cannot. - -An example using shapelets is given at `autogalaxy_workspace/scripts/modeling/imaging/features/shapelets.py`. - -Feel free to experiment with using shapelets as the galaxy by yourself. However they incur higher computational -overheads than the MGE and include a free parameter which governs the size of the basis functions and therefore source, -slowing down convergence of the non-linear search. We have found that MGEs perform better than shapelets in most -lens modeling problems. - -If you have a desire to fit sources with even more complex morphologies we recommend you look at how to reconstruct -sources using pixelizations in the `modeling/features` section or chapter 4 of **HowToGalaxy**. - -__Wrap Up__ - -In this tutorial we described how linearizing light profiles allows us to fit more complex light profiles to -galaxies using fewer non-linear parameters, keeping the fit performed by the non-linear search fast, accurate -and robust. - -Perhaps the biggest downside to basis functions is that they are only as good as the features they can capture -in the data. For example, a baiss of Gaussians still assumes that they have a well defined centre, but there are -galaxies which may have multiple components with multiple centres (e.g. many star forming knots) which such a -basis cannot catprue. - -In chapter 4 of **HowToGalaxy** we introduce non-parametric pixelizations, which reconstruct the data in way -that does not make assumptions like a centre and can thus reconstruct even more complex, asymmetric and irregular -galaxy morphologies. -""" -basis = ag.lp_basis.Basis( - profile_list=gaussian_list, regularization=ag.reg.Constant(coefficient=1.0) -) diff --git a/scripts/howtogalaxy/chapter_2_modeling/tutorial_6_masking.py b/scripts/howtogalaxy/chapter_2_modeling/tutorial_6_masking.py deleted file mode 100644 index 7ae0c276..00000000 --- a/scripts/howtogalaxy/chapter_2_modeling/tutorial_6_masking.py +++ /dev/null @@ -1,154 +0,0 @@ -""" -Tutorial 6: Masking -=================== - -We have learnt everything we need to know about non-linear searches to model a galaxy and infer a good lens -model solution. Now, lets consider masking in more detail, something we have not given much consideration previously. -We'll also learn a neat trick to improve the speed and accuracy of a non-linear search. - -__Contents__ - -**Initial Setup:** Load the dataset and apply a mask. -**Mask:** Apply a custom mask to focus the fit on specific regions. -**Model + Search + Analysis:** Fit the model using the custom mask. -**Discussion:** How the mask affects the fit quality and run time. -**Wrap Up:** Summary of masking strategies for galaxy modeling. -""" - -# from autoconf import setup_notebook; setup_notebook() - -from pathlib import Path -import autogalaxy as ag -import autogalaxy.plot as aplt -import autofit as af - -""" -__Initial Setup__ - -we'll use the same galaxy data as tutorials 1 & 2, where: - - - The galaxy's `LightProfile` is an `Sersic`. -""" -dataset_name = "simple__sersic" -dataset_path = Path("dataset") / "imaging" / dataset_name - -""" -__Dataset Auto-Simulation__ - -If the dataset does not already exist on your system, it will be created by running the corresponding -simulator script. This ensures that all example scripts can be run without manually simulating data first. -""" -if not dataset_path.exists(): - import subprocess - import sys - - subprocess.run( - [sys.executable, "scripts/howtogalaxy/simulators/sersic.py"], - check=True, - ) - - -dataset = ag.Imaging.from_fits( - data_path=dataset_path / "data.fits", - noise_map_path=dataset_path / "noise_map.fits", - psf_path=dataset_path / "psf.fits", - pixel_scales=0.1, -) - -""" -__Mask__ - -In tutorials 1 and 2 we used a 3.0" circular mask. - -However, there is very faint flux emitted at the outskirts of the galaxy, which the model will benefit from fitting -by using a larger mask. -""" -mask = ag.Mask2D.circular( - shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=6.0 -) - -dataset = dataset.apply_mask(mask=mask) - -aplt.subplot_imaging_dataset(dataset=dataset) - -""" -__Over Sampling__ - -Apply adaptive over sampling to ensure the calculation is accurate, you can read up on over-sampling in more detail via -the `autogalaxy_workspace/*/guides/over_sampling.ipynb` notebook. -""" -over_sample_size = ag.util.over_sample.over_sample_size_via_radial_bins_from( - grid=dataset.grid, - sub_size_list=[8, 4, 1], - radial_list=[0.3, 0.6], - centre_list=[(0.0, 0.0)], -) - -dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size) - -""" -__Model + Search + Analysis__ - -Lets fit the data using this mask, by creating the search as per usuag. Note that the `imaging` data with this mask -applied is passed into the `AnalysisImaging` object, ensuring that this is the mask the model-fit uses. -""" -galaxy = af.Model(ag.Galaxy, redshift=1.0, bulge=ag.lp.Sersic) -model = af.Collection(galaxies=af.Collection(galaxy=galaxy)) - -search = af.Nautilus( - path_prefix=Path("howtogalaxy", "chapter_2"), - name="tutorial_6_with_custom_mask", - unique_tag=dataset_name, - n_live=80, - n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. -) - -analysis = ag.AnalysisImaging(dataset=dataset, use_jax=True) - -search.fit(model=model, analysis=analysis) - -""" -__Discussion__ - -So, we can choose the mask we use in a model-fit. We know that we want the mask remove as little of the galaxy's light, -but is this the 'right' mask? What is the 'right' mask? Maybe we want a bigger mask? a smaller mask? - -When it comes to choosing a mask, we are essentially balancing two things: computational run-time and accuracy. When we -use a bigger mask the model-fit will take longer to perform. Why? Because a bigger mask includes more image-pixels -in the analysis, and for every additional image-pixel we have to compute its light, blur it with the PSF, compare -it to the data, etc. - -If run-time was not a consideration we would always choose a bigger mask, for two reasons: - - 1) The galaxy may have very faint emission that when you choose the mask you simply do not notice. Overly aggressive - masking runs the risk of us inadvertantly masking out some of the galaxy's light, which would otherwise better - constrain the model! - - 2) When the data is fitted with a model image, the fit is performed only within the masked region. For certain galaxies - it is possible that it may produce extraneous emission outside of the masked region that is not actually observed in - the data itself. If this region had not been masked-out, the model would create residuals in these locations and - reduce the value of likelihood appropriately, whereas if it is masked out this reduction in likelihood is - not fed through to the analysis. - -As you use **PyAutoGalaxy** more you will get a feel for how fast a model-fit will run given the quality of data, -model complexity, non-linear search settings, etc. As you develop this intuition, I recommend that you always aim to -use as large of a mask as possible (whilst still achieving reasonable run-times). Aggressive masking will make -**PyAutoGalaxy** run very fast, but could lead you to infer an incorrect model! - -In chapter 3, where we introduce 'non-linear search chaining' we will see how we can use tighter masks in earlier -searches to achieve faster run times. - -If your data includes the light of additional galaxies nearby you may much have no choice but to use a smaller -circular mask, because it is important these objects do not interfere with the fit. - -In fact, you can drawcustom masks that remove their light entirely. You may now wish to checkout -the `autogalaxy_workspace/*/imaging/preprocess` package. This includes tools for creating custom masks and -marking the positions on a galaxy (via a GUI) so you can use them in a model-fit. - -__Wrap Up__ - -There are is one thing you should bare in mind in terms of masking: - - 1) Customizing the mask for the analysis of one galaxy gets the analysis running fast and can provide accurate - non-linear sampling. However, for a large sample of galaxies, this high level of customization may take a lot of time. -""" diff --git a/scripts/howtogalaxy/chapter_2_modeling/tutorial_7_results.py b/scripts/howtogalaxy/chapter_2_modeling/tutorial_7_results.py deleted file mode 100644 index 71d49ec2..00000000 --- a/scripts/howtogalaxy/chapter_2_modeling/tutorial_7_results.py +++ /dev/null @@ -1,163 +0,0 @@ -""" -Tutorial 5: Results -=================== - -In the previous tutorials, each search returned a `Result` object, which we used to plot the maximum log likelihood -fit each model-fit. In this tutorial, we'll take a look at the result object in a little more detail. - -__Contents__ - -**Initial Setup:** Perform a model-fit to obtain a Result object. -**Galaxies & Fit:** Access the maximum likelihood galaxies and fit from the result. -**Samples:** Inspect the non-linear search samples, including parameter estimates and errors. -**Workspace:** Pointers to more detailed results examples in the workspace. -**Database:** Overview of the database functionality for managing large numbers of results. -**Wrap Up:** Summary of the Result object and its key attributes. -""" - -# from autoconf import setup_notebook; setup_notebook() - -from pathlib import Path -import autogalaxy as ag -import autogalaxy.plot as aplt -import autofit as af - -""" -__Initial Setup__ - -Lets use the model-fit performed in tutorial 1 to get a `Result` object. -""" -dataset_name = "simple__sersic" -dataset_path = Path("dataset") / "imaging" / dataset_name - -""" -__Dataset Auto-Simulation__ - -If the dataset does not already exist on your system, it will be created by running the corresponding -simulator script. This ensures that all example scripts can be run without manually simulating data first. -""" -if not dataset_path.exists(): - import subprocess - import sys - - subprocess.run( - [sys.executable, "scripts/howtogalaxy/simulators/sersic.py"], - check=True, - ) - - -dataset = ag.Imaging.from_fits( - data_path=dataset_path / "data.fits", - noise_map_path=dataset_path / "noise_map.fits", - psf_path=dataset_path / "psf.fits", - pixel_scales=0.1, -) - -mask = ag.Mask2D.circular( - shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=3.0 -) - -dataset = dataset.apply_mask(mask=mask) - -over_sample_size = ag.util.over_sample.over_sample_size_via_radial_bins_from( - grid=dataset.grid, - sub_size_list=[8, 4, 1], - radial_list=[0.3, 0.6], - centre_list=[(0.0, 0.0)], -) - -dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size) - -model = af.Collection( - galaxies=af.Collection(galaxy=af.Model(ag.Galaxy, redshift=0.5, mass=ag.lp.Sersic)) -) - -search = af.Nautilus( - path_prefix=Path("howtogalaxy", "chapter_2"), - name="tutorial_1_non_linear_search", - unique_tag=dataset_name, - n_live=80, - n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. -) - -analysis = ag.AnalysisImaging(dataset=dataset, use_jax=True) - -result = search.fit(model=model, analysis=analysis) - -""" -__Galaxies & Fit__ - -In the previous tutorials, we saw that this result contains the maximum log likelihood fit, which provide -a fast way to visualize the result. - -It also contains the maximum log likelihood galaxies. -""" -aplt.subplot_galaxies( - galaxies=result.max_log_likelihood_galaxies, grid=mask.derive_grid.all_false -) - -aplt.subplot_fit_imaging(fit=result.max_log_likelihood_fit) - -""" -__Samples__ - -The result contains a lot more information about the model-fit. - -For example, the `Samples` object contains all of the non-linear search samples, including the parameters -of every successful model evaluation and their log likelihood values. These are used for computing information -about the model-fit, such as the most probably parameter estimates and the error inferred for every parameter. -""" -print(result.samples) -print("Parameters of 1st Sample:") -print(result.samples.parameter_lists[0][:]) -print("Log Likelihood of 1st Sample:") -print(result.samples.log_likelihood_list[0]) - -""" -__Workspace__ - -We are not going into any more detail on the result variable in this tutorial, or in the **HowToGalaxy** lectures. - -A comprehensive description of the results can be found at the following packages: - - `autogalaxy_workspace/*/results` - -The results API for CCD imaging data is the same as for other data types (e.g. interferometer, point-soures). This -package can therefore be used to learn the API and then translate to other data types. - -__Database__ - -Once a search has completed running, we have a set of results on our hard disk which we can manually inspect and -analyse. Alternatively, we can return the results from the search.fit() method and manipulate them in a Python script -or notebook. - -However, imagine you have a large dataset consisting of many images of galaxies. You analyse each image -individually using a search, producing a large library of results on your hard disk. There will be lots of paths and -directories to navigate, and at some point there will simply be too many results for it to be an efficient use of your -time to analyse the results by sifting through the outputs on your hard disk one-by-one. - -**PyAutoGalaxy**'s database tools solve this problem, by making it possible for us to write the results to a .sqlite -database file and load the results from hard-disk to a Python script or Jupyter notebook. This database supports -advanced queries, so specific results can be loaded and inspected. - -We won't go into any more detail on the database in this tutorial. If you think the database will be useful, checkout -the full set of database tutorials which can be found in the folder `autogalaxy_workspace/*/imaging/advanced/database`. - -Here, you'll learn how to: - - - Use the database to query for results which fit a certain model or give a certain result. - - - Use the `Samples` to produce many different results from the fit, including error estimates on parameters and - plots of the probability density function of parameters in 1D and 2D. - - - Visualize results, for example the fit to a lens dataset. - - -__Wrap Up__ - -Even if you are only modeling a small sample of galaxies, if you anticipate using **PyAutoGalaxy** for the long-term I -strongly recommend you begin using the database to inspect and analyse your result. - -This is because it makes it simple to perform all analyse in a Jupyter notebook, which is the most flexible and -versatile way to check results and make figures. -""" diff --git a/scripts/howtogalaxy/chapter_2_modeling/tutorial_8_need_for_speed.py b/scripts/howtogalaxy/chapter_2_modeling/tutorial_8_need_for_speed.py deleted file mode 100644 index 80d37c33..00000000 --- a/scripts/howtogalaxy/chapter_2_modeling/tutorial_8_need_for_speed.py +++ /dev/null @@ -1,83 +0,0 @@ -""" -Tutorial 8: Need For Speed -========================== - -In this chapter, we have learnt how to model galaxies and how to balance complexity and realism to ensure that we -infer a good model. - -__Contents__ - -**Searching Non-linear Parameter Space:** How dimensionality, priors and settings drive search run-times. -**Algorithmic Optimization:** How PyAutoGalaxy uses numba and JAX for fast computation. -**Data Quantity:** How the number of image pixels affects run-time. -**Wrap Up:** Summary of strategies for keeping model-fit run-times manageable. - -For fitting more complex models, the final challenge that we face is keeping the run-time low. One can easily end -up in a situation where a model-fit takes days, or longer, to fit just one image. For fitting complex models and high -resolution datasets this is somewhat unavoidable. However, it is worth us discussing what drives the long run-times of -the modeling process and how we might speed it up. - -__Searching Non-linear Parameter Space__ - -The time it takes for the non-linear search to sample parameter space and find the high likelihood models is driven by: - - - Dimensionality: A more complex parameter space (e.g. more parameters) takes longer to search. - - Priors: The broader the priors on each parameter the longer the search. - - Settings: Non-linear search settings which sample parameter space more thoroughly lead to longer run-times. - -When we use only one search to fit a model, we are somewhat restricted in how we can try to achieve faster run -times by changing these 3 aspects of the search. - -In the next chapter, we introduce 'non-linear search chaining', which fits a model using multiple searches that -are performed back-to-back. A key motivation for this is that it gives us a lot more flexibility in juggling the -dimensionality, priors and settings so as to perform faster and more efficient modeling. - -In the optional **HowToGalaxy** tutorial `chapter_optional/tutorial_searches.ipynb` we discuss other non-linear -searches supported by **HowToGalaxy** which use a different approach to sample parameter sample than `nautilus`. For -those familiar with statistical inference, this includes maximum likelihood estimators and MCMC algorithms. - -For galaxy modeling, there are maximum likelihood estimator methods (e.g. the Levenberg-Marquardt) that for simple -models (e.g. a single `Sersic`) can often reliable infer the maximum likelihood model, using 10 times or fewer -likelihood evaluations than `nautilus` therefore running over ten times or more faster). However, these methods do -not infer reliable errors and are subject to inferring local maximum. Nevertheless, users modeling large galaxy -samples may wish to investigate these methods. - -__Algorithmic Optimization__ - -Every operation **PyAutoGalaxy** performs to fit galaxy data with a model takes time, for example: - - - Computing the intensity values from a light profile. - - Convolving the galaxies image with the PSF to compare it to the data. - -One can therefore in principle make **PyAutoGalaxy** run faster by using more efficient algorithms. However, I am -confident that for many tasks and operations we have written code that is already very fast! - -I often get asked, given that **PyAutoGalaxy** is written in Python (a synonymously slow programming language), is it -not really slow? **PyAutoGalaxy** uses a library called `numba` to ensure that it runs fast, which recompiles Python -functions into C functions before **PyAutoGalaxy** runs. This gives us C-like speed, but in Python code. If you`ve got -your own code that needs speeding up, I strongly recommend that you look up Numba: - -http://numba.pydata.org/ - -Therefore, **PyAutoGalaxy** is pretty well optimized and there are no 'low hanging fruit' speed ups available by -writing the code in a different language. - -__Data Quantity__ - -The final factor driving run-speed is the quantity of data that is fitted. For every image-pixel that we fit, -we have to compute the light profile intensities and convolve it with the telescope's PSF. The larger that PSF is, -the more convolution operations we have to perform too. - -In the previous exercises, we used images with a pixel scale of 0.1". This value is relatively low resolution: most -Hubble Space Telescope images have a pixel scale of 0.05", which is four times the number of pixels! Some telescopes -observe at scales of 0.03" or, dare I say it, 0.01". At these resolutions things can *really* slow down, if we -do not think carefully about run speed beforehand. - -There are ways that we can reduce the number of image-pixels we fit, via masking. If we mask out more of the image, -we will fit fewer pixels and **PyAutoGalaxy** will run faster. If you want the best, most perfect model possible, -aggressive masking and cutting the data in this way is a bad idea, as discussed in tutorial 5. - -__Wrap Up__ - -This tutorial simply wanted to get you thinking about *why* a model takes as long to fit as it does. -""" diff --git a/scripts/howtogalaxy/chapter_2_modeling/tutorial_9_summary b/scripts/howtogalaxy/chapter_2_modeling/tutorial_9_summary deleted file mode 100644 index e78d06ca..00000000 --- a/scripts/howtogalaxy/chapter_2_modeling/tutorial_9_summary +++ /dev/null @@ -1,17 +0,0 @@ -Congratulations, you've completed the second chapter of the **PyAutoGalaxy** tutorials! - -In this chapter, you learnt: - -1) What a non-linear search is. -2) How to use a non-linear search to sample the parameter space of a model. -3) How the priors we give parameters specify our search of this parameter space. -4) How one must carefully balance complexity and realism when fitting a model to a data-set. -5) The importance of factoring in run-speed when modeling a galaxy, and tricks to speed up the analysis. -6) About masking data in the analysis. - -At this point, you are ready to begin modeling galaxies. The 'modeling' folders in the -autogalaxy_workspace contains a number of scripts that can be easily adopted to model galaxies using a variety of -different approaches, models and non-linear searches. If you have your own galaxy data, I'd recommend you adapt these -scripts to your data. If you don't have your own data, checkout the 'simulators' folder to simulate your own dataset! - -Alternatively, you may wish to continue on to chapter 3 on search chaining. \ No newline at end of file diff --git a/scripts/howtogalaxy/chapter_3_search_chaining/README.rst b/scripts/howtogalaxy/chapter_3_search_chaining/README.rst deleted file mode 100644 index b44e0fa6..00000000 --- a/scripts/howtogalaxy/chapter_3_search_chaining/README.rst +++ /dev/null @@ -1,25 +0,0 @@ -In chapter 3, we introduce non-linear search chaining, whereby modeling pipelines are composed which each fit a -different model. - -**Colab** links to every tutorial are included. - -Files ------ - -`Tutorial 1: Search Chaining `_ -- Breaking the modeling procedure into a chained sequence of model-fits. - -`Tutorial 2: Prior Passing `_ -- How the results of earlier searches are passed to later searches. - -`Tutorial 3: Lens `_ -- Fitting the galaxy's light followed by its mass using chained searches. - -`Tutorial 4: Two Lens galaxies `_ -- Modeling a galaxy with two lens galaxies using chained searches. - -`Tutorial 5: Complex Source `_ -- Using multiple light profiles to fit a complex and irregular source using chained searches. - -`Tutorial 6: SLaM `_ -- Template pipelines for fitting model is standardized ways. \ No newline at end of file diff --git a/scripts/howtogalaxy/chapter_3_search_chaining/__init__.py b/scripts/howtogalaxy/chapter_3_search_chaining/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/scripts/howtogalaxy/chapter_3_search_chaining/introduction b/scripts/howtogalaxy/chapter_3_search_chaining/introduction deleted file mode 100644 index 24f9842d..00000000 --- a/scripts/howtogalaxy/chapter_3_search_chaining/introduction +++ /dev/null @@ -1,12 +0,0 @@ -You are now familiar and have a clear understanding of modeling. In this chapter, we'll -introduce search chaining, a concept we mentioned in the previous chapter. Search chaining allows us to generically link -searches together, so that we can seamlessly navigate the complex non-linear parameter spaces that come when fitting -realistic models. - -With these pipelines, you'll be able to: - -1) Fit a lens mass model light model to an image of a strongly lensed source. -2) Additionally fit the galaxy's light, if it is present. -3) Write customized pipelines for galaxy systems with multiple lens galaxies or source galaxies. -4) Customize pipelines such that the priors on parameters during the fit are adjusted to provide a more robust or - efficient fit. \ No newline at end of file diff --git a/scripts/howtogalaxy/chapter_3_search_chaining/tutorial_1_search_chaining.py b/scripts/howtogalaxy/chapter_3_search_chaining/tutorial_1_search_chaining.py deleted file mode 100644 index 38af9fe8..00000000 --- a/scripts/howtogalaxy/chapter_3_search_chaining/tutorial_1_search_chaining.py +++ /dev/null @@ -1,311 +0,0 @@ -""" -Tutorial 1: Search Chaining -=========================== - -In chapter 2, we learnt how to perform modeling using a non-linear search. In all of the tutorials, we fitted the -data using just one non-linear search. In this chapter, we introduce a technique called 'non-linear search chaining', -which fits a model using a sequence of non-linear searches. The initial searches fit simpler models whose parameter -spaces can be more accurately and efficiently sampled. The results of this search are then passed to later searches -which fit models of gradually increasing complexity. - -Lets think back to tutorial 4 of chapter 2. We learnt there were three approaches one could take fitting a model -accurately if we found that a model fit failed. These were: - - 1) Tuning our priors to the galaxy we're fitting. - 2) Making our model less complex. - 3) Searching non-linear parameter space for longer. - -However, each of the above approaches has disadvantages. The more we tune our priors, the less we can generalize our -analysis to a different galaxy. The less complex we make our model, the less realistic it is. And if we rely too -much on searching parameter space for longer, we could end up with search`s that take days, weeks or months to run. - -In this tutorial, we are going to show how search chaining combines these 3 approaches such that we can fit -complex and realistic models in a way that that can be generalized to many different galaxies. To do this, -we'll run 2 searches, and chain the model inferred in the first search to the priors of the second search`s lens -model. - -Our first search will make the same bulge-disk alignment assumption we made in the previous tutorial. We saw that this -gives a reasonable model. However, we'll make a couple of extra simplifying assumptions, to really try and bring -our model complexity down and get the non-linear search running fast. - -The model we infer above will therefore be a lot less realistic. But it does not matter, because in the second search -we are going to relax these assumptions and fit the more realistic model. The beauty is that, by running the first -search, we can use its results to tune the priors of our second search. For example: - - 1) The first search should give us a pretty good idea of the galaxy's bulge and disk profiles, for example its - centre, intensity, effective radius. - -__Contents__ - -**Initial Setup:** Load the dataset and apply a mask. -**Model:** Compose a simplified model for the first search with aligned bulge-disk assumptions. -**Search + Analysis:** Run the first search with the simplified model. -**Result:** Inspect the result of the first search. -**Prior Passing:** Manually pass results from search 1 as priors for search 2. -**Result:** Inspect the final result of the chained search. -**Wrap Up:** Summary of how search chaining improves model-fitting reliability. -""" - -# from autoconf import setup_notebook; setup_notebook() - -import numpy as np -from pathlib import Path -import autogalaxy as ag -import autogalaxy.plot as aplt -import autofit as af - -""" -__Initial Setup__ - -we'll use the same galaxy data as tutorial 4 of chapter 2, where: - - - The galaxy's bulge is an `Sersic`. - - The galaxy's disk is an `Exponential`. -""" -dataset_name = "simple" -dataset_path = Path("dataset") / "imaging" / dataset_name - -""" -__Dataset Auto-Simulation__ - -If the dataset does not already exist on your system, it will be created by running the corresponding -simulator script. This ensures that all example scripts can be run without manually simulating data first. -""" -if not dataset_path.exists(): - import subprocess - import sys - - subprocess.run( - [sys.executable, "scripts/imaging/simulator.py"], - check=True, - ) - - -dataset = ag.Imaging.from_fits( - data_path=dataset_path / "data.fits", - noise_map_path=dataset_path / "noise_map.fits", - psf_path=dataset_path / "psf.fits", - pixel_scales=0.1, -) - -mask = ag.Mask2D.circular( - shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=2.5 -) - -dataset = dataset.apply_mask(mask=mask) - -over_sample_size = ag.util.over_sample.over_sample_size_via_radial_bins_from( - grid=dataset.grid, - sub_size_list=[8, 4, 1], - radial_list=[0.3, 0.6], - centre_list=[(0.0, 0.0)], -) - -dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size) - -aplt.subplot_imaging_dataset(dataset=dataset) - -""" -__Model__ - -As we've eluded to before, one can look at an image and immediately identify the centre of the galaxy. It's -that bright blob of light in the middle! Given that we know we're going to make the model more complex in the -next search, lets take a more liberal approach than before and fix the centre of the bulge and -disk to $(y,x)$ = (0.0", 0.0").. - -Now, you might be thinking, doesn`t this prevent our search from generalizing to other galaxies? What if the -centre of their galaxy isn't at (0.0", 0.0")? - -Well, this is true if our dataset reduction centres the galaxy somewhere else. But we get to choose where we -centre it when we make the image. Therefore, I`d recommend you always centre the galaxy at the same location, -and (0.0", 0.0") seems the best choice! -""" -bulge = af.Model(ag.lp_linear.Sersic) -disk = af.Model(ag.lp_linear.Exponential) - -""" -You haven't actually seen a line like this one before. By setting a parameter to a number (and not a prior) it is be -removed from non-linear parameter space and always fixed to that value. Pretty neat, huh? -""" -bulge.centre_0 = 0.0 -bulge.centre_1 = 0.0 -disk.centre_0 = 0.0 -disk.centre_1 = 0.0 - -galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=bulge, disk=disk) - -model_1 = af.Collection(galaxies=af.Collection(galaxy=galaxy)) - -""" -The `info` attribute shows the model in a readable format. -""" -print(model_1.info) - -""" -__Search + Analysis__ - -Now lets create the search and analysis. -""" -search_1 = af.Nautilus( - path_prefix=Path("howtogalaxy", "chapter_3"), - name="tutorial_1_search_chaining_1", - unique_tag=dataset_name, - n_live=100, - n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. -) - -analysis_1 = ag.AnalysisImaging(dataset=dataset, use_jax=True) - -""" -Lets run the search, noting that our liberal approach to reducing the model complexity has reduced it to just -6 parameters. -""" -print( - "The non-linear search has begun running - checkout the workspace/output/5_chaining_searches" - " folder for live output of the results, images and model." - " This Jupyter notebook cell with progress once search has completed - this could take some time!" -) - -result_1 = search_1.fit(model=model_1, analysis=analysis_1) - -print("Search has finished run - you may now continue the notebook.") - -""" -__Result__ - -The results are summarised in the `info` attribute. -""" -print(result_1.info) - -""" -And indeed, we get a reasonably good model and fit to the data, in a much shorter space of time! -""" -aplt.subplot_fit_imaging(fit=result_1.max_log_likelihood_fit) - -""" -__Prior Passing__ - -Now all we need to do is look at the results of search 1 and pass the results as priors for search 2. Lets setup -a custom search that does exactly that. - -`TruncatedGaussianPrior`'s are a nice way to pass priors. They tell the non-linear search where to look, but leave open the -possibility that there might be a better solution nearby. In contrast, `UniformPrior`'s put hard limits on what values a -parameter can or can`t take. It makes it more likely we will accidentally cut-out the global maxima solution. - -Note that below the `disk` has become an `Sersic`. -""" -bulge = af.Model(ag.lp_linear.Sersic) -disk = af.Model(ag.lp_linear.Sersic) - -""" -What I've done below is looked at the results of search 1 and manually specified a prior for every parameter. If a -parameter was fixed in the previous search, its prior is based around the previous value. Don't worry about the sigma -values for now, I've chosen values that I know will ensure reasonable sampling, but we'll cover this later. - -__LENS BULGE PRIORS:__ -""" -bulge.centre.centre_0 = af.TruncatedGaussianPrior( - mean=0.0, sigma=0.1, lower_limit=-np.inf, upper_limit=np.inf -) -bulge.centre.centre_1 = af.TruncatedGaussianPrior( - mean=0.0, sigma=0.1, lower_limit=-np.inf, upper_limit=np.inf -) -bulge.ell_comps.ell_comps_0 = af.TruncatedGaussianPrior( - mean=0.11, sigma=0.2, lower_limit=-1.0, upper_limit=1.0 -) -bulge.ell_comps.ell_comps_1 = af.TruncatedGaussianPrior( - mean=0.05, sigma=0.2, lower_limit=-1.0, upper_limit=1.0 -) -bulge.effective_radius = af.TruncatedGaussianPrior( - mean=0.75, sigma=0.4, lower_limit=0.0, upper_limit=np.inf -) -bulge.sersic_index = af.TruncatedGaussianPrior( - mean=4.0, sigma=2.0, lower_limit=0.0, upper_limit=np.inf -) - -""" -__LENS DISK PRIORS:__ -""" -disk.centre.centre_0 = af.TruncatedGaussianPrior( - mean=0.0, sigma=0.1, lower_limit=-np.inf, upper_limit=np.inf -) -disk.centre.centre_1 = af.TruncatedGaussianPrior( - mean=0.0, sigma=0.1, lower_limit=-np.inf, upper_limit=np.inf -) -disk.ell_comps.ell_comps_0 = af.TruncatedGaussianPrior( - mean=0.11, sigma=0.2, lower_limit=-1.0, upper_limit=1.0 -) -disk.ell_comps.ell_comps_1 = af.TruncatedGaussianPrior( - mean=0.05, sigma=0.2, lower_limit=-1.0, upper_limit=1.0 -) -disk.effective_radius = af.TruncatedGaussianPrior( - mean=1.52, sigma=0.4, lower_limit=0.0, upper_limit=np.inf -) -disk.sersic_index = af.TruncatedGaussianPrior( - mean=1.0, sigma=2.0, lower_limit=0.0, upper_limit=np.inf -) - -""" -We now compose the model with these components that have had their priors customized. -""" -galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=bulge, disk=disk) - -model_2 = af.Collection(galaxies=af.Collection(galaxy=galaxy)) - -""" -The `info` attribute shows the model, including the priors specified above. -""" -print(model_2.info) - -""" -Lets setup and run the search. As expected, it gives us the correct model. However, it does so significantly -faster than we are used to! -""" -batch_size = 50 # Explained chapter 2 tutorial 2 - -search_2 = af.Nautilus( - path_prefix=Path("howtogalaxy", "chapter_3"), - name="tutorial_1_search_chaining_2", - unique_tag=dataset_name, - n_live=100, - n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. -) - -analysis_2 = ag.AnalysisImaging(dataset=dataset, use_jax=True) - -print( - "The non-linear search has begun running - checkout the workspace/output/5_chaining_searches" - " folder for live output of the results, images and model." - " This Jupyter notebook cell with progress once search has completed - this could take some time!" -) - -result_2 = search_2.fit(model=model_2, analysis=analysis_2) - -print("Search has finished run - you may now continue the notebook.") - -""" -__Result__ - -We can again inspect the results via the `info` attribute. -""" -print(result_2.info) - -""" -And a plot of the image shows we get a good model again! -""" -aplt.subplot_fit_imaging(fit=result_2.max_log_likelihood_fit) - -""" -__Wrap Up__ - -Chaining two searches together was a huge success. We managed to fit a complex and realistic model, but were able to -begin by making simplifying assumptions that eased our search of non-linear parameter space. We could apply search 1 to -pretty much any galaxy and therefore get ourselves a decent model with which to tune search 2`s priors. - -You are probably thinking though that there is one huge, giant, glaring flaw in all of this that I've not mentioned. -Search 2 can`t be generalized to another lens, because its priors are tuned to the image we fitted. If we had a lot -of galaxies, we`d have to write a new search for every single one. This isn't ideal, is it? - -Fortunately, we can pass priors in **PyAutoGalaxy** without specifying the specific values. The API for this technique, -called prior passing, is the topic of the next tutorial. -""" diff --git a/scripts/howtogalaxy/chapter_3_search_chaining/tutorial_2_prior_passing.py b/scripts/howtogalaxy/chapter_3_search_chaining/tutorial_2_prior_passing.py deleted file mode 100644 index a9d2b276..00000000 --- a/scripts/howtogalaxy/chapter_3_search_chaining/tutorial_2_prior_passing.py +++ /dev/null @@ -1,332 +0,0 @@ -""" -Tutorial 2: Prior Passing -========================= - -In the previous tutorial, we used non-linear search chaining to break the model-fitting procedure down into two -non-linear searches. This used an initial search to fit a simple model, whose results were used to tune and -initialize the priors of a more complex model that was fitted by the second search. - -However, the results were passed between searches were passed manually. I explicitly wrote out every result as a prior -containing the values inferred in the first search. **PyAutoGalaxy** has an API for passing priors in a more generalized -way, which is the topic of this tutorial. - -__Contents__ - -**Initial Setup:** Load the dataset and apply a mask. -**Model:** Compose the model for the first search. -**Search:** Run the first search. -**Result (Search 1):** Inspect the result of the first search. -**Prior Passing:** Use the prior passing API to pass results to the second search. -**Result:** Inspect the final result of the chained search. -**Wrap Up:** Summary of the prior passing API. -**Detailed Explanation Of Prior Passing:** In-depth explanation of how priors are passed between searches. -**EXAMPLE:** A worked example of prior passing for a Sersic sersic_index parameter. -""" - -# from autoconf import setup_notebook; setup_notebook() - -import numpy as np -from pathlib import Path -import autogalaxy as ag -import autogalaxy.plot as aplt -import autofit as af - -""" -__Initial Setup__ - -we'll use the same galaxying data as the previous tutorial, where: - - - The galaxy's bulge is an `Sersic`. - - The galaxy's disk is an `Exponential`. - -All the usual steps for setting up a model fit (masking, analysis, etc.) are included below. -""" -dataset_name = "simple" -dataset_path = Path("dataset") / "imaging" / dataset_name - -""" -__Dataset Auto-Simulation__ - -If the dataset does not already exist on your system, it will be created by running the corresponding -simulator script. This ensures that all example scripts can be run without manually simulating data first. -""" -if not dataset_path.exists(): - import subprocess - import sys - - subprocess.run( - [sys.executable, "scripts/imaging/simulator.py"], - check=True, - ) - - -dataset = ag.Imaging.from_fits( - data_path=dataset_path / "data.fits", - noise_map_path=dataset_path / "noise_map.fits", - psf_path=dataset_path / "psf.fits", - pixel_scales=0.1, -) - -mask = ag.Mask2D.circular( - shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=2.5 -) - -dataset = dataset.apply_mask(mask=mask) - -over_sample_size = ag.util.over_sample.over_sample_size_via_radial_bins_from( - grid=dataset.grid, - sub_size_list=[8, 4, 1], - radial_list=[0.3, 0.6], - centre_list=[(0.0, 0.0)], -) - -dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size) - -aplt.subplot_imaging_dataset(dataset=dataset) - -""" -__Model__ - -We are going to use the same result of search 1 from the previous tutorial. Thus, we set up an identical model such -that we instantly load the result from hard-disk. -""" -bulge = af.Model(ag.lp_linear.Sersic) -disk = af.Model(ag.lp_linear.Exponential) - -bulge.centre_0 = 0.0 -bulge.centre_1 = 0.0 -disk.centre_0 = 0.0 -disk.centre_1 = 0.0 - -disk.ell_comps = bulge.ell_comps - -bulge.sersic_index = 4.0 - -galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=bulge, disk=disk) - -model_1 = af.Collection(galaxies=af.Collection(galaxy=galaxy)) - -""" -The `info` attribute shows the model in a readable format. -""" -print(model_1.info) - -""" -__Search__ - -We also create the same search as the previous tutorial, using the same name to ensure we use the same results, and -run it. -""" -analysis_1 = ag.AnalysisImaging(dataset=dataset, use_jax=True) - -search_1 = af.Nautilus( - path_prefix=Path("howtogalaxy", "chapter_3"), - name="tutorial_1_search_chaining_1", - unique_tag=dataset_name, - n_live=100, - n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. -) - -result_1 = search_1.fit(model=model_1, analysis=analysis_1) - -""" -__Result (Search 1)__ - -The results which are used for prior passing are summarised in the `info` attribute. -""" -print(result_1.info) - -""" -__Prior Passing__ - -We are now going to use the prior passing API to pass these results, in a way which does not require us to manually -write out the inferred parameter values of each component. The details of how prior passing is performed will be -expanded upon at the end of the tutorial. - -We start with the bulge, which in the previous search was an `Sersic` with its centre fixed to (0.0, 0.0) -and its `sersic_index` fixed to 4.0. The API for passing priors is shown below and there are two things worth noting: - - 1) We pass the priors using the `model` attribute of the result. This informs **PyAutoGalaxy** to pass the result as a - model component that is to be fitted for in the next search, using priors that are initialized from the previous - search's result. Note, if we pass as a `model` a parameter that was fixed in search 1 (e.g. the `sersic_index`) it - will be fixed to the same value in search 2. - - 2) We do not pass the `centre` or `sersic_index` using `model`, because it would be fixed to the values that it was in - the first search. By omitting the centre, it uses the default priors on a galaxy, whereas we manually tell the - Sersic index to use a `TruncatedGaussianPrior` centred on 4.0. -""" -bulge = af.Model(ag.lp_linear.Sersic) - -bulge.ell_comps = result_1.model.galaxies.galaxy.bulge.ell_comps -bulge.effective_radius = result_1.model.galaxies.galaxy.bulge.effective_radius -bulge.sersic_index = af.TruncatedGaussianPrior( - mean=4.0, sigma=2.0, lower_limit=0.0, upper_limit=5.0 -) - -""" -For the disk, we are passing the result of an `Exponential` to an `Sersic`. - -We do not pass the `ell_comps` because this would pair them to the `bulge`, as was performed in the first -model-fit. -""" -disk = af.Model(ag.lp_linear.Sersic) - -disk.effective_radius = result_1.model.galaxies.galaxy.disk.effective_radius -disk.sersic_index = af.TruncatedGaussianPrior( - mean=1.0, sigma=2.0, lower_limit=0.0, upper_limit=5.0 -) - -""" -We now compose the model with these components that have had their priors customized. -""" -galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=bulge, disk=disk) - -model_2 = af.Collection(galaxies=af.Collection(galaxy=galaxy)) - -""" -The `info` attribute shows the model, including how all priors are updated via prior passing. -""" -print(model_2.info) - -""" -__Search__ - -Lets setup and run the search. I have given it a different name to the previous tutorial so we can compare the priors -that were passed. -""" -analysis_2 = ag.AnalysisImaging(dataset=dataset, use_jax=True) - -search_2 = af.Nautilus( - path_prefix=Path("howtogalaxy", "chapter_3"), - name="tutorial_2_search_chaining_2", - unique_tag=dataset_name, - n_live=100, - n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. -) - -print( - "The non-linear search has begun running - checkout the workspace/output/5_chaining_searches" - " folder for live output of the results, images and model." - " This Jupyter notebook cell with progress once search has completed - this could take some time!" -) - -result_2 = search_2.fit(model=model_2, analysis=analysis_2) - -print("Search has finished run - you may now continue the notebook.") - - -""" -__Result__ - -We can again inspect the results via the `info` attribute. -""" -print(result_2.info) - -""" -And a plot of the image shows we get a good model again! -""" -aplt.subplot_fit_imaging(fit=result_2.max_log_likelihood_fit) - -""" -__Wrap Up__ - -We will expand on the prior passing API in the following tutorials. The main thing to note is that we can pass -entire profiles or galaxies using prior passing, if their model does not change (which for the bulge and disk, was -not true). The API to pass a whole profile or galaxy is as follows: - - bulge = result_1.model.galaxies.galaxy.bulge - galaxy = result_1.model.galaxies.galaxy - -We can also pass priors using an `instance` instead of a `model`. When an `instance` is used, the maximum likelihood -parameter values are passed as fixed values that are therefore not fitted for by the non-linear search (reducing its -dimensionality). We will use this in the next tutorial to fit data with two galaxies, where fit one galaxy, fix it to -the best-fit model in a second search that fits the second galaxy, and then go on to fit both simultaneously in the -final search. - -Lets now think about how priors are passed. Checkout the `model.info` file of the second search of this tutorial. The -parameters do not use the default priors we saw in search 1 (which are typically broad UniformPriors). Instead, -they use GaussianPrior`s where: - - - The mean values are the median PDF results of every parameter in search 1. - - The sigma values are specified in the `width_modifier` field of the profile's entry in the `priors.yaml' config - file (we will discuss why this is used in a moment). - -Like the manual `GaussianPrior`'s that were used in tutorial 1, the prior passing API sets up the prior on each -parameter with a `GaussianPrior` centred on the high likelihood regions of parameter space! - -__Detailed Explanation Of Prior Passing__ - -To end, I provide a detailed overview of how prior passing works and illustrate tools that can be used to customize -its behaviour. It is up to you whether you want read this, or go ahead to the next tutorial! - -Lets say I chain two parameters as follows: - - `bulge.effective_radius = result_1.model.galaxies.galaxy.bulge.effective_radius` - -By invoking the `model` attribute, the prior is passed following 3 rules: - - 1) The new parameter, in this case the einstein radius, uses a `GaussianPrior`.This is ideal, as the 1D pdf results - we compute at the end of a search are easily summarised as a Gaussian. - - 2) The mean of the `GaussianPrior` is the median PDF value of the parameter estimated in search 1. - - This ensures that the initial sampling of the new search's non-linear starts by searching the region of non-linear - parameter space that correspond to highest log likelihood solutions in the previous search. Our priors therefore - correspond to the `correct` regions of parameter space. - - 3) The sigma of the Gaussian uses the value specified for the profile in the `config/priors/*.yaml` config file's - `width_modifer` field (check these files out now). - -The idea here is simple. We want a value of sigma that gives a `GaussianPrior` wide enough to search a broad -region of parameter space, so that the model can change if a better solution is nearby. However, we want it -to be narrow enough that we don't search too much of parameter space, as this will be slow or risk leading us -into an incorrect solution! - -The `width_modifier` values in the priors config file have been chosen based on our experience as being a good -balance broadly sampling parameter space but not being so narrow important solutions are missed. - -There are two ways a value is specified using the priors/width file: - - 1) Absolute: In this case, the error assumed on the parameter is the value given in the config file. - For example, if for the width on centre_0 of a light profile, the width modifier reads "Absolute" with a value - 0.05. This means if the error on the parameter centre_0 was less than 0.05 in the previous search, the sigma of - its `GaussianPrior` in this search will be 0.05. - - 2) Relative: In this case, the error assumed on the parameter is the % of the value of the estimated value given in - the config file. For example, if the intensity estimated in the previous search was 2.0, and the relative error in - the config file reads "Relative" with a value 0.5, then the sigma of the `GaussianPrior` will be 50% of this - value, i.e. sigma = 0.5 * 2.0 = 1.0. - -We use absolute and relative values for different parameters, depending on their properties. For example, using the -relative value of a parameter like the `Profile` centre makes no sense. If our galaxy is centred at (0.0, 0.0), -the relative error will always be tiny and thus poorly defined. Therefore, the default configs in **PyAutoGalaxy** use -absolute errors on the centre. - -However, there are parameters where using an absolute value does not make sense. Intensity is a good example of this. -The intensity of an image depends on its units, S/N, galaxy brightness, etc. There is no single absolute value that -one can use to generically chain the intensity of any two proflies. Thus, it makes more sense to chain them using -the relative value from a previous search. - -We can customize how priors are passed from the results of a search and non-linear search by editing the - `prior_passer` settings in the `general.yaml` config file. - -__EXAMPLE__ - -Lets go through an example using a real parameter. Lets say in search 1 we fit the galaxy's light with an -elliptical Sersic profile, and we estimate that its sersic index is equal to 4.0. - -To pass this as a prior to search 2 we write: - - galaxy.bulge.sersic_index = result_1.model.galaxy.bulge.sersic_index - -The prior on the galaxy's bulge sersic index in search 2 would thus be a `GaussianPrior` with mean=4.0. - -The value of the Sersic index `width_modifier` in the priors config file sets sigma. The prior config file specifies -that we use an "Absolute" value of 0.8 to chain this prior. Thus, the `GaussianPrior` in search 2 would have a -mean=4.0 and sigma=0.8. - -If the prior config file had specified that we use an relative value of 0.8, the GaussianPrior in search 2 would have a -mean=4.0 and sigma = 4.0 * 0.8 = 3.2. - -And with that, we're done. Chaining priors is a bit of an art form, but one that works really well. -""" diff --git a/scripts/howtogalaxy/chapter_3_search_chaining/tutorial_3_x2_galaxies.py b/scripts/howtogalaxy/chapter_3_search_chaining/tutorial_3_x2_galaxies.py deleted file mode 100644 index 538eaf2d..00000000 --- a/scripts/howtogalaxy/chapter_3_search_chaining/tutorial_3_x2_galaxies.py +++ /dev/null @@ -1,286 +0,0 @@ -""" -Tutorial 3: Two Galaxies -======================== - -Up to now, all the images we've fitted had one galaxy. However, we saw in chapter 1 that our galaxies object can -consist of multiple galaxies which each contribute to the overall emission. Multi-galaxy systems are challenging to -model, because they add an extra 5-10 parameters to the non-linear search per galaxy and, more problematically, the -degeneracies between the parameters of the light profiles of the galaxies can be severe. - -However, we can still break their analysis down using multiple searches and give ourselves a shot at getting a good -model. Here, we're going to fit a double galaxy system, fitting as much about each individual galaxy before -fitting them simultaneously. - -Up to now, I've put a focus on an analysis being generag. The script we write in this example is going to be the -opposite, specific to the image we're modeling. Fitting multiple galaxies is really difficult and writing a -pipeline that we can generalize to many galaxies isn't currently possible. - -__Contents__ - -**Initial Setup:** Load the double-galaxy dataset and apply a mask. -**Paths:** Set the output path for chained search results. -**Search Chaining Approach:** Strategy for fitting two galaxies using a sequence of searches. -**Model + Search + Analysis + Model-Fit (Search 1):** Fit the first galaxy alone. -**Result (Search 1):** Inspect the result of fitting the first galaxy. -**Model (Search 2):** Compose a model for the second galaxy, fixing the first. -**Search + Analysis + Model-Fit (Search 2):** Fit the second galaxy. -**Result (Search 2):** Inspect the result of fitting the second galaxy. -**Model + Search + Analysis + Model-Fit (Search 4):** Fit both galaxies simultaneously. -**Result (Search 3):** Inspect the final simultaneous fit result. -**Wrap Up:** Summary of search chaining for multi-galaxy systems. -""" - -# from autoconf import setup_notebook; setup_notebook() - -from pathlib import Path -import autofit as af -import autogalaxy as ag -import autogalaxy.plot as aplt - -""" -__Initial Setup__ - -we'll use new galaxying data, where: - - - There are two galaxy's whose `LightProfile`'s are both `Sersic`'s. -""" -dataset_name = "sersic_x2" -dataset_path = Path("dataset") / "imaging" / dataset_name - -""" -__Dataset Auto-Simulation__ - -If the dataset does not already exist on your system, it will be created by running the corresponding -simulator script. This ensures that all example scripts can be run without manually simulating data first. -""" -if not dataset_path.exists(): - import subprocess - import sys - - subprocess.run( - [sys.executable, "scripts/guides/plot/simulator.py"], - check=True, - ) - - -dataset = ag.Imaging.from_fits( - data_path=dataset_path / "data.fits", - noise_map_path=dataset_path / "noise_map.fits", - psf_path=dataset_path / "psf.fits", - pixel_scales=0.05, -) - -""" -__Mask__ - -We need to choose our mask for the analysis. Given the light of both galaxies is present in the image we'll need to -include all their light in the image, so lets use a large circular mask. - -We'll use this mask in all three searches, however you could imagine customizing it on a per-search basis to speed up -the analysis. -""" -mask = ag.Mask2D.circular( - shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=6.0 -) - -dataset = dataset.apply_mask(mask=mask) - -over_sample_size = ag.util.over_sample.over_sample_size_via_radial_bins_from( - grid=dataset.grid, - sub_size_list=[8, 4, 1], - radial_list=[0.3, 0.6], - centre_list=[(0.0, -1.0), (0.0, 1.0)], -) - -dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size) - -aplt.subplot_imaging_dataset(dataset=dataset) - - -""" -__Paths__ - -All four searches will use the same `path_prefix`, so we write it here to avoid repetition. -""" -path_prefix = Path("howtogalaxy") / "chapter_3" / "tutorial_3_x2_galaxies" - -""" -__Search Chaining Approach__ - -Looking at the image, there are two blobs of light corresponding to the two galaxies. - -So, how can we break the modeling up? As follows: - - 1) Fit and subtract the light of the left galaxy individually. - 2) Fit and subtract the light of the right galaxy individually. - 3) Use these results to initialize a fit which fits both galaxy's simultaneously. - -So, with this in mind, we'll perform an analysis using 3 searches: - - 1) Fit the light of the galaxy on the left of the image, at coordinates (0.0", -1.0"). - 2) Fit the light of the galaxy on the right of the image, at coordinates (0.0", 1.0"). - 4) Fit all relevant parameters simultaneously, using priors from searches 1, and 2. - -__Model + Search + Analysis + Model-Fit (Search 1)__ - -Search 1 we fit a model where: - - - The left galaxy's light is a parametric linear `DevVaucouleurs` bulge with fixed centre [3 parameters]. - - - the right galaxy's light is omitted. - -The number of free parameters and therefore the dimensionality of non-linear parameter space is N=3. - -__Notes__ - -The `DevVaucouleurs` is an `Sersic` profile with `sersic_index=4`. - -We fix the centre of its light to (0.0, -1.0), the pixel we know the left galaxy's light centre peaks. - -We use linear light profiles througout this script, given that the model is quite complex and this helps -simplify it. -""" -left_galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=ag.lp_linear.DevVaucouleurs) -left_galaxy.bulge.centre_0 = 0.0 -left_galaxy.bulge.centre_1 = -1.0 - -model_1 = af.Collection(galaxies=af.Collection(left_galaxy=left_galaxy)) - -""" -The `info` attribute shows the model in a readable format. -""" -print(model_1.info) - -""" -__Search + Analysis + Model-Fit (Search 1)__ -""" -analysis_1 = ag.AnalysisImaging(dataset=dataset, use_jax=True) - -search_1 = af.Nautilus( - path_prefix=path_prefix, - name="search[1]__left_galaxy_light[bulge_linear]", - unique_tag=dataset_name, - n_live=75, - n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. -) - -result_1 = search_1.fit(model=model_1, analysis=analysis_1) - -""" -__Result (Search 1)__ - -The results which are used for prior passing are summarised in the `info` attribute. -""" -print(result_1.info) - -""" -__Model (Search 2)__ - -Search 2 we fit a model where: - - - The left galaxy's light is a parametric linear `DevVaucouleurs` bulge [0 parameters: fixed from search 1]. - - - The right galaxy's light is a parametric linear `DevVaucouleurs` bulge with a fixed centre [3 parameters]. - - - The galaxy's mass galaxy are omitted. - -The number of free parameters and therefore the dimensionality of non-linear parameter space is N=3. - -We fix the centre of the right lens's light to (0.0, 1.0), the pixel we know the right galaxy's light centre peaks. - -We also pass the result of the `left_galaxy` from search ` as an `instance`, which should improve the fitting of the -right lens. -""" -right_galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=ag.lp_linear.DevVaucouleurs) -right_galaxy.bulge.centre_0 = 0.0 -right_galaxy.bulge.centre_1 = 1.0 - -model_2 = af.Collection( - galaxies=af.Collection( - left_galaxy=result_1.instance.galaxies.left_galaxy, right_galaxy=right_galaxy - ) -) - -""" -The `info` attribute shows the model, including how all priors are updated via prior passing. -""" -print(model_2.info) - -""" -__Search + Analysis + Model-Fit (Search 2)__ -""" -analysis_2 = ag.AnalysisImaging(dataset=dataset, use_jax=True) - -search_2 = af.Nautilus( - path_prefix=path_prefix, - name="search[2]__right_galaxy_light[bulge_linear]", - unique_tag=dataset_name, - n_live=75, - n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. -) - -result_2 = search_2.fit(model=model_2, analysis=analysis_2) - -""" -__Result (Search 2)__ -The results which are used for prior passing are summarised in the `info` attribute. -""" -print(result_2.info) - -""" -__Model + Search + Analysis + Model-Fit (Search 4)__ - -Search 4 we fit a model where: - - - The left galaxy's light is a parametric linear `Sersic` bulge with centre fixed [4 parameters: priors initialized - from search 1]. - - - The right galaxy's light is a parametric linear `Sersic` bulge with centre fixed [4 parameters: priors initialized - from search 2]. - -The number of free parameters and therefore the dimensionality of non-linear parameter space is N=8. - -We can use a special prior passing method to do this, called `take_attributes`. This scans the `DevVaucouleurs` -passed to the `take_attributes` method for all parameters which have the same name as the `Sersic` model, -and if their names are the same it passes their prior as a `model` (like we did above). Thus, it will locate all 6 -parameters in common between the two profiles (centre, ell_comps, intensity, effective_radius) and pass those, -leaving the `sersic_index`'s priors as the default values. - -The `take_attributes` method is used in many examples of prior passing, when we pass a simpler parameterization of a -model to a more complex model. Another good example would be passing the result of a `IsothermalSph` to an -`Isothermal`. -""" -left_galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=ag.lp_linear.Sersic) -left_galaxy.bulge.take_attributes(result_1.model.galaxies.left_galaxy.bulge) - -right_galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=ag.lp_linear.Sersic) -right_galaxy.bulge.take_attributes(result_2.model.galaxies.right_galaxy.bulge) - -model_3 = af.Collection( - galaxies=af.Collection(left_galaxy=left_galaxy, right_galaxy=right_galaxy) -) - -analysis_3 = ag.AnalysisImaging(dataset=dataset, use_jax=True) - -search_3 = af.Nautilus( - path_prefix=path_prefix, - name="search[3]_light_x2[bulge_linear]", - unique_tag=dataset_name, - n_live=100, - n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. -) - -result_3 = search_3.fit(model=model_3, analysis=analysis_3) - -""" -__Result (Search 3)__ - -The final results are summarised in the `info` attribute. -""" -print(result_3.info) - -""" -__Wrap Up__ - -We have successfully fitted multiple galaxies, but fitting each one-by-one. -""" diff --git a/scripts/howtogalaxy/chapter_4_pixelizations/README.rst b/scripts/howtogalaxy/chapter_4_pixelizations/README.rst deleted file mode 100644 index 75d680f2..00000000 --- a/scripts/howtogalaxy/chapter_4_pixelizations/README.rst +++ /dev/null @@ -1,30 +0,0 @@ -In chapter 4, we use **Pixelizations** to reconstruct complex source galaxies on pixelized grids. - -**Colab** links to every tutorial are included. - -Files ------ - -`Tutorial 1: Pixelizations `_ -- Creating a pixel-grid in the source-plane. - -`Tutorial 2: Mappers `_ -- How a pixelization maps source-pixels to image-pixels. - -`Tutorial 3: Inversions `_ -- Inverting the mappings to reconstruct the source's light. - -`Tutorial 4: Bayesian Regularization `_ -- Smoothing the source within a Bayesian framework. - -`Tutorial 5: Borders `_ -- Preventing highly demagnified image-pixels ruining the inversion. - -`Tutorial 6: Lens Modeling `_ -- How to use inversions to fit a model. - -`Tutorial 7: Adaptive Pixelization `_ -- A Voronoi mesh which adapts to the mass model's magnification. - -`Tutorial 8: Model Fit `_ -- An example modeling pipeline which uses an inversion. \ No newline at end of file diff --git a/scripts/howtogalaxy/chapter_4_pixelizations/__init__.py b/scripts/howtogalaxy/chapter_4_pixelizations/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/scripts/howtogalaxy/chapter_4_pixelizations/introduction b/scripts/howtogalaxy/chapter_4_pixelizations/introduction deleted file mode 100644 index d85c8059..00000000 --- a/scripts/howtogalaxy/chapter_4_pixelizations/introduction +++ /dev/null @@ -1,27 +0,0 @@ -So, we've learnt how to build pipelines that model galaxies and customize them to our science case. - -We consider how using search chaining, we can fix complex galaxies made of multiple components or multiple -galaxies at the same time. - -However, we saw a pretty huge barrier when trying to fit very complex galaxies, that it requires an unwieldy number -of non-linear parameters. The non-linear parameter space would become very complex, and even a well crafted pipeline -may not fix the problem. - -In this chapter, we'll learn about pixelizations and inversions. These tools allow us to reconstruct components of a -galaxy using a pixel-grid. This makes no assumption about the galaxy's morphology, breaking the simplifying assumptions -inherent to analytic light profiles (e.g. symmetry). - -Remarkably, pixels grids use just a couple of non-linear parameters, meaning the 30+ non-linear parameters we required -to fit complex galaxy components before are going to be reduced to just 3 or 4! - -By combining these fits with parametric model, we are therefore able to fit certain structures (e.g. a bulge and disk) -whilst using these non-parametric pixelizations to fit other components that may otherwise degrade the fit. - -In particular, you'll learn how we: - -1) Pixelize a galaxy into a set of pixels that define mappings to image pixels. -2) Invert this pixelization to fit the galaxy and thus reconstruct its light. -3) Apply a smoothness prior on the galaxy reconstruction, called 'regularization', to ensure the solution is physicag. -4) Apply this prior in a Bayesian framework to objectively quantify the galaxy reconstruction's log likelihood. -6) Use alternative pixelizations that use Voronoi pixels adapted to the galaxy's mass model. -7) Use these features in PyAutoGalaxy pipelines. \ No newline at end of file diff --git a/scripts/howtogalaxy/chapter_4_pixelizations/tutorial_1_pixelizations.py b/scripts/howtogalaxy/chapter_4_pixelizations/tutorial_1_pixelizations.py deleted file mode 100644 index 3c2adfd3..00000000 --- a/scripts/howtogalaxy/chapter_4_pixelizations/tutorial_1_pixelizations.py +++ /dev/null @@ -1,135 +0,0 @@ -""" -Tutorial 1: pixelizations -========================= - -In the previous chapters, we used light profiles to model the light of a galaxy, where the light profile was an -analytic description of how the luminosity varies as a function of radius. - -In this chapter, we are instead going to reconstruct the galaxy's light on a pixel-grid, and in this tutorial we will -learn how to create a pixelization in **PyAutoGalaxy**. - -__Contents__ - -**Initial Setup:** Create a grid for illustration. -**Mesh:** Set up a rectangular mesh for the pixelization. -**Wrap Up:** Summary of pixelization concepts. -""" - -# from autoconf import setup_notebook; setup_notebook() - -import autogalaxy as ag -import autogalaxy.plot as aplt -from autoarray.inversion.plot.mapper_plots import plot_mapper - -""" -__Initial Setup__ - -Lets setup a grid. -""" -grid = ag.Grid2D.uniform(shape_native=(100, 100), pixel_scales=0.05) - -""" -__Mesh__ - -Next, lets set up a `Mesh` using the `mesh` module. The mesh represents the pixel-grid used by the pixelization -to reconstruct the galaxy. - -There are multiple `Mesh`'s available in **PyAutoGalaxy**. For now, we'll keep it simple and use a uniform -rectangular grid, whose `shape` defines its $(y,x)$ dimensions. We will make it the same shape as the 2D grid. -""" -mesh = ag.mesh.RectangularAdaptDensity(shape=(100, 100)) - -""" -We now pass the mesh to a `Pixelization`. -""" -pixelization = ag.Pixelization(mesh=mesh) - -""" -By itself, a pixelization does not tell us much. It has no grid of $(y,x)$ coordinates, no image, and no information -about the galaxy we are fitting. - -This information comes when we use the pixelization to create up a `Mapper`, which we perform below using the grid -that we created above. -""" -interpolator = mesh.interpolator_from( - source_plane_data_grid=grid, source_plane_mesh_grid=None -) - -mapper = ag.Mapper(interpolator=interpolator) - -""" -This `Mapper` is a `RectangularMapper` -- every `Mesh` and `Pixelization` generates it owns mapper. -""" -print(type(mapper)) - -""" -By plotting our mapper, we now see our `Pixelization`. Its a fairly boring grid of rectangular pixels. -""" -plot_mapper( - mapper=mapper, title="Fairly Boring Grid2D of RectangularAdaptDensity Pixels" -) - -""" -However, the `Mapper` does contain lots of interesting information about our `Pixelization`, for example its -pixelization_grid tells us where the pixel centers are located. -""" -print("RectangularAdaptDensity Grid2D Pixel Centre 1:") -print(mapper.source_plane_mesh_grid[0]) -print("RectangularAdaptDensity Grid2D Pixel Centre 2:") -print(mapper.source_plane_mesh_grid[1]) -print("RectangularAdaptDensity Grid2D Pixel Centre 3:") -print(mapper.source_plane_mesh_grid[2]) -print("etc.") - -""" -We can plot these centre on our grid, to make it look slightly less boring! -""" -plot_mapper( - mapper=mapper, - mesh_grid=mapper.source_plane_mesh_grid, - title="Recntagular Grid With Pixel Cenres", -) - -""" -The `Mapper` also has the grid that we passed when we set it up. Lets check they`re the same. -""" -print("Source Grid2D Pixel 1") -print(grid[0]) -print(mapper.source_plane_data_grid[0]) -print("Source Grid2D Pixel 2") -print(grid[1]) -print(mapper.source_plane_data_grid[1]) -print("etc.") - -""" -We can over-lay this grid on the figure, which is starting to look a bit less boring now! -""" -plot_mapper( - mapper=mapper, - mesh_grid=mapper.source_plane_data_grid, - title="Even less Boring Grid2D of RectangularAdaptDensity Pixels", -) - -plot_mapper( - mapper=mapper, - mesh_grid=mapper.source_plane_data_grid, - title="Zoomed Grid2D of RectangularAdaptDensity Pixels", -) - -""" -Finally, the mapper`s `mesh_grid` has lots of information about the pixelization, for example, the arc-second -size and dimensions. -""" -print(mapper.source_plane_mesh_grid.geometry.shape_native_scaled) -print(mapper.source_plane_mesh_grid.geometry.scaled_maxima) -print(mapper.source_plane_mesh_grid.geometry.scaled_minima) - -""" -__Wrap Up__ - -This was a relatively gentle overview of pixelizations, but one that was hopefully easy to follow. Think about the -following questions before moving on to the next tutorial: - - 1) The rectangular pixelization`s edges are aligned with the most exterior coordinates of the source-grid. This is - intentional, why do you think this is? -""" diff --git a/scripts/howtogalaxy/chapter_4_pixelizations/tutorial_2_mappers.py b/scripts/howtogalaxy/chapter_4_pixelizations/tutorial_2_mappers.py deleted file mode 100644 index 9ab375ae..00000000 --- a/scripts/howtogalaxy/chapter_4_pixelizations/tutorial_2_mappers.py +++ /dev/null @@ -1,166 +0,0 @@ -""" -Tutorial 2: Mappers -=================== - -In the previous tutorial, we used a pixelization to create made a `Mapper`. However, it was not clear what a `Mapper` -does, why it was called a mapper and whether it was mapping anything at all! - -Therefore, in this tutorial, we'll cover mappers in more detail. - -__Contents__ - -**Initial Setup:** Load the dataset for illustration. -**Mappers:** Understand how mappers map image-plane pixels to pixelization pixels. -**Mask:** Apply a mask and see how it affects the mapper. -**Wrap Up:** Summary of mapper concepts. -""" - -# from autoconf import setup_notebook; setup_notebook() - -from pathlib import Path -import autogalaxy as ag -import autogalaxy.plot as aplt -import autoarray.plot as aaplt - -""" -__Initial Setup__ - -we'll use complex galaxy data, where: - - - The galaxy's bulge is an `Sersic`. - - The galaxy's disk is an `Exponential`. - - The galaxy's has four star forming clumps which are `Sersic` profiles. -""" -dataset_name = "simple" -dataset_path = Path("dataset") / "imaging" / dataset_name - -""" -__Dataset Auto-Simulation__ - -If the dataset does not already exist on your system, it will be created by running the corresponding -simulator script. This ensures that all example scripts can be run without manually simulating data first. -""" -if not dataset_path.exists(): - import subprocess - import sys - - subprocess.run( - [sys.executable, "scripts/imaging/simulator.py"], - check=True, - ) - -dataset = ag.Imaging.from_fits( - data_path=dataset_path / "data.fits", - noise_map_path=dataset_path / "noise_map.fits", - psf_path=dataset_path / "psf.fits", - pixel_scales=0.1, -) - -""" -Now, lets set up our `Grid2D` (using the image above). -""" -grid = ag.Grid2D.uniform( - shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales -) - -""" -__Mappers__ - -We now setup a `Pixelization` and use it to create a `Mapper` via the plane`s source-plane grid, just like we did in -the previous tutorial. - -We will make its pixelization resolution half that of the grid above. -""" -mesh = ag.mesh.RectangularAdaptDensity( - shape=(dataset.shape_native[0] / 2, dataset.shape_native[1] / 2) -) - -pixelization = ag.Pixelization(mesh=mesh) - -interpolator = mesh.interpolator_from( - source_plane_data_grid=grid, source_plane_mesh_grid=None -) - -mapper = ag.Mapper(interpolator=interpolator) - -""" -We now plot the `Mapper` alongside the image we used to generate the source-plane grid. - -Using the `Visuals2D` object we are also going to highlight specific grid coordinates certain colors, such that we -can see how they map from the image grid to the pixelization grid. -""" -indexes = [range(250), [150, 250, 350, 450, 550, 650, 750, 850, 950, 1050]] - -aaplt.subplot_image_and_mapper(mapper=mapper, image=dataset.data) - - -""" -Using a mapper, we can now make these mappings appear the other way round. That is, we can input a pixelization pixel -index (of our rectangular grid) and highlight how all of the image-pixels that it contains map to the image-plane. - -Lets map source pixel 313, the central source-pixel, to the image. We observe that for a given rectangular pixelization -pixel, there are four image pixels. -""" -pix_indexes = [[312]] - -indexes = mapper.slim_indexes_for_pix_indexes(pix_indexes=pix_indexes) - -aaplt.subplot_image_and_mapper(mapper=mapper, image=dataset.data) - -""" -Okay, so I think we can agree, mapper's map things! More specifically, they map pixelization pixels to multiple pixels -in the observed image of a galaxy. - -__Mask__ - -Finally, lets repeat the steps that we performed above, but now using a masked image. By applying a `Mask2D`, the -mapper only maps image-pixels that are not removed by the mask. This removes the (many) image pixels at the edge of the -image, where the galaxy is not present. - -Lets just have a quick look at these edges pixels: - -Lets use an circular `Mask2D`, which will capture the central galaxy light and clumps. -""" -mask = ag.Mask2D.circular( - shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=2.0 -) - -dataset = dataset.apply_mask(mask=mask) -aplt.plot_array(array=dataset.data, title="Data") - -""" -We can now use the masked grid to create a new `Mapper` (using the same rectangular 25 x 25 pixelization -as before). -""" -interpolator = mesh.interpolator_from( - source_plane_data_grid=dataset.grids.pixelization, source_plane_mesh_grid=None -) -mapper = ag.Mapper(interpolator=interpolator) - -""" -Lets plot it. -""" -aaplt.subplot_image_and_mapper(mapper=mapper, image=dataset.data) - -""" -First, We can see a red circle of dots in both the image and pixelization, showing where the edge of the mask -maps too in the pixelization. - -Now lets show that when we plot pixelization pixel indexes, they still appear in the same place in the image. -""" -pix_indexes = [[312], [314], [316], [318]] - -indexes = mapper.slim_indexes_for_pix_indexes(pix_indexes=pix_indexes) - -aaplt.subplot_image_and_mapper(mapper=mapper, image=dataset.data) - -""" -__Wrap Up__ - -In this tutorial, we learnt about mappers, and we used them to understand how the image and pixelization map to one -another. Your exercises are: - - 1) Think about how this could help us actually model galaxies. We have said we're going to reconstruct our galaxies - on the pixel-grid. So, how does knowing how each pixel maps to the image actually help us? If you`ve not got - any bright ideas, then worry not, that exactly what we're going to cover in the next tutorial. -""" diff --git a/scripts/howtogalaxy/chapter_4_pixelizations/tutorial_3_inversions.py b/scripts/howtogalaxy/chapter_4_pixelizations/tutorial_3_inversions.py deleted file mode 100644 index 139248e4..00000000 --- a/scripts/howtogalaxy/chapter_4_pixelizations/tutorial_3_inversions.py +++ /dev/null @@ -1,225 +0,0 @@ -""" -Tutorial 3: Inversions -====================== - -In the previous two tutorials, we introduced: - - - `Pixelization`'s: which place a pixel-grid over the image data. - - `Mappers`'s: which describe how each pixelization pixel maps to one or more image pixels. - -However, non of this has actually helped us fit galaxy data or reconstruct the galaxy. This is the subject -of this tutorial, where the process of reconstructing the galaxy's light on the pixelization is called an `Inversion`. - -__Contents__ - -**Initial Setup:** Load the dataset for illustration. -**Pixelization:** Create a pixelization and perform an inversion to reconstruct the galaxy. -**Positive Only Solver:** Ensure the reconstruction has only positive intensity values. -**Wrap Up:** Summary of inversion concepts. -**Detailed Explanation:** In-depth explanation of the linear algebra behind inversions. -""" - -# from autoconf import setup_notebook; setup_notebook() - -from pathlib import Path -import autogalaxy as ag -import autogalaxy.plot as aplt -from autoarray.inversion.plot.inversion_plots import subplot_of_mapper - -""" -__Initial Setup__ - -we'll use the same complex galaxy data as the previous tutorial, where: - - - The galaxy's bulge is an `Sersic`. - - The galaxy's disk is an `Exponential`. - - The galaxy's has four star forming clumps which are `Sersic` profiles. -""" -dataset_name = "simple" -dataset_path = Path("dataset") / "imaging" / dataset_name - -""" -__Dataset Auto-Simulation__ - -If the dataset does not already exist on your system, it will be created by running the corresponding -simulator script. This ensures that all example scripts can be run without manually simulating data first. -""" -if not dataset_path.exists(): - import subprocess - import sys - - subprocess.run( - [sys.executable, "scripts/imaging/simulator.py"], - check=True, - ) - -dataset = ag.Imaging.from_fits( - data_path=dataset_path / "data.fits", - noise_map_path=dataset_path / "noise_map.fits", - psf_path=dataset_path / "psf.fits", - pixel_scales=0.1, -) - -""" -Lets create a circular mask which contains the galaxy's emission: -""" -mask = ag.Mask2D.circular( - shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=2.0 -) - -aplt.plot_array(array=dataset.data, title="Data", mask=mask) - -""" -We now create the masked imaging, as we did in the previous tutorial. -""" -dataset = dataset.apply_mask(mask=mask) - -""" -we again use the rectangular pixelization to create the mapper. - -(Ignore the regularization input below for now, we will cover this in the next tutorial). -""" -mesh = ag.mesh.RectangularAdaptDensity(shape=dataset.shape_native) - -pixelization = ag.Pixelization(mesh=mesh) - -interpolator = mesh.interpolator_from( - source_plane_data_grid=dataset.grids.pixelization, - source_plane_mesh_grid=None, -) - -mapper = ag.Mapper( - interpolator=interpolator, - regularization=ag.reg.Constant(coefficient=1.0), -) - -aplt.subplot_image_and_mapper(mapper=mapper, image=dataset.data) - -""" -__Pixelization__ - -Finally, we can now use the `Mapper` to reconstruct the galaxy via an `Inversion`. I'll explain how this works in a -second, but lets just go ahead and create the inversion first. -""" -inversion = ag.Inversion(dataset=dataset, linear_obj_list=[mapper]) - -""" -The inversion has reconstructed the galaxy's light on the rectangular pixel grid, which is called the -`reconstruction`. - -This reconstruction can be mapped back to the same resolution as the image to produce the `mapped_reconstructed_operated_data`. -""" -print(inversion.reconstruction) -print(inversion.mapped_reconstructed_operated_data) - -""" -Both of these can be plotted using `subplot_of_mapper`. - -It is possible for an inversion to have multiple `Mapper`'s, therefore for certain figures we specify the index -of the mapper we wish to plot. In this case, because we only have one mapper we specify the index 0. -""" -aplt.plot_array(array=inversion.reconstruction_to_native, title="Reconstruction") -subplot_of_mapper(inversion=inversion, mapper_index=0) - -""" -There we have it, we have successfully reconstructed the galaxy using a rectangular pixel-grid. This has reconstructed -the complex blobs of light of the galaxy. - -Pretty great, huh? If you ran the complex source pipeline in chapter 3, you'll remember that getting a model image -that looked this good simply *was not possible*. With an inversion, we can do this with ease and without having to -perform model-fitting with 20+ parameters for the galaxy's light! - -We will now briefly discuss how an inversion actually works, however the explanation I give in this tutorial will be -overly-simplified. To be good at modeling you do not need to understand the details of how an inversion works, you -simply need to be able to use an inversion to model a galaxy. - -To begin, lets consider some random mappings between our mapper`s pixelization pixels and the image. -""" -pix_indexes = [[445], [285], [313], [132], [11]] - -indexes = mapper.slim_indexes_for_pix_indexes(pix_indexes=pix_indexes) - -aplt.subplot_image_and_mapper(mapper=mapper, image=dataset.data) - -""" -These mappings are known before the inversion reconstructs the galaxy, which means before this inversion is performed -we know two key pieces of information: - - 1) The mappings between every pixelization pixel and a set of image-pixels. - 2) The flux values in every observed image-pixel, which are the values we want to fit successfully. - -It turns out that with these two pieces of information we can linearly solve for the set of pixelization pixel fluxes -that best-fit (e.g. maximize the log likelihood) our observed image. Essentially, we set up the mappings between -pixelization and image pixels as a large matrix and solve for the pixelization pixel fluxes in an analogous fashion to -how you would solve a set of simultaneous linear equations. This process is called a `linear inversion`. - -There are three more things about a linear inversion that are worth knowing: - - 1) When performing fits using light profiles, we discussed how a `model_image` was generated by convolving the light - profile images with the data's PSF. A similar blurring operation is incorporated into the inversion, such that it - reconstructs a galaxy (and therefore image) which fully accounts for the telescope optics and effect of the PSF. - - 2) You may be familiar with image sub-gridding, which splits each image-pixel into a sub-pixel (if you are not - familiar then feel free to checkout the optional **HowToGalaxy** tutorial on sub-gridding. If a sub-grid is used, it is - the mapping between every sub-pixel -pixel that is computed and used to perform the inversion. This prevents - aliasing effects degrading the image reconstruction. By default **PyAutoGalaxy** uses sub-gridding of degree 4x4. - - 3) The inversion`s solution is regularized. But wait, that`s what we'll cover in the next tutorial! - -Finally, let me show you how easy it is to fit an image with an `Inversion` using a `FitImaging` object. Instead of -giving the galaxy a light profile, we simply pass it a `Pixelization` and regularization, and pass it to a -galaxies. -""" -pixelization = ag.Pixelization( - mesh=ag.mesh.RectangularAdaptDensity(shape=(25, 25)), - regularization=ag.reg.Constant(coefficient=1.0), -) - -galaxy = ag.Galaxy(redshift=1.0, pixelization=pixelization) - -galaxies = ag.Galaxies(galaxies=[galaxy]) - -""" -Then, like before, we pass the imaging and galaxies `FitImaging` object. - -We see some pretty good looking residuals, albeit there is faint flux leftover. We will consider how we can address -this in the next tutorial. - -We can use the `subplot_of_galaxies` method to specifically visualize the inversion and plot the reconstruction. -""" -fit = ag.FitImaging(dataset=dataset, galaxies=galaxies) - -aplt.subplot_fit_imaging(fit=fit) - -""" -__Positive Only Solver__ - -All pixelized source reconstructions use a positive-only solver, meaning that every source-pixel is only allowed -to reconstruct positive flux values. This ensures that the source reconstruction is physical and that we don't -reconstruct negative flux values that don't exist in the real source galaxy (a common systematic solution in lens -analysis). - -It may be surprising to hear that this is a feature worth pointing out, but it turns out setting up the linear algebra -to enforce positive reconstructions is difficult to make efficient. A lot of development time went into making this -possible, where a bespoke fast non-negative linear solver was developed to achieve this. - -Other methods in the literature often do not use a positive only solver, and therefore suffer from these -unphysical solutions, which can degrade the results of lens model in general. - -__Wrap Up__ - -And, we're done, here are a few questions to get you thinking about inversions: - - 1) The inversion provides the maximum log likelihood solution to the observed image. Is there a problem with seeking - the highest likelihood solution? Is there a risk that we're going to fit other things in the image than just the - galaxy? What happens if you reduce the `coefficient` of the regularization object above to zero? - - 2) The exterior pixels in the rectangular pixel-grid have no image-pixels in them. However, they are still given a - reconstructed flux. Given these pixels do not map to the data, where is this value coming from? - -__Detailed Explanation__ - -If you are interested in a more detailed description of how inversions work, then checkout the file -`autogalaxy_workspace/*/imaging/log_likelihood_function/inversion.ipynb` which gives a visual step-by-step -guide of the process alongside equations and references to literature on the subject. -""" diff --git a/scripts/howtogalaxy/chapter_4_pixelizations/tutorial_4_bayesian_regularization.py b/scripts/howtogalaxy/chapter_4_pixelizations/tutorial_4_bayesian_regularization.py deleted file mode 100644 index e32b4af8..00000000 --- a/scripts/howtogalaxy/chapter_4_pixelizations/tutorial_4_bayesian_regularization.py +++ /dev/null @@ -1,289 +0,0 @@ -""" -Tutorial 4: Bayesian Regularization -=================================== - -So far, we have: - - - Used pixelizations and mappers to map pixelization pixels to image-pixels and visa versa. - - Successfully used an inversion to reconstruct a galaxy. - - Seen that this reconstruction provides a good fit of the observed image, providing a high likelihood solution. - -The explanation of *how* an inversion works has so far been overly simplified. You'll have noted the regularization -inputs which we have not so far discussed. This will be the topic of this tutorial, and where inversions become more -conceptually challenging! - -__Contents__ - -**Initial Setup:** Load the dataset for illustration. -**Convenience Function:** A helper function for performing inversions. -**Pixelization:** Perform inversions with different regularization coefficients. -**Regularization:** Understand how regularization smooths the reconstruction. -**Bayesian Evidence:** Use the Bayesian evidence to objectively choose the regularization coefficient. -**Non-Linear and Linear:** Discussion of how regularization interacts with the non-linear search. -**Detailed Description:** In-depth explanation of how the Bayesian evidence penalizes overfitting. -""" - -# from autoconf import setup_notebook; setup_notebook() - -from pathlib import Path -import autogalaxy as ag -import autogalaxy.plot as aplt -from autoarray.inversion.plot.inversion_plots import subplot_of_mapper - -""" -__Initial Setup__ - -we'll use the same complex galaxy data as the previous tutorial, where: - - - The galaxy's bulge is an `Sersic`. - - The galaxy's disk is an `Exponential`. - - The galaxy's has four star forming clumps which are `Sersic` profiles. -""" -dataset_name = "simple" -dataset_path = Path("dataset") / "imaging" / dataset_name - -""" -__Dataset Auto-Simulation__ - -If the dataset does not already exist on your system, it will be created by running the corresponding -simulator script. This ensures that all example scripts can be run without manually simulating data first. -""" -if not dataset_path.exists(): - import subprocess - import sys - - subprocess.run( - [sys.executable, "scripts/imaging/simulator.py"], - check=True, - ) - -dataset = ag.Imaging.from_fits( - data_path=dataset_path / "data.fits", - noise_map_path=dataset_path / "noise_map.fits", - psf_path=dataset_path / "psf.fits", - pixel_scales=0.1, -) - -""" -__Convenience Function__ - -we're going to perform a lot of fits using an `Inversion` this tutorial. This would create a lot of code, so to keep -things tidy, I've setup this function which handles it all for us. -""" - - -def perform_fit_with_galaxy(dataset, galaxy): - mask = ag.Mask2D.circular( - shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=2.0 - ) - - dataset = dataset.apply_mask(mask=mask) - - galaxies = ag.Galaxies(galaxies=[galaxy]) - - return ag.FitImaging(dataset=dataset, galaxies=galaxies) - - -""" -__Pixelization__ - -Okay, so lets look at our fit from the previous tutorial in more detail. -""" -pixelization = ag.Pixelization( - mesh=ag.mesh.RectangularAdaptDensity(shape=(50, 50)), - regularization=ag.reg.Constant(coefficient=1.0), -) - -galaxy = ag.Galaxy(redshift=1.0, pixelization=pixelization) - -fit = perform_fit_with_galaxy(dataset=dataset, galaxy=galaxy) - -aplt.subplot_fit_imaging(fit=fit) -subplot_of_mapper(inversion=fit.inversion, mapper_index=0) - -""" -__Regularization__ - -The galaxy reconstruction looks pretty good! - -However, the high quality of this solution was possible because I chose a `coefficient` for the regularization input of -1.0. If we reduce this `coefficient` to 0.01, the galaxy reconstruction goes *very* weird. -""" -pixelization = ag.Pixelization( - mesh=ag.mesh.RectangularAdaptDensity(shape=(50, 50)), - regularization=ag.reg.Constant(coefficient=0.01), -) - -galaxy = ag.Galaxy(redshift=1.0, pixelization=pixelization) - -no_regularization_fit = perform_fit_with_galaxy(dataset=dataset, galaxy=galaxy) - -aplt.subplot_fit_imaging(fit=no_regularization_fit) -subplot_of_mapper(inversion=no_regularization_fit.inversion, mapper_index=0) - -""" -So, what is happening here? Why does reducing the `coefficient` do this to our reconstruction? First, we need -to understand what regularization actually does! - -When the inversion reconstructs the galaxy, it does not *only* compute the set of pixelization pixel fluxes that -best-fit the image. It also regularizes this solution, whereby it goes to every pixel on the rectangular grid -and computes the different between the reconstructed flux values of every pixel with its 4 neighboring pixels. -If the difference in flux is large the solution is penalized, reducing its log likelihood. You can think of this as -us applying a 'smoothness prior' on the reconstructed galaxy's light. - -This smoothing adds a 'penalty term' to the log likelihood of an inversion which is the summed difference between the -reconstructed fluxes of every pixelization pixel pair multiplied by the `coefficient`. By setting the regularization -coefficient to zero, we set this penalty term to zero, meaning that regularization is completely omitted. - -Why do we need to regularize our solution? We just saw why, if we do not apply this smoothness prior to the galaxy -reconstruction, we `over-fit` the image and reconstruct a noisy galaxy with lots of extraneous features. This is what -the aliasing chequer-board effect is caused by. If the inversions's sole aim is to maximize the log likelihood, it can -do this by fitting *everything* accurately, including the noise. - -Over-fitting is why regularization is necessary. Solutions like this will completely ruin our attempts to model a -galaxy. By smoothing our galaxy reconstruction we ensure it does not over fit noise in the image. - -So, what happens if we apply a high value for the regularization coefficient? -""" -pixelization = ag.Pixelization( - mesh=ag.mesh.RectangularAdaptDensity(shape=(50, 50)), - regularization=ag.reg.Constant(coefficient=100.0), -) - -galaxy = ag.Galaxy(redshift=1.0, pixelization=pixelization) - -high_regularization_fit = perform_fit_with_galaxy(dataset=dataset, galaxy=galaxy) - -aplt.subplot_fit_imaging(fit=high_regularization_fit) -subplot_of_mapper(inversion=high_regularization_fit.inversion, mapper_index=0) - -""" -The figure above shows that we completely remove over-fitting. However, we now fit the image data less poorly, -due to the much higher level of smoothing. - -So, we now understand what regularization is and why it is necessary. There is one nagging question that remains, how -do I choose the regularization coefficient value? We can not use the log likelihood, as decreasing the regularization -coefficient will always increase the log likelihood, because less smoothing allows the reconstruction to fit -the data better. -""" -print("Likelihood Without Regularization:") -print(no_regularization_fit.log_likelihood_with_regularization) -print("Likelihood With Normal Regularization:") -print(fit.log_likelihood_with_regularization) -print("Likelihood With High Regularization:") -print(high_regularization_fit.log_likelihood_with_regularization) - -""" -__Bayesian Evidence__ - -For inversions, we therefore need a different goodness-of-fit measure to choose the appropriate level of regularization. - -For this, we invoke the `Bayesian Evidence`, which quantifies the goodness of the fit as follows: - - - It requires that the residuals of the fit are consistent with Gaussian noise (which is the type of noise expected - in the imaging data). If this Gaussian pattern is not visible in the residuals, the noise must have been over-fitted - by the inversion. The Bayesian evidence will therefore decrease. If the image is fitted poorly due to over smoothing, - the residuals will again not appear Gaussian either, again producing a decrease in the Bayesian evidence value. - - - There can be many solutions which fit the data to the noise level, without over-fitting. To determine the best - solutions from these solutions, the Bayesian evidence therefore also quantifies the complexity of the galaxy - reconstruction. If an inversion requires many pixels and a low level of regularization to achieve a good fit, the - Bayesian evidence will decrease. The evidence penalizes solutions which are complex, which, in a Bayesian sense, are - less probable (you may want to look up `Occam`s Razor`). - -The Bayesian evidence therefore ensures we only invoke a more complex galaxy reconstruction when the data absolutely -necessitates it. - -Lets take a look at the Bayesian evidence of the fits that we performed above, which is accessible from a `FitImaging` -object via the `log_evidence` property: -""" -print("Bayesian Evidence Without Regularization:") -print(no_regularization_fit.log_evidence) -print("Bayesian Evidence With Normal Regularization:") -print(fit.log_evidence) -print("Bayesian Evidence With High Regularization:") -print(high_regularization_fit.log_evidence) - -""" -As expected, the solution that we could see `by-eye` was the best solution corresponds to the highest log evidence -solution. - -__Non-Linear and Linear__ - -Before we end, lets consider which aspects of an inversion are linear and which are non-linear. - -The linear part of the inversion is the step that solves for the reconstruct pixelization pixel fluxes, including -accounting for the smoothing via regularizaton. We do not have to perform a non-linear search to determine the pixel -fluxes or compute the Bayesian evidence discussed above. - -However, determining the regularization `coefficient` that maximizes the Bayesian log evidence is a non-linear problem -that requires a non-linear search. The Bayesian evidence also depends on the grid resolution, which means the -pixel-grid's `shape` parameter may also now become dimensions of non linear parameter space (albeit it is common -practise for us to simply use the resolution of the image data, or a multiple of this). - -Nevertheless, these total only 3 non-linear parameters, far fewer than the 20+ that are required when modeling such a -complex galaxy using light profiles for every individual clump! - -Here are a few questions for you to think about. - - 1) We maximize the log evidence by using simpler galaxy reconstructions. Therefore, decreasing the pixel-grid - size should provide a higher log_evidence, provided it still has sufficiently high resolution to fit the image well - (and provided that the regularization coefficient is set to an appropriate value). Can you increase the log evidence - from the value above by changing these parameters, I've set you up with a code to do so below. -""" -pixelization = ag.Pixelization( - mesh=ag.mesh.RectangularAdaptDensity(shape=(50, 50)), - regularization=ag.reg.Constant(coefficient=1.0), -) - -galaxy = ag.Galaxy(redshift=1.0, pixelization=pixelization) - -fit = perform_fit_with_galaxy(dataset=dataset, galaxy=galaxy) - -print("Previous Bayesian Evidence:") -print(3988.0716851250163) -print("New Bayesian Evidence:") -print(fit.log_evidence) - -aplt.subplot_fit_imaging(fit=fit) - -""" -__Detailed Description__ - -Below, I provide a more detailed discussion of the Bayesian evidence. It is not paramount that you understand this to -use **PyAutoGalaxy**, but I recommend you give it a read to get an intuition for how the evidence works. - -The Bayesian log evidence quantifies the following 3 aspects of a fit to galaxy imaging data: - -1) *The quality of the image reconstruction:* The galaxy reconstruction is a linear inversion which uses the observed - values in the image-data to fit it and reconstruct the galaxy. It is in principle able to perfectly reconstruct the - image regardless of the image’s noise or the accuracy of the model (e.g. at infinite resolution without - regularization). The problem is therefore ‘ill-posed’ and this is why regularization is necessary. - - However, this raises the question of what constitutes a ‘good’ solution? The Bayesian evidence defines this by - assuming that the image data consists of independent Gaussian noise in every image pixel. A ‘good’ solution is one - whose chi-squared residuals are consistent with Gaussian noise, producing a reduced chi-squared near 1.0 .Solutions - which give a reduced chi squared below 1 are penalized for being overly complex and fitting the image’s noise, whereas - solutions with a reduced chi-squared above are penalized for not invoking a more complex galaxy model when the data it - is necessary to fit the data bettter. In both circumstances, these penalties reduce the inferred Bayesian evidence! - -2) *The complexity of the galaxy reconstruction:* The log evidence estimates the number of pixelization pixels that are used - to reconstruct the image, after accounting for their correlation with one another due to regularization. Solutions that - require fewer correlated galaxy pixels increase the Bayesian evidence. Thus, simpler and less complex galaxy - reconstructions are favoured. - -3) *The signal-to-noise (S/N) of the image that is fitted:* The Bayesian evidence favours models which fit higher S/N - realizations of the observed data (where the S/N is determined using the image-pixel variances, e.g. the noise-map). Up - to now, all **PyAutoGalaxy** fits assumed fixed variances, meaning that this aspect of the Bayeisan evidence has no impact - on the inferred evidence values. - - The premise is that whilst increasing the variances of image pixels lowers their S/N values and therefore also - decreases the log evidence, doing so may produce a net increase in log evidence. This occurs when the chi-squared - values of the image pixels whose variances are increased were initially very high (e.g. they were fit poorly by the - model). - -In summary, the log evidence is maximized for solutions which most accurately reconstruct the highest S/N realization of -the observed image, without over-fitting its noise and using the fewest correlated pixelization pixels. By employing -this framework throughout, **PyAutoGalaxy** objectively determines the final model following the principles of Bayesian -analysis and Occam’s Razor. -""" diff --git a/scripts/howtogalaxy/chapter_4_pixelizations/tutorial_5_model_fit.py b/scripts/howtogalaxy/chapter_4_pixelizations/tutorial_5_model_fit.py deleted file mode 100644 index 899bbc75..00000000 --- a/scripts/howtogalaxy/chapter_4_pixelizations/tutorial_5_model_fit.py +++ /dev/null @@ -1,246 +0,0 @@ -""" -Tutorial 5: Model-Fit -===================== - -In the previous tutorials we used an inversion to reconstruct a complex galaxy. However, from the perspective of -a scientific analysis, it is not clear how useful this was. When I fit a galaxy with light profiles, I learn about -its brightness (`intensity`), size (`effective_radius`), compactness (`sersic_index`), etc. - -What did I learn about the galaxy I reconstructed? Not a lot, perhaps. - -Inversions are most useful when combined with light profiles. For the complex galaxy we used throughout this tutorial, -we can fit it with light profiles to quantify the properties of its `bulge` and `disk` components, whilst -simultaneously fitting the clumps with the inversion so as to ensure they do not impact the fit. - -To illustrate modeling using an inversion this tutorial therefore revisits the complex galaxy model-fit that we -performed in tutorial 4 of chapter 3. This time, as you have probably guessed, we will fit part of the complex galaxy -using an inversion. - -We will use search chaining to do this, first fitting the main galaxy components with light profiles, thereby -initializing the bulge and disk components. In the later searches we will switch to an `Inversion`. - -__Contents__ - -**Initial Setup:** Load the complex galaxy dataset and apply a mask. -**Model + Search + Analysis + Model-Fit (Search 1):** Fit light profiles to the main galaxy components. -**Mesh Shape:** Discussion of how mesh shape affects the inversion. -**Model + Search + Analysis + Model-Fit (Search 2):** Fit with a pixelization for residual structure. -**Model + Search (Search 3):** Final simultaneous fit of light profiles and pixelization. -**Wrap Up:** Summary of combining light profiles with pixelizations via search chaining. -""" - -# from autoconf import setup_notebook; setup_notebook() - -from pathlib import Path -import autofit as af -import autogalaxy as ag -import autogalaxy.plot as aplt - -""" -__Initial Setup__ - -we'll use complex galaxy data, where: - - - The galaxy's bulge is an `Sersic`. - - The galaxy's disk is an `Exponential`. - - The galaxy's has four star forming clumps which are `Sersic` profiles. -""" -dataset_name = "simple" -dataset_path = Path("dataset") / "imaging" / dataset_name - -""" -__Dataset Auto-Simulation__ - -If the dataset does not already exist on your system, it will be created by running the corresponding -simulator script. This ensures that all example scripts can be run without manually simulating data first. -""" -if not dataset_path.exists(): - import subprocess - import sys - - subprocess.run( - [sys.executable, "scripts/imaging/simulator.py"], - check=True, - ) - -dataset = ag.Imaging.from_fits( - data_path=dataset_path / "data.fits", - noise_map_path=dataset_path / "noise_map.fits", - psf_path=dataset_path / "psf.fits", - pixel_scales=0.1, -) - -mask = ag.Mask2D.circular( - shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=2.0 -) - - -dataset = dataset.apply_mask(mask=mask) - -over_sample_size = ag.util.over_sample.over_sample_size_via_radial_bins_from( - grid=dataset.grid, - sub_size_list=[8, 4, 1], - radial_list=[0.3, 0.6], - centre_list=[(0.0, 0.0)], -) - -dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size) - -aplt.subplot_imaging_dataset(dataset=dataset) - - -""" -__Model + Search + Analysis + Model-Fit (Search 1)__ - -Search 1 we fit a model where: - - - The galaxy's bulge is an `Sersic` with fixed centre [5 parameters]. - - - The galaxy's disk is an `Exponential` with fixed centre [4 parameters]. - -The number of free parameters and therefore the dimensionality of non-linear parameter space is N=9. -""" -bulge = af.Model(ag.lp.Sersic) -disk = af.Model(ag.lp.Exponential) - -bulge.centre_0 = 0.0 -bulge.centre_1 = 0.0 -disk.centre_0 = 0.0 -disk.centre_1 = 0.0 - -galaxy = af.Model(ag.Galaxy, redshift=0.5, bulge=bulge, disk=disk) - -model_1 = af.Collection(galaxies=af.Collection(galaxy=galaxy)) - -search_1 = af.Nautilus( - path_prefix=Path("howtogalaxy", "chapter_4"), - name="search[1]", - unique_tag=dataset_name, - n_live=100, - n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. -) - -analysis_1 = ag.AnalysisImaging(dataset=dataset, use_jax=True) - -result_1 = search_1.fit(model=model_1, analysis=analysis_1) - -""" -__Mesh Shape__ - -The `mesh_shape` parameter defines number of pixels used by the rectangular mesh to reconstruct the source, -set below to 28 x 28. - -The `mesh_shape` must be fixed before modeling and cannot be a free parameter of the model, because JAX uses the -mesh shape to define static shaped arrays which use the mesh to reconstruct the source. For a rectangular -mesh, the same number of pixels must be used in the y and x directions. -""" -mesh_pixels_yx = 28 -mesh_shape = (mesh_pixels_yx, mesh_pixels_yx) - -""" -__Model + Search + Analysis + Model-Fit (Search 2)__ - -We use the results of search 1 to create the model fitted in search 2, where: - - - The galaxy's bulge is an `Sersic` [0 parameters: parameters fixed from search 1]. - - - The galaxy's disk is an `Exponential` [0 parameters: parameters fixed from search 1]. - - - The galaxy's clumps are reconstructed `RectangularAdaptDensity` mesh with resolution as free parameters [2 parameters]. - - - This pixelization is regularized using a `Constant` scheme [1 parameter]. - -The number of free parameters and therefore the dimensionality of non-linear parameter space is N=3. - -This search allows us to very efficiently set up the resolution of the mesh and regularization coefficient -of the regularization scheme, before using these models to refit the galaxy mass model. -""" -pixelization = af.Model( - ag.Pixelization, - mesh=ag.mesh.RectangularAdaptDensity(shape=mesh_shape), - regularization=ag.reg.GaussianKernel, -) - -galaxy = af.Model( - ag.Galaxy, - redshift=0.5, - bulge=result_1.instance.galaxies.galaxy.bulge, - disk=result_1.instance.galaxies.galaxy.disk, - pixelization=pixelization, -) - -model_2 = af.Collection(galaxies=af.Collection(galaxy=galaxy)) - -search_2 = af.Nautilus( - path_prefix=Path("howtogalaxy", "chapter_4"), - name="search[2]", - unique_tag=dataset_name, - n_live=50, - n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. -) - -analysis_2 = ag.AnalysisImaging( - dataset=dataset, - settings=ag.Settings(use_border_relocator=True), - use_jax=True, -) - -result_2 = search_2.fit(model=model_2, analysis=analysis_2) - -""" -__Model + Search (Search 3)__ - -We use the results of searches 1 and 2 to create the model fitted in search 3, where: - - - The galaxy's bulge is an `Sersic` [7 parameters: priors initialized from search 1]. - - - The galaxy's disk is an `Exponential` [6 parameters: priors initialized from search 1]. - - - The galaxy's light uses a `RectangularAdaptDensity` mesh[parameters fixed to results of search 2]. - - - This pixelization is regularized using a `Constant` scheme [parameters fixed to results of search 2]. - -The number of free parameters and therefore the dimensionality of non-linear parameter space is N=13. - -This search allows us to refit the bulge and disk components with an inversion that takes care of the clumps. -""" -bulge = af.Model(ag.lp.Sersic) -bulge.ell_comps = result_1.model.galaxies.galaxy.bulge.ell_comps -bulge.intensity = result_1.model.galaxies.galaxy.bulge.intensity -bulge.effective_radius = result_1.model.galaxies.galaxy.bulge.effective_radius -bulge.sersic_index = result_1.model.galaxies.galaxy.bulge.sersic_index - -disk = af.Model(ag.lp.Sersic) -disk.ell_comps = result_1.model.galaxies.galaxy.disk.ell_comps -disk.intensity = result_1.model.galaxies.galaxy.disk.intensity -disk.effective_radius = result_1.model.galaxies.galaxy.disk.effective_radius - -pixelization = af.Model( - ag.Pixelization, - mesh=ag.mesh.RectangularAdaptDensity(shape=mesh_shape), - regularization=ag.reg.GaussianKernel, -) - -galaxy = af.Model( - ag.Galaxy, redshift=0.5, bulge=bulge, disk=disk, pixelization=pixelization -) - -model_3 = af.Collection(galaxies=af.Collection(galaxy=galaxy)) - -search_3 = af.Nautilus( - path_prefix=Path("howtogalaxy", "chapter_4"), - name="search[3]", - unique_tag=dataset_name, - n_live=100, - n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. -) - -analysis_3 = ag.AnalysisImaging(dataset=dataset, use_jax=True) - -result_3 = search_3.fit(model=model_3, analysis=analysis_3) - -""" -__Wrap Up__ - -And with that, we now have a pipeline to model galaxies using an inversion! -""" diff --git a/scripts/howtogalaxy/chapter_optional/__init__.py b/scripts/howtogalaxy/chapter_optional/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/scripts/howtogalaxy/chapter_optional/tutorial_searches.py b/scripts/howtogalaxy/chapter_optional/tutorial_searches.py deleted file mode 100644 index d41923df..00000000 --- a/scripts/howtogalaxy/chapter_optional/tutorial_searches.py +++ /dev/null @@ -1,298 +0,0 @@ -""" -Tutorial: Alternative Searches -============================== - -Up to now, we've always used the non-linear search Nautilus and not considered the input parameters that control its -sampling. In this tutorial, we'll consider how we can change these setting to balance finding the global maxima -solution with fast run time. - -We will also discuss other types of non-linear searches, such as MCMC and optimizers, which we can use to perform lens -modeling. So far, we have no found any of these alternatives to give anywhere near as robust and efficient results as -Nautilus, and we recommend users use Nautilus unless they are particularly interested in investigating different -model-fitting techniques. - -__Contents__ - -**Nested Sampling:** Customize Nautilus settings and use Dynesty as an alternative nested sampler. -**Optimizers:** Use maximum likelihood optimizers like LBFGS for fast but less robust fitting. -**MCMC:** Use Markov Chain Monte Carlo methods like Emcee for parameter estimation. -""" - -# from autoconf import setup_notebook; setup_notebook() - -from pathlib import Path -import autogalaxy as ag -import autogalaxy.plot as aplt -import autofit as af - -""" -we'll use new galaxying data, where: - - - The galaxy's light is an `Sersic`. - - The galaxy's total mass distribution is an `Isothermal` and `ExternalShear`. - - The source galaxy's `LightProfile` is an `Sersic`. -""" -dataset_name = "simple__sersic" -dataset_path = Path("dataset") / "imaging" / dataset_name - -""" -__Dataset Auto-Simulation__ - -If the dataset does not already exist on your system, it will be created by running the corresponding -simulator script. This ensures that all example scripts can be run without manually simulating data first. -""" -if not dataset_path.exists(): - import subprocess - import sys - - subprocess.run( - [sys.executable, "scripts/howtogalaxy/simulators/sersic.py"], - check=True, - ) - - -dataset = ag.Imaging.from_fits( - data_path=dataset_path / "data.fits", - noise_map_path=dataset_path / "noise_map.fits", - psf_path=dataset_path / "psf.fits", - pixel_scales=0.1, -) - -""" -we'll create and use a smaller 2.0" `Mask2D` again. -""" -mask = ag.Mask2D.circular( - shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=2.6 -) - -dataset = dataset.apply_mask(mask=mask) - -aplt.subplot_imaging_dataset(dataset=dataset) - -""" -__Nested Sampling__ - -Lets first perform the model-fit using Nautilus, but look at different parameters that control how long it takes to run. -We'll therefore discuss in a bit more detail how Nautilus works, but still keep this description conceptually simple -and avoid technical terms and jargon. For a complete description of Nautilus you should check out the Nautilus -publication `https://arxiv.org/abs/2306.16923`. - -nlive: - -Nautilus is a `nested sampling` algorithm. As we described in chapter 2, it throws down a set of `live points` in -parameter space, where each live point corresponds to a model with a given set of parameters. These points are -initially distributed according to our priors, hence why tuning our priors allows us to sample parameter space faster. - -The number of live points is set by the parameter `n_live`. More points provide a more thorough sampling of -parameter space, increasing the probability that we locate the global maxima solution. Therefore, if you think your -model-fit has gone to a local maxima, you should try increasing `n_live`. The downside of this is Nautilus will -take longer to sample parameter space and converge on a solution. Ideally, we will use as few live points as possible -to locate the global maxima as quickly as possible. - -f_live: - -A nested sampling algorithm estimates the *Bayesian Evidence* of the model-fit, which is quantity the non-linear -search algorithms we introduce later do not. The Bayesian evidence quantifies how well the model as a whole fits -the data, following a principle called Occam's Razor (`https://simple.wikipedia.org/wiki/Occam%27s_razor`). This -penalizes models for being more complex (e.g. more parameters) and requires that their additional complexity improve -their overall fit to the data compared to a simpler model. By computing the comparing the Bayesian evidence of -different models one can objectively choose the model that best fits the data. - -A nested sampling algorithm stops sampling when it estimates that continuing sampling will not increase the Bayesian -evidence (called the `log_evidence`) by more than the `f_live`. As Nautilus progresses and converges on the -solution, the rate of increase of the estimated Bayesian evidence slows down. Therefore, higher `f_live`s -mean Nautilus terminate sooner. - -A high `f_live` will make the errors estimated on every parameter unreliable and its value must be kept -below 0.8 for reliable error estimates. However, when chaining searches, we typically *do not care* about the errors -in the first search, therefore setting a high evidence tolerance can be an effective means to make Nautilus converge -faster (we'll estimate reliable errors in the second search when the `f_live is 0.8 or less). - - -Lets perform two fits, where: - - - One has many live points and a higher evidence tolerance, causing the non-linear search to - take a longer time to run. - - - One has few live points, a high sampling efficiency and evidence tolerance, causing the non-linear search to - converge and end quicker. -""" -model = af.Collection( - galaxies=af.Collection( - lens=af.Model( - ag.Galaxy, redshift=0.5, bulge=ag.lp.Sersic, mass=ag.mp.Isothermal - ), - source=af.Model(ag.Galaxy, redshift=1.0, bulge=ag.lp.Sersic), - ) -) - -search = af.Nautilus( - path_prefix=Path("howtogalaxy", "chapter_optional"), - name="tutorial_searches_slow", - unique_tag=dataset_name, - n_live=400, - n_batch=50, # GPU lens model fits are batched and run simultaneously, see VRAM section below. -) - -analysis = ag.AnalysisImaging(dataset=dataset, use_jax=True) - -print( - "The non-linear search has begun running - checkout the workspace/output" - " folder for live output of the results, images and model." - " This Jupyter notebook cell with progress once search has completed - this could take some time!" -) - -result_slow = search.fit(model=model, analysis=analysis) - -""" -Lets check that we get a good model and fit to the data. -""" -aplt.subplot_fit_imaging(fit=result_slow.max_log_likelihood_fit) - -""" -We can use the result to tell us how many iterations Nautilus took to convergence on the solution. -""" -print("Total Nautilus Iterations (If you skip running the search, this is ~ 500000):") -print(result_slow.samples.total_samples) - -""" -Now lets run the search with fast settings, so we can compare the total number of iterations required. -""" -search = af.Nautilus( - path_prefix=Path("howtogalaxy", "chapter_2"), - name="tutorial_searches_fast", - unique_tag=dataset_name, - n_live=75, - n_batch=50, # GPU lens model fits are batched and run simultaneously, see VRAM section below. -) - -print( - "The non-linear search has begun running - checkout the workspace/output" - " folder for live output of the results, images and model." - " This Jupyter notebook cell with progress once search has completed - this could take some time!" -) - -result_fast = search.fit(model=model, analysis=analysis) - -print("Search has finished run - you may now continue the notebook.") - -""" -Lets check that this search, despite its faster sampling settings, still gives us the global maxima solution. -""" -aplt.subplot_fit_imaging(fit=result_fast.max_log_likelihood_fit) - -""" -And now lets confirm it uses significantly fewer iterations. -""" -print("Total Nautilus Iterations:") -print("Slow settings: ~500000") -print(result_slow.samples.total_samples) -print("Fast settings: ", result_fast.samples.total_samples) - -""" -__Optimizers__ - -Nested sampling algorithms like Nautilus provides the errors on all of the model parameters, by fully mapping out all -of the high likelihood regions of parameter space. This provides knowledge on the complete *range* of models that do -and do not provide high likelihood fits to the data, but takes many extra iterations to perform. If we require precise -error estimates (perhaps this is our final model fit before we publish the results in a paper), these extra -iterations are acceptable. - -However, we often don't care about the errors. For example, in the previous tutorial when chaining searches, the only -result we used from the fit performed in the first search was the maximum log likelihood model, omitting the errors -entirely! Its seems wasteful to use a nested sampling algorithm like Nautilus to map out the entirity of parameter -space when we don't use this information! - -There are a class of non-linear searches called `optimizers`, which seek to optimize just one thing, the log -likelihood. They want to find the model that maximizes the log likelihood, with no regard for the errors, thus not -wasting time mapping out in intricate detail every facet of parameter space. - -PyAutoFit supports the LBFGS optimizer (from scipy), which can be used as an alternative to nested sampling when -only the maximum likelihood model is needed. However, optimizers generally need a good starting point to work well, -and in our experience the parameter spaces fitted by models are often too complex for optimizers without careful -setup of initialization priors. - -__MCMC__ - -For users familiar with Markov Chain Monte Carlo (MCMC) non-linear samplers, PyAutoFit supports the non-linear -searches `Emcee` and `Zeus`. Like LBFGS, these also need a good starting point, and are generally less effective at -modeling than Nautilus. - -I've included an example runs of Emcee and Zeus below, where the model is set up using `UniformPriors` to give -the starting point of the MCMC walkers. -""" -lens_bulge = af.Model(ag.lp.Sersic) -lens_bulge.centre.centre_0 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1) -lens_bulge.centre.centre_1 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1) -lens_bulge.ell_comps.ell_comps_0 = af.UniformPrior(lower_limit=-0.3, upper_limit=0.3) -lens_bulge.ell_comps.ell_comps_1 = af.UniformPrior(lower_limit=-0.3, upper_limit=0.3) -lens_bulge.intensity = af.UniformPrior(lower_limit=0.5, upper_limit=1.5) -lens_bulge.effective_radius = af.UniformPrior(lower_limit=0.2, upper_limit=1.6) -lens_bulge.sersic_index = af.UniformPrior(lower_limit=3.0, upper_limit=5.0) - - -mass = af.Model(ag.mp.Isothermal) -mass.centre.centre_0 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1) -mass.centre.centre_1 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1) -mass.ell_comps.ell_comps_0 = af.UniformPrior(lower_limit=-0.3, upper_limit=0.3) -mass.ell_comps.ell_comps_1 = af.UniformPrior(lower_limit=-0.3, upper_limit=0.3) -mass.einstein_radius = af.UniformPrior(lower_limit=1.0, upper_limit=2.0) - -shear = af.Model(ag.mp.ExternalShear) -shear.gamma_1 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1) -shear.gamma_2 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1) - -bulge = af.Model(ag.lp.Sersic) -bulge.centre.centre_0 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1) -bulge.centre.centre_1 = af.UniformPrior(lower_limit=-0.1, upper_limit=0.1) -bulge.ell_comps.ell_comps_0 = af.UniformPrior(lower_limit=-0.3, upper_limit=0.3) -bulge.ell_comps.ell_comps_1 = af.UniformPrior(lower_limit=-0.3, upper_limit=0.3) -bulge.intensity = af.UniformPrior(lower_limit=0.1, upper_limit=0.5) -bulge.effective_radius = af.UniformPrior(lower_limit=0.0, upper_limit=0.4) -bulge.sersic_index = af.UniformPrior(lower_limit=0.5, upper_limit=2.0) - -lens = af.Model(ag.Galaxy, redshift=0.5, mass=mass, shear=shear) -source = af.Model(ag.Galaxy, redshift=1.0, bulge=bulge) - -model = af.Collection(galaxies=af.Collection(lens=lens, source=source)) - -search = af.Zeus( - path_prefix=Path("howtogalaxy", "chapter_2"), - name="tutorial_searches_zeus", - unique_tag=dataset_name, - nwalkers=50, - nsteps=1000, -) - -print( - "Zeus has begun running - checkout the workspace/output" - " folder for live output of the results, images and model." - " This Jupyter notebook cell with progress once search has completed - this could take some time!" -) - -result_zeus = search.fit(model=model, analysis=analysis) - -print("Zeus has finished run - you may now continue the notebook.") - -aplt.subplot_fit_imaging(fit=result_zeus.max_log_likelihood_fit) - - -search = af.Emcee( - path_prefix=Path("howtogalaxy", "chapter_2"), - name="tutorial_searches_emcee", - unique_tag=dataset_name, - nwalkers=50, - nsteps=1000, -) - -print( - "The non-linear search has begun running - checkout the workspace/output" - " folder for live output of the results, images and model." - " This Jupyter notebook cell with progress once search has completed - this could take some time!" -) - -result_emcee = search.fit(model=model, analysis=analysis) - -print("The search has finished run - you may now continue the notebook.") - -aplt.subplot_fit_imaging(fit=result_emcee.max_log_likelihood_fit) diff --git a/scripts/howtogalaxy/simulators/__init__.py b/scripts/howtogalaxy/simulators/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/scripts/imaging/features/pixelization/fit.py b/scripts/imaging/features/pixelization/fit.py index febfa959..c6aba6fe 100644 --- a/scripts/imaging/features/pixelization/fit.py +++ b/scripts/imaging/features/pixelization/fit.py @@ -116,7 +116,7 @@ import sys subprocess.run( - [sys.executable, "scripts/howtogalaxy/simulators/sersic.py"], + [sys.executable, "scripts/imaging/simulator_sersic.py"], check=True, ) diff --git a/scripts/imaging/features/pixelization/modeling.py b/scripts/imaging/features/pixelization/modeling.py index 09d62462..0cb0a2fd 100644 --- a/scripts/imaging/features/pixelization/modeling.py +++ b/scripts/imaging/features/pixelization/modeling.py @@ -157,7 +157,7 @@ import sys subprocess.run( - [sys.executable, "scripts/howtogalaxy/simulators/sersic.py"], + [sys.executable, "scripts/imaging/simulator_sersic.py"], check=True, ) diff --git a/scripts/imaging/features/pixelization/source_science.py b/scripts/imaging/features/pixelization/source_science.py index c11cd40e..218a785e 100644 --- a/scripts/imaging/features/pixelization/source_science.py +++ b/scripts/imaging/features/pixelization/source_science.py @@ -48,7 +48,7 @@ import sys subprocess.run( - [sys.executable, "scripts/howtogalaxy/simulators/sersic.py"], + [sys.executable, "scripts/imaging/simulator_sersic.py"], check=True, ) diff --git a/scripts/imaging/features/shapelets/fit.py b/scripts/imaging/features/shapelets/fit.py index 3382576f..b50eedeb 100644 --- a/scripts/imaging/features/shapelets/fit.py +++ b/scripts/imaging/features/shapelets/fit.py @@ -97,7 +97,7 @@ import sys subprocess.run( - [sys.executable, "scripts/howtogalaxy/simulators/sersic.py"], + [sys.executable, "scripts/imaging/simulator_sersic.py"], check=True, ) diff --git a/scripts/imaging/features/shapelets/modeling.py b/scripts/imaging/features/shapelets/modeling.py index c956c9c0..93515b4a 100644 --- a/scripts/imaging/features/shapelets/modeling.py +++ b/scripts/imaging/features/shapelets/modeling.py @@ -99,7 +99,7 @@ import sys subprocess.run( - [sys.executable, "scripts/howtogalaxy/simulators/sersic.py"], + [sys.executable, "scripts/imaging/simulator_sersic.py"], check=True, ) diff --git a/scripts/imaging/modeling.py b/scripts/imaging/modeling.py index 7d98eb79..f49648d9 100644 --- a/scripts/imaging/modeling.py +++ b/scripts/imaging/modeling.py @@ -509,8 +509,8 @@ You do not need to be able to answer these questions in order to fit models with PyAutoGalaxy and do science. However, having a deeper understanding of how it all works is both interesting and will benefit you as a scientist -This deeper insight is offered by the **HowToGalaxy** Jupyter notebook lectures, found -at `autogalaxy_workspace/*/howtogalaxy`. +This deeper insight is offered by the **HowToGalaxy** Jupyter notebook lectures, which live +at https://github.com/PyAutoLabs/HowToGalaxy. I recommend that you check them out if you are interested in more details! """ diff --git a/scripts/howtogalaxy/simulators/sersic.py b/scripts/imaging/simulator_sersic.py similarity index 95% rename from scripts/howtogalaxy/simulators/sersic.py rename to scripts/imaging/simulator_sersic.py index fd0eaa46..6e5d2af5 100644 --- a/scripts/howtogalaxy/simulators/sersic.py +++ b/scripts/imaging/simulator_sersic.py @@ -15,9 +15,10 @@ **Visualize:** Output subplot and image PNGs of the simulated dataset. **Plane Output:** Save the Galaxies object as a JSON file. -__Start Here Notebook__ +__Start Here__ -If any code in this script is unclear, refer to the `simulators/start_here.ipynb` notebook. +If any code in this script is unclear, refer to `imaging/simulator.py` which is a more detailed +walk-through of the same simulation steps for a bulge + disk galaxy. """ # from autoconf import setup_notebook; setup_notebook() diff --git a/scripts/interferometer/modeling.py b/scripts/interferometer/modeling.py index aa75a207..729194db 100644 --- a/scripts/interferometer/modeling.py +++ b/scripts/interferometer/modeling.py @@ -386,8 +386,8 @@ You do not need to be able to answer these questions in order to fit galaxy models with PyAutoGalaxy and do science. However, having a deeper understanding of how it all works is both interesting and will benefit you as a scientist. -This deeper insight is offered by the **HowToGalaxy** Jupyter notebook lectures, found -at `autogalaxy_workspace/*/howtogalaxy`. +This deeper insight is offered by the **HowToGalaxy** Jupyter notebook lectures, which live +at https://github.com/PyAutoLabs/HowToGalaxy. I recommend that you check them out if you are interested in more details! diff --git a/scripts/multi/plot.py b/scripts/multi/plot.py index ed7e629e..3231b002 100644 --- a/scripts/multi/plot.py +++ b/scripts/multi/plot.py @@ -47,7 +47,7 @@ import sys subprocess.run( - [sys.executable, "scripts/howtogalaxy/simulators/sersic.py"], + [sys.executable, "scripts/imaging/simulator_sersic.py"], check=True, ) diff --git a/start_here.ipynb b/start_here.ipynb index a357ff31..6ebffc00 100644 --- a/start_here.ipynb +++ b/start_here.ipynb @@ -1,609 +1,602 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "PyAutoGalaxy\n", - "============\n", - "\n", - "**PyAutoGalaxy** is software for analysing the morphologies and structures of galaxies:\n", - "\n", - "![HST Image](https://raw.githubusercontent.com/Jammy2211/PyAutoGalaxy/main/paper/hstcombined.png)\n", - "\n", - "**PyAutoGalaxy** has three core aims:\n", - "\n", - "- **Big Data**: Scaling automated S\u00e9rsic fitting to extremely large datasets, *accelerated with JAX on GPUs and using\n", - " tools like an SQL database to **build a scalable scientific workflow***.\n", - "\n", - "- **Model Complexity**: Fitting complex galaxy morphology models (e.g. Multi Gaussian Expansion, Shapelets, Ellipse\n", - " Fitting, Irregular Meshes) that go beyond just simple S\u00e9rsic fitting.\n", - "\n", - "- **Data Variety**: Support for many data types (e.g. CCD imaging, interferometry, multi-band imaging) which can be\n", - " fitted independently or simultaneously.\n", - "\n", - "This notebook gives an overview of **PyAutoGalaxy**'s API, core features and details of the autogalaxy_workspace.\n", - "\n", - "__Google Colab Setup__\n", - "\n", - "The introduction `start_here` examples are available on Google Colab, which allows you to run them in a web browser\n", - "without manual local PyAutoGalaxy installation.\n", - "\n", - "The code below sets up your environment if you are using Google Colab, including installing autolens and downloading\n", - "files required to run the notebook. If you are running this script not in Colab (e.g. locally on your own computer),\n", - "running the code will still check correctly that your environment is set up and ready to go." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "import subprocess\n", - "import sys\n", - "\n", - "try:\n", - " import google.colab\n", - "\n", - " subprocess.check_call(\n", - " [sys.executable, \"-m\", \"pip\", \"install\", \"autoconf\", \"--no-deps\"]\n", - " )\n", - "except ImportError:\n", - " pass\n", - "\n", - "from autoconf import setup_colab\n", - "\n", - "setup_colab.for_autogalaxy(\n", - " raise_error_if_not_gpu=False # Switch to False for CPU Google Colab\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Imports__\n", - "\n", - "Lets first import autolens, its plotting module and the other libraries we'll need.\n", - "\n", - "You'll see these imports in the majority of workspace examples." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "# %matplotlib inline\n", - "\n", - "import autogalaxy as ag\n", - "import autogalaxy.plot as aplt\n", - "\n", - "import matplotlib.pyplot as plt\n", - "from os import path" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Lets illustrate a simple galaxy structure calculations creating an an image of a galaxy using a light profile.\n", - "\n", - "__Grid__\n", - "\n", - "The emission of light from a galaxy is described using the `Grid2D` data structure, which is two-dimensional\n", - "Cartesian grids of (y,x) coordinates where the light profile of the galaxy is evaluated on the grid.\n", - "\n", - "We make and plot a uniform Cartesian grid:" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "grid = ag.Grid2D.uniform(\n", - " shape_native=(150, 150), # The [pixels x pixels] shape of the grid in 2D.\n", - " pixel_scales=0.05, # The pixel-scale describes the conversion from pixel units to arc-seconds.\n", - ")\n", - "\n", - "aplt.plot_grid(grid=grid, title=\"Uniform Grid\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Light Profiles__\n", - "\n", - "Our aim is to create an image of the morphological structures that make up a galaxy.\n", - "\n", - "This uses analytic functions representing a galaxy's light, referred to as `LightProfile` objects. \n", - "\n", - "The most common light profile in Astronomy is the elliptical Sersic, which we create an instance of below:" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "sersic_light_profile = ag.lp.Sersic(\n", - " centre=(0.0, 0.0), # The light profile centre [units of arc-seconds].\n", - " ell_comps=(\n", - " 0.2,\n", - " 0.1,\n", - " ), # The light profile elliptical components [can be converted to axis-ratio and position angle].\n", - " intensity=0.005, # The overall intensity normalisation [units arbitrary and are matched to the data].\n", - " effective_radius=2.0, # The effective radius containing half the profile's total luminosity [units of arc-seconds].\n", - " sersic_index=4.0, # Describes the profile's shape [higher value -> more concentrated profile].\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "By passing the light profile the `grid`, we evaluate the light emitted at every (y,x) coordinate and therefore create \n", - "an image of the Sersic light profile." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "image = sersic_light_profile.image_2d_from(grid=grid)\n", - "\n", - "plt.imshow(image.native) # Dont worry about the use of .native for now." - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Plotting__\n", - "\n", - "In-built plotting methods are provided for plotting objects and their properties, like the image of\n", - "a light profile we just created.\n", - "\n", - "By using the plot module, the image is improved: axis units are scaled to arc-seconds, a color-bar is added,\n", - "descriptive labels are included, etc.\n", - "\n", - "The plot module is highly customizable and designed to make it straight forward to create clean and informative figures\n", - "for fits to large datasets." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.plot_array(array=sersic_light_profile.image_2d_from(grid=grid), title=\"Sersic Light Profile Image\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Galaxy__\n", - "\n", - "A `Galaxy` object is a collection of light profiles at a specific redshift.\n", - "\n", - "This object is highly extensible and is what ultimately allows us to fit complex models to galaxy images.\n", - "\n", - "Below, we combine the Sersic light profile above with an Exponential light profile to create a galaxy containing both\n", - "a bulge and disk component." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "exponential_light_profile = ag.lp.Exponential(\n", - " centre=(0.0, 0.0), ell_comps=(0.1, 0.0), intensity=0.1, effective_radius=0.5\n", - ")\n", - "\n", - "galaxy = ag.Galaxy(\n", - " redshift=0.5, bulge=sersic_light_profile, disk=exponential_light_profile\n", - ")\n", - "\n", - "aplt.plot_array(array=galaxy.image_2d_from(grid=grid), title=\"Galaxy Image\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The individual light profiles of the galaxy can be plotted on a subplot." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "aplt.subplot_galaxy_light_profiles(galaxy=galaxy, grid=grid)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Galaxies__\n", - "\n", - "The `Galaxies` object is a collection of galaxies at the same redshift.\n", - "\n", - "In a moment, we will see it is integral to the model-fitting API. \n", - "\n", - "For now, lets use it to create an image of a pair of merging galaxies, noting that a more concise API for creating\n", - "the galaxy is used below where the `Sersic` is passed directly to the `Galaxy` object." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "galaxy_1 = ag.Galaxy(\n", - " redshift=0.5,\n", - " bulge=ag.lp.Sersic(\n", - " centre=(0.5, 0.2), intensity=1.0, effective_radius=1.0, sersic_index=2.0\n", - " ),\n", - ")\n", - "\n", - "galaxies = ag.Galaxies(\n", - " galaxies=[galaxy, galaxy_1],\n", - ")\n", - "\n", - "aplt.plot_array(array=galaxies.image_2d_from(grid=grid), title=\"Galaxies Image\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Units__\n", - "\n", - "The units used throughout the galaxy structure literature vary, therefore lets quickly describe the units used in\n", - "**PyAutoGalaxy**.\n", - "\n", - "Most distance quantities, like an `effective_radius` are quantities in terms of angles, which are defined in units\n", - "of arc-seconds. To convert these to physical units (e.g. kiloparsecs), we use the redshift of the galaxy and an \n", - "input cosmology. A run through of all normal unit conversions is given in guides in the workspace outlined below.\n", - "\n", - "The use of angles in arc-seconds has an important property, it means that calculations are independent of\n", - "the galaxy's redshifts and the input cosmology. This has a number of benefits, for example it makes it straight\n", - "forward to compare the properties of different galaxies even when the redshifts of the galaxies are unknown.\n", - "\n", - "__Extensibility__\n", - "\n", - "All of the objects we've introduced so far are highly extensible, for example a galaxy can be made up of any number of\n", - "light profiles and many galaxy objects can be combined into a galaxies object.\n", - "\n", - "To further illustrate this, we create a merging galaxy system with 4 star forming clumps of light, using a \n", - "`SersicSph` profile to make each spherical." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "galaxy_0 = ag.Galaxy(\n", - " redshift=0.5,\n", - " bulge=ag.lp.Sersic(\n", - " centre=(0.0, 0.0),\n", - " ell_comps=ag.convert.ell_comps_from(axis_ratio=0.9, angle=45.0),\n", - " intensity=0.2,\n", - " effective_radius=0.8,\n", - " sersic_index=4.0,\n", - " ),\n", - " disk=ag.lp.Exponential(\n", - " centre=(0.0, 0.0),\n", - " ell_comps=ag.convert.ell_comps_from(axis_ratio=0.7, angle=30.0),\n", - " intensity=0.1,\n", - " effective_radius=1.6,\n", - " ),\n", - " extra_galaxy_0=ag.lp.SersicSph(\n", - " centre=(1.0, 1.0), intensity=0.5, effective_radius=0.2\n", - " ),\n", - " extra_galaxy_1=ag.lp.SersicSph(\n", - " centre=(0.5, 0.8), intensity=0.5, effective_radius=0.2\n", - " ),\n", - " extra_galaxy_2=ag.lp.SersicSph(\n", - " centre=(-1.0, -0.7), intensity=0.5, effective_radius=0.2\n", - " ),\n", - " extra_galaxy_3=ag.lp.SersicSph(\n", - " centre=(-1.0, 0.4), intensity=0.5, effective_radius=0.2\n", - " ),\n", - ")\n", - "\n", - "galaxy_1 = ag.Galaxy(\n", - " redshift=0.5,\n", - " bulge=ag.lp.Sersic(\n", - " centre=(0.0, 1.0),\n", - " ell_comps=(0.0, 0.1),\n", - " intensity=0.1,\n", - " effective_radius=0.6,\n", - " sersic_index=3.0,\n", - " ),\n", - ")\n", - "\n", - "galaxies = ag.Galaxies(galaxies=[galaxy_0, galaxy_1])\n", - "\n", - "aplt.plot_array(array=galaxies.image_2d_from(grid=grid), title=\"Galaxies Image\")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Galaxy Modeling__\n", - "\n", - "Galaxy modeling is the process of fitting a physical model to imaging data in order to infer the structural\n", - "and photometric properties of galaxies, such as their light distribution, size, shape, and orientation.\n", - "\n", - "The primary goal of **PyAutoGalaxy** is to make galaxy modeling **simple, scalable to large datasets, and fast**,\n", - "with GPU acceleration provided via JAX.\n", - "\n", - "The animation below illustrates the galaxy modeling workflow. Many models are fitted to the data iteratively,\n", - "progressively improving the quality of the fit until the model closely reproduces the observed image.\n", - "\n", - "NOTE: Placeholder showing strong lens modeling animation used currently.\n", - "\n", - "![Lens Modeling Animation](https://github.com/Jammy2211/auto_files/blob/main/lensmodel.gif?raw=true \"model\")\n", - "\n", - "**Credit: Amy Etherington**\n", - "\n", - "The next documentation page guides you through galaxy modeling for a variety of data types (e.g. CCD imaging at \n", - "different resolutions) and scientific use-cases (e.g. galaxy morphology studies, bulge\u2013disk decomposition).\n", - "\n", - "__Simulations__\n", - "\n", - "Simulating galaxy images is often essential, for example to:\n", - "\n", - "- Practice galaxy modeling before working with real data.\n", - "- Generate large training sets (e.g. for machine learning).\n", - "- Test galaxy formation and structural models in a fully controlled environment.\n", - "\n", - "The next documentation page guides you through how to simulate galaxies for different types of data\n", - "(e.g. CCD imaging) and different modeling goals (e.g. single-component galaxies, multi-component systems).\n", - "\n", - "__Wrap Up__\n", - "\n", - "This completes the introduction to **PyAutoGalaxy**, including a brief overview of the core API for galaxy light\n", - "profiles, galaxy modeling, and data simulation.\n", - "\n", - "__What Data Type?__\n", - "\n", - "If you are interested in modeling galaxies, you now need to decide what type of imaging data you want to work with:\n", - "\n", - "- **CCD Imaging**: For image data from telescopes like Hubble, James Webb, or ground-based observatories,\n", - " go to `imaging/start_here.ipynb`.\n", - "\n", - "- **Interferometer**: For radio / sub-mm interferometer data from instruments like ALMA, where galaxies are\n", - " observed via visibilities in the uv-plane, go to `interferometer/start_here.ipynb`.\n", - "\n", - "- **Multi-Band Imaging**: For galaxies observed in multiple wavebands (e.g. colour gradients, stellar population\n", - " studies), go to `multi_band/start_here.ipynb`.\n", - "\n", - "__Google Colab__\n", - "\n", - "You can also open and run each notebook directly in Google Colab, which provides a free cloud computing\n", - "environment with all the required dependencies already installed.\n", - "\n", - "This is a great way to get started quickly without needing to install **PyAutoGalaxy** on your own machine,\n", - "so you can check it is the right software for you before going through the installation process:\n", - "\n", - "- [imaging/start_here.ipynb](https://colab.research.google.com/github/PyAutoLabs/autogalaxy_workspace/blob/2026.4.13.6/notebooks/imaging/start_here.ipynb>):\n", - " Galaxy modeling with CCD imaging (e.g. Hubble, James Webb, ground-based telescopes).\n", - "\n", - "- [interferometer/start_here.ipynb](https://colab.research.google.com/github/PyAutoLabs/autogalaxy_workspace/blob/2026.4.13.6/notebooks/interferometer/start_here.ipynb):\n", - " Galaxy modeling with interferometer data (e.g. ALMA), fitting directly in the uv-plane.\n", - "\n", - "- [multi_band/start_here.ipynb](https://colab.research.google.com/github/PyAutoLabs/autogalaxy_workspace/blob/2026.4.13.6/notebooks/multi/start_here.ipynb):\n", - " Multi-band galaxy modeling to study colour gradients and wavelength-dependent structure.\n", - " \n", - "__Still Unsure?__\n", - "\n", - "Each notebook is short and self-contained, and can be completed and adapted quickly to your particular task. \n", - "Therefore, if you're unsure exactly which scale of lensing applies to you, or quite what data you want to use, you \n", - "should just read through a few different notebooks and go from there.\n", - "\n", - "__HowToGalaxy Lectures__\n", - "\n", - "For experienced scientists, the run through above will have been a breeze. Concepts surrounding galaxy structure and \n", - "morphology were already familiar and the statistical techniques used for fitting and modeling already understood.\n", - "\n", - "For those less familiar with these concepts (e.g. undergraduate students, new PhD students or interested members of the \n", - "public), things may have been less clear and a slower more detailed explanation of each concept would be beneficial.\n", - "\n", - "The **HowToGalaxy** Jupyter Notebook lectures are provide exactly this They are a 3+ chapter guide which thoroughly \n", - "take you through the core concepts of galaxy light profiles, teach you the principles ofthe statistical techniques \n", - "used in modeling and ultimately will allow you to undertake scientific research like a professional astronomer.\n", - "\n", - "If this sounds like it suits you, checkout the `autogalaxy_workspace/notebooks/howtogalaxy` package now, its it\n", - "recommended you go here before anywhere else!\n", - "\n", - "__Features__\n", - "\n", - "Here is a brief overview of the advanced features of **PyAutoGalaxy**. \n", - "\n", - "Firstly, brief one sentence descriptions of each feature are given, with more detailed descriptions below including \n", - "links to the relevant workspace examples.\n", - "\n", - "**Interferometry**: Modeling of interferometer data (e.g. ALMA, LOFAR) directly in the uv-plane.\n", - "**Multi-Wavelength**: Simultaneous analysis of imaging and / or interferometer datasets observed at different wavelengths.\n", - "**Ellipse Fitting**: Fitting ellipses to determine a galaxy's ellipticity, position angle and centre.\n", - "**Multi Gaussian Expansion (MGE)**: Decomposing a galaxy into hundreds of Gaussians, capturing more complex structures than simple light profiles.\n", - "**Shapelets**: Decomposing a galaxy into a set of shapelet orthogonal basis functions, capturing more complex structures than simple light profiles.\n", - "**Sky Background**: Including the background sky in the model to ensure robust fits to the outskirts of galaxies.\n", - "**Operated Light Profiles**: Assuming a light profile has already been convolved with the PSF, for when the PSF is a significant effect.\n", - "**Pixelizations**: Reconstructing a galaxy's on a mesh of pixels, to capture extremely irregular structures like spiral arms.\n", - "\n", - "\n", - "__Interferometry__\n", - "\n", - "Modeling interferometer data from submillimeter (e.g. ALMA) and radio (e.g. LOFAR) observatories:\n", - "\n", - "![ALMA Image](https://raw.githubusercontent.com/Jammy2211/PyAutoGalaxy/main/paper/almacombined.png)\n", - "\n", - "Visibilities data is fitted directly in the uv-plane, circumventing issues that arise when fitting a dirty image\n", - "such as correlated noise. This uses the non-uniform fast fourier transform algorithm\n", - "[PyNUFFT](https://github.com/jyhmiinlin/pynufft) to efficiently map the galaxy model images to the uv-plane.\n", - "\n", - "Checkout the`autogalaxy_workspace/*/interferometer` package to get started.\n", - "\n", - "\n", - "__Multi-Wavelength__\n", - "\n", - "Modeling imaging datasets observed at different wavelengths (e.g. HST F814W and F150W) simultaneously or simultaneously\n", - "analysing imaging and interferometer data:\n", - "\n", - "![g-band](https://raw.githubusercontent.com/Jammy2211/PyAutoGalaxy/main/docs/overview/images/overview_3/g_image.png)\n", - "\n", - "![r-band](https://raw.githubusercontent.com/Jammy2211/PyAutoGalaxy/main/docs/overview/images/overview_3/r_image.png)\n", - "\n", - "The appearance of the galaxy changes as a function of wavelength, therefore multi-wavelength analysis means we can learn\n", - "more about the different components in a galaxy (e.g a redder bulge and bluer disk) or when imaging and interferometer\n", - "data are combined, we can compare the emission from stars and dust.\n", - "\n", - "Checkout the `autogalaxy_workspace/*/multi` package to get started, however combining datasets is a more advanced\n", - "feature and it is recommended you first get to grips with the core API.\n", - "\n", - "\n", - "__Ellipse Fitting__\n", - "\n", - "Ellipse fitting is a technique which fits many ellipses to a galaxy's emission to determine its ellipticity, position\n", - "angle and centre, without assuming a parametric form for its light (e.g. a Sersic profile):\n", - "\n", - "![ellipse](https://raw.githubusercontent.com/Jammy2211/PyAutoGalaxy/main/docs/overview/images/overview_3/ellipse.png)\n", - "\n", - "This provides complementary information to parametric light profile fitting, for example giving insights on whether\n", - "the ellipticity and position angle are constant with radius or if the galaxy's emission is lopsided. \n", - "\n", - "There are also multipole moment extensions to ellipse fitting, which determine higher order deviations from elliptical \n", - "symmetry providing even more information on the galaxy's structure.\n", - "\n", - "The following paper describes the technique in detail: https://arxiv.org/html/2407.12983v1\n", - "\n", - "Checkout `autogalaxy_workspace/notebooks/features/ellipse_fitting.ipynb` to learn how to use ellipse fitting.\n", - "\n", - "\n", - "__Multi Gaussian Expansion (MGE)__\n", - "\n", - "An MGE decomposes the light of a galaxy into tens or hundreds of two dimensional Gaussians:\n", - "\n", - "![MGE](https://raw.githubusercontent.com/Jammy2211/PyAutoGalaxy/main/docs/overview/images/overview_3/mge.png)\n", - "\n", - "In the image above, 30 Gaussians are shown, where their sizes go from below the pixel scale (in order to resolve\n", - "point emission) to beyond the size of the galaxy (to capture its extended emission).\n", - "\n", - "Scientific Applications include capturing departures from elliptical symmetry in the light of galaxies, providing a \n", - "flexible model to deblend the emission of point sources (e.g. quasars) from the emission of their host galaxy and \n", - "deprojecting the light of a galaxy from 2D to 3D.\n", - "\n", - "Checkout `autogalaxy_workspace/notebooks/features/multi_gaussian_expansion.ipynb` to learn how to use an MGE.\n", - "\n", - "\n", - "__Shapelets__\n", - "\n", - "Shapelets are a set of orthogonal basis functions that can be combined the represent galaxy structures:\n", - "\n", - "Scientific Applications include capturing symmetric structures in a galaxy which are more complex than a Sersic profile,\n", - "irregular and asymmetric structures in a galaxy like spiral arms and providing a flexible model to deblend the emission \n", - "of point sources (e.g. quasars) from the emission of their host galaxy.\n", - "\n", - "Checkout `autogalaxy_workspace/notebooks/features/shapelets.ipynb` to learn how to use shapelets.\n", - "\n", - "\n", - "__Sky Background__\n", - "\n", - "When an image of a galaxy is observed, the background sky contributes light to the image and adds noise:\n", - "\n", - "For detailed studies of the outskirts of galaxies (e.g. stellar halos, faint extended disks), the sky background must be\n", - "accounted for in the model to ensure robust and accurate fits.\n", - "\n", - "Checkout `autogalaxy_workspace/notebooks/features/sky_background.ipynb` to learn how to use include the sky\n", - "background in your model.\n", - "\n", - "\n", - "__Operated Light Profiles__\n", - "\n", - "An operated light profile is one where it is assumed to already be convolved with the PSF of the data, with the \n", - "`Moffat` and `Gaussian` profiles common choices:\n", - "\n", - "They are used for certain scientific applications where the PSF convolution is known to be a significant effect and\n", - "the knowledge of the PSF allows for detailed modeling abd deblending of the galaxy's light.\n", - "\n", - "Checkout `autogalaxy_workspace/notebooks/features/operated_light_profiles.ipynb` to learn how to use operated profiles.\n", - "\n", - "\n", - "__Pixelizations__\n", - "\n", - "A pixelization reconstructs a galaxy's light on a mesh of pixels, for example a rectangular mesh, Delaunay \n", - "triangulation or Voronoi grid. \n", - "\n", - "These models are highly flexible and can capture complex structures in a galaxy's light that parametric models\n", - "like a Sersic profile cannot, for example spiral arms or asymmetric merging features.\n", - "\n", - "The image below shows a non parametric of a galaxy observed in the Hubble Ultra Deep Field. Its bulge and disk are\n", - "fitted accurately using light profiles, whereas its asymmetric and irregular spiral arm features are accurately\n", - "captured using a rectangular mesh:\n", - "\n", - "![HST Image](https://raw.githubusercontent.com/Jammy2211/PyAutoGalaxy/main/paper/hstcombined.png)\n", - "\n", - "Checkout `autogalaxy_workspace/notebooks/features/pixelizations.ipynb` to learn how to use a pixelization, however\n", - "this is a more advanced feature and it is recommended you first get to grips with the core API.\n", - "\n", - "\n", - "__Other:__\n", - "\n", - "- Automated pipelines / database tools.\n", - "- Graphical models." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [], - "outputs": [], - "execution_count": null - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "PyAutoGalaxy\n", + "============\n", + "\n", + "**PyAutoGalaxy** is software for analysing the morphologies and structures of galaxies:\n", + "\n", + "![HST Image](https://raw.githubusercontent.com/Jammy2211/PyAutoGalaxy/main/paper/hstcombined.png)\n", + "\n", + "**PyAutoGalaxy** has three core aims:\n", + "\n", + "- **Big Data**: Scaling automated S\u00e9rsic fitting to extremely large datasets, *accelerated with JAX on GPUs and using\n", + " tools like an SQL database to **build a scalable scientific workflow***.\n", + "\n", + "- **Model Complexity**: Fitting complex galaxy morphology models (e.g. Multi Gaussian Expansion, Shapelets, Ellipse\n", + " Fitting, Irregular Meshes) that go beyond just simple S\u00e9rsic fitting.\n", + "\n", + "- **Data Variety**: Support for many data types (e.g. CCD imaging, interferometry, multi-band imaging) which can be\n", + " fitted independently or simultaneously.\n", + "\n", + "This notebook gives an overview of **PyAutoGalaxy**'s API, core features and details of the autogalaxy_workspace.\n", + "\n", + "__Google Colab Setup__\n", + "\n", + "The introduction `start_here` examples are available on Google Colab, which allows you to run them in a web browser\n", + "without manual local PyAutoGalaxy installation.\n", + "\n", + "The code below sets up your environment if you are using Google Colab, including installing autolens and downloading\n", + "files required to run the notebook. If you are running this script not in Colab (e.g. locally on your own computer),\n", + "running the code will still check correctly that your environment is set up and ready to go." + ] }, - "nbformat": 4, - "nbformat_minor": 4 + { + "cell_type": "code", + "metadata": {}, + "source": [ + "\n", + "import subprocess\n", + "import sys\n", + "\n", + "try:\n", + " import google.colab\n", + "\n", + " subprocess.check_call(\n", + " [sys.executable, \"-m\", \"pip\", \"install\", \"autoconf\", \"--no-deps\"]\n", + " )\n", + "except ImportError:\n", + " pass\n", + "\n", + "from autoconf import setup_colab\n", + "\n", + "setup_colab.for_autogalaxy(\n", + " raise_error_if_not_gpu=False # Switch to False for CPU Google Colab\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Imports__\n", + "\n", + "Lets first import autolens, its plotting module and the other libraries we'll need.\n", + "\n", + "You'll see these imports in the majority of workspace examples." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "# %matplotlib inline\n", + "\n", + "import autogalaxy as ag\n", + "import autogalaxy.plot as aplt\n", + "\n", + "import matplotlib.pyplot as plt\n", + "from os import path" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Lets illustrate a simple galaxy structure calculations creating an an image of a galaxy using a light profile.\n", + "\n", + "__Grid__\n", + "\n", + "The emission of light from a galaxy is described using the `Grid2D` data structure, which is two-dimensional\n", + "Cartesian grids of (y,x) coordinates where the light profile of the galaxy is evaluated on the grid.\n", + "\n", + "We make and plot a uniform Cartesian grid:" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "grid = ag.Grid2D.uniform(\n", + " shape_native=(150, 150), # The [pixels x pixels] shape of the grid in 2D.\n", + " pixel_scales=0.05, # The pixel-scale describes the conversion from pixel units to arc-seconds.\n", + ")\n", + "\n", + "aplt.plot_grid(grid=grid, title=\"Uniform Grid\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Light Profiles__\n", + "\n", + "Our aim is to create an image of the morphological structures that make up a galaxy.\n", + "\n", + "This uses analytic functions representing a galaxy's light, referred to as `LightProfile` objects. \n", + "\n", + "The most common light profile in Astronomy is the elliptical Sersic, which we create an instance of below:" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "sersic_light_profile = ag.lp.Sersic(\n", + " centre=(0.0, 0.0), # The light profile centre [units of arc-seconds].\n", + " ell_comps=(\n", + " 0.2,\n", + " 0.1,\n", + " ), # The light profile elliptical components [can be converted to axis-ratio and position angle].\n", + " intensity=0.005, # The overall intensity normalisation [units arbitrary and are matched to the data].\n", + " effective_radius=2.0, # The effective radius containing half the profile's total luminosity [units of arc-seconds].\n", + " sersic_index=4.0, # Describes the profile's shape [higher value -> more concentrated profile].\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "By passing the light profile the `grid`, we evaluate the light emitted at every (y,x) coordinate and therefore create \n", + "an image of the Sersic light profile." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "image = sersic_light_profile.image_2d_from(grid=grid)\n", + "\n", + "plt.imshow(image.native) # Dont worry about the use of .native for now." + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Plotting__\n", + "\n", + "In-built plotting methods are provided for plotting objects and their properties, like the image of\n", + "a light profile we just created.\n", + "\n", + "By using the plot module, the image is improved: axis units are scaled to arc-seconds, a color-bar is added,\n", + "descriptive labels are included, etc.\n", + "\n", + "The plot module is highly customizable and designed to make it straight forward to create clean and informative figures\n", + "for fits to large datasets." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.plot_array(array=sersic_light_profile.image_2d_from(grid=grid), title=\"Sersic Light Profile Image\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Galaxy__\n", + "\n", + "A `Galaxy` object is a collection of light profiles at a specific redshift.\n", + "\n", + "This object is highly extensible and is what ultimately allows us to fit complex models to galaxy images.\n", + "\n", + "Below, we combine the Sersic light profile above with an Exponential light profile to create a galaxy containing both\n", + "a bulge and disk component." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "exponential_light_profile = ag.lp.Exponential(\n", + " centre=(0.0, 0.0), ell_comps=(0.1, 0.0), intensity=0.1, effective_radius=0.5\n", + ")\n", + "\n", + "galaxy = ag.Galaxy(\n", + " redshift=0.5, bulge=sersic_light_profile, disk=exponential_light_profile\n", + ")\n", + "\n", + "aplt.plot_array(array=galaxy.image_2d_from(grid=grid), title=\"Galaxy Image\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The individual light profiles of the galaxy can be plotted on a subplot." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.subplot_galaxy_light_profiles(galaxy=galaxy, grid=grid)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Galaxies__\n", + "\n", + "The `Galaxies` object is a collection of galaxies at the same redshift.\n", + "\n", + "In a moment, we will see it is integral to the model-fitting API. \n", + "\n", + "For now, lets use it to create an image of a pair of merging galaxies, noting that a more concise API for creating\n", + "the galaxy is used below where the `Sersic` is passed directly to the `Galaxy` object." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "galaxy_1 = ag.Galaxy(\n", + " redshift=0.5,\n", + " bulge=ag.lp.Sersic(\n", + " centre=(0.5, 0.2), intensity=1.0, effective_radius=1.0, sersic_index=2.0\n", + " ),\n", + ")\n", + "\n", + "galaxies = ag.Galaxies(\n", + " galaxies=[galaxy, galaxy_1],\n", + ")\n", + "\n", + "aplt.plot_array(array=galaxies.image_2d_from(grid=grid), title=\"Galaxies Image\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Units__\n", + "\n", + "The units used throughout the galaxy structure literature vary, therefore lets quickly describe the units used in\n", + "**PyAutoGalaxy**.\n", + "\n", + "Most distance quantities, like an `effective_radius` are quantities in terms of angles, which are defined in units\n", + "of arc-seconds. To convert these to physical units (e.g. kiloparsecs), we use the redshift of the galaxy and an \n", + "input cosmology. A run through of all normal unit conversions is given in guides in the workspace outlined below.\n", + "\n", + "The use of angles in arc-seconds has an important property, it means that calculations are independent of\n", + "the galaxy's redshifts and the input cosmology. This has a number of benefits, for example it makes it straight\n", + "forward to compare the properties of different galaxies even when the redshifts of the galaxies are unknown.\n", + "\n", + "__Extensibility__\n", + "\n", + "All of the objects we've introduced so far are highly extensible, for example a galaxy can be made up of any number of\n", + "light profiles and many galaxy objects can be combined into a galaxies object.\n", + "\n", + "To further illustrate this, we create a merging galaxy system with 4 star forming clumps of light, using a \n", + "`SersicSph` profile to make each spherical." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "galaxy_0 = ag.Galaxy(\n", + " redshift=0.5,\n", + " bulge=ag.lp.Sersic(\n", + " centre=(0.0, 0.0),\n", + " ell_comps=ag.convert.ell_comps_from(axis_ratio=0.9, angle=45.0),\n", + " intensity=0.2,\n", + " effective_radius=0.8,\n", + " sersic_index=4.0,\n", + " ),\n", + " disk=ag.lp.Exponential(\n", + " centre=(0.0, 0.0),\n", + " ell_comps=ag.convert.ell_comps_from(axis_ratio=0.7, angle=30.0),\n", + " intensity=0.1,\n", + " effective_radius=1.6,\n", + " ),\n", + " extra_galaxy_0=ag.lp.SersicSph(\n", + " centre=(1.0, 1.0), intensity=0.5, effective_radius=0.2\n", + " ),\n", + " extra_galaxy_1=ag.lp.SersicSph(\n", + " centre=(0.5, 0.8), intensity=0.5, effective_radius=0.2\n", + " ),\n", + " extra_galaxy_2=ag.lp.SersicSph(\n", + " centre=(-1.0, -0.7), intensity=0.5, effective_radius=0.2\n", + " ),\n", + " extra_galaxy_3=ag.lp.SersicSph(\n", + " centre=(-1.0, 0.4), intensity=0.5, effective_radius=0.2\n", + " ),\n", + ")\n", + "\n", + "galaxy_1 = ag.Galaxy(\n", + " redshift=0.5,\n", + " bulge=ag.lp.Sersic(\n", + " centre=(0.0, 1.0),\n", + " ell_comps=(0.0, 0.1),\n", + " intensity=0.1,\n", + " effective_radius=0.6,\n", + " sersic_index=3.0,\n", + " ),\n", + ")\n", + "\n", + "galaxies = ag.Galaxies(galaxies=[galaxy_0, galaxy_1])\n", + "\n", + "aplt.plot_array(array=galaxies.image_2d_from(grid=grid), title=\"Galaxies Image\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Galaxy Modeling__\n", + "\n", + "Galaxy modeling is the process of fitting a physical model to imaging data in order to infer the structural\n", + "and photometric properties of galaxies, such as their light distribution, size, shape, and orientation.\n", + "\n", + "The primary goal of **PyAutoGalaxy** is to make galaxy modeling **simple, scalable to large datasets, and fast**,\n", + "with GPU acceleration provided via JAX.\n", + "\n", + "The animation below illustrates the galaxy modeling workflow. Many models are fitted to the data iteratively,\n", + "progressively improving the quality of the fit until the model closely reproduces the observed image.\n", + "\n", + "NOTE: Placeholder showing strong lens modeling animation used currently.\n", + "\n", + "![Lens Modeling Animation](https://github.com/Jammy2211/auto_files/blob/main/lensmodel.gif?raw=true \"model\")\n", + "\n", + "**Credit: Amy Etherington**\n", + "\n", + "The next documentation page guides you through galaxy modeling for a variety of data types (e.g. CCD imaging at \n", + "different resolutions) and scientific use-cases (e.g. galaxy morphology studies, bulge\u2013disk decomposition).\n", + "\n", + "__Simulations__\n", + "\n", + "Simulating galaxy images is often essential, for example to:\n", + "\n", + "- Practice galaxy modeling before working with real data.\n", + "- Generate large training sets (e.g. for machine learning).\n", + "- Test galaxy formation and structural models in a fully controlled environment.\n", + "\n", + "The next documentation page guides you through how to simulate galaxies for different types of data\n", + "(e.g. CCD imaging) and different modeling goals (e.g. single-component galaxies, multi-component systems).\n", + "\n", + "__Wrap Up__\n", + "\n", + "This completes the introduction to **PyAutoGalaxy**, including a brief overview of the core API for galaxy light\n", + "profiles, galaxy modeling, and data simulation.\n", + "\n", + "__What Data Type?__\n", + "\n", + "If you are interested in modeling galaxies, you now need to decide what type of imaging data you want to work with:\n", + "\n", + "- **CCD Imaging**: For image data from telescopes like Hubble, James Webb, or ground-based observatories,\n", + " go to `imaging/start_here.ipynb`.\n", + "\n", + "- **Interferometer**: For radio / sub-mm interferometer data from instruments like ALMA, where galaxies are\n", + " observed via visibilities in the uv-plane, go to `interferometer/start_here.ipynb`.\n", + "\n", + "- **Multi-Band Imaging**: For galaxies observed in multiple wavebands (e.g. colour gradients, stellar population\n", + " studies), go to `multi_band/start_here.ipynb`.\n", + "\n", + "__Google Colab__\n", + "\n", + "You can also open and run each notebook directly in Google Colab, which provides a free cloud computing\n", + "environment with all the required dependencies already installed.\n", + "\n", + "This is a great way to get started quickly without needing to install **PyAutoGalaxy** on your own machine,\n", + "so you can check it is the right software for you before going through the installation process:\n", + "\n", + "- [imaging/start_here.ipynb](https://colab.research.google.com/github/PyAutoLabs/autogalaxy_workspace/blob/2026.4.13.6/notebooks/imaging/start_here.ipynb>):\n", + " Galaxy modeling with CCD imaging (e.g. Hubble, James Webb, ground-based telescopes).\n", + "\n", + "- [interferometer/start_here.ipynb](https://colab.research.google.com/github/PyAutoLabs/autogalaxy_workspace/blob/2026.4.13.6/notebooks/interferometer/start_here.ipynb):\n", + " Galaxy modeling with interferometer data (e.g. ALMA), fitting directly in the uv-plane.\n", + "\n", + "- [multi_band/start_here.ipynb](https://colab.research.google.com/github/PyAutoLabs/autogalaxy_workspace/blob/2026.4.13.6/notebooks/multi/start_here.ipynb):\n", + " Multi-band galaxy modeling to study colour gradients and wavelength-dependent structure.\n", + " \n", + "__Still Unsure?__\n", + "\n", + "Each notebook is short and self-contained, and can be completed and adapted quickly to your particular task. \n", + "Therefore, if you're unsure exactly which scale of lensing applies to you, or quite what data you want to use, you \n", + "should just read through a few different notebooks and go from there.\n", + "\n", + "__HowToGalaxy Lectures__\n", + "\n", + "If you are new to galaxy modeling or want a slower, first-principles walk-through of the statistical techniques\n", + "behind it, the **HowToGalaxy** lecture series takes you from the basics through to modeling real galaxy imaging\n", + "data. It now lives in its own repository:\n", + "\n", + "https://github.com/PyAutoLabs/HowToGalaxy\n", + "\n", + "__Features__\n", + "\n", + "Here is a brief overview of the advanced features of **PyAutoGalaxy**. \n", + "\n", + "Firstly, brief one sentence descriptions of each feature are given, with more detailed descriptions below including \n", + "links to the relevant workspace examples.\n", + "\n", + "**Interferometry**: Modeling of interferometer data (e.g. ALMA, LOFAR) directly in the uv-plane.\n", + "**Multi-Wavelength**: Simultaneous analysis of imaging and / or interferometer datasets observed at different wavelengths.\n", + "**Ellipse Fitting**: Fitting ellipses to determine a galaxy's ellipticity, position angle and centre.\n", + "**Multi Gaussian Expansion (MGE)**: Decomposing a galaxy into hundreds of Gaussians, capturing more complex structures than simple light profiles.\n", + "**Shapelets**: Decomposing a galaxy into a set of shapelet orthogonal basis functions, capturing more complex structures than simple light profiles.\n", + "**Sky Background**: Including the background sky in the model to ensure robust fits to the outskirts of galaxies.\n", + "**Operated Light Profiles**: Assuming a light profile has already been convolved with the PSF, for when the PSF is a significant effect.\n", + "**Pixelizations**: Reconstructing a galaxy's on a mesh of pixels, to capture extremely irregular structures like spiral arms.\n", + "\n", + "\n", + "__Interferometry__\n", + "\n", + "Modeling interferometer data from submillimeter (e.g. ALMA) and radio (e.g. LOFAR) observatories:\n", + "\n", + "![ALMA Image](https://raw.githubusercontent.com/Jammy2211/PyAutoGalaxy/main/paper/almacombined.png)\n", + "\n", + "Visibilities data is fitted directly in the uv-plane, circumventing issues that arise when fitting a dirty image\n", + "such as correlated noise. This uses the non-uniform fast fourier transform algorithm\n", + "[PyNUFFT](https://github.com/jyhmiinlin/pynufft) to efficiently map the galaxy model images to the uv-plane.\n", + "\n", + "Checkout the`autogalaxy_workspace/*/interferometer` package to get started.\n", + "\n", + "\n", + "__Multi-Wavelength__\n", + "\n", + "Modeling imaging datasets observed at different wavelengths (e.g. HST F814W and F150W) simultaneously or simultaneously\n", + "analysing imaging and interferometer data:\n", + "\n", + "![g-band](https://raw.githubusercontent.com/Jammy2211/PyAutoGalaxy/main/docs/overview/images/overview_3/g_image.png)\n", + "\n", + "![r-band](https://raw.githubusercontent.com/Jammy2211/PyAutoGalaxy/main/docs/overview/images/overview_3/r_image.png)\n", + "\n", + "The appearance of the galaxy changes as a function of wavelength, therefore multi-wavelength analysis means we can learn\n", + "more about the different components in a galaxy (e.g a redder bulge and bluer disk) or when imaging and interferometer\n", + "data are combined, we can compare the emission from stars and dust.\n", + "\n", + "Checkout the `autogalaxy_workspace/*/multi` package to get started, however combining datasets is a more advanced\n", + "feature and it is recommended you first get to grips with the core API.\n", + "\n", + "\n", + "__Ellipse Fitting__\n", + "\n", + "Ellipse fitting is a technique which fits many ellipses to a galaxy's emission to determine its ellipticity, position\n", + "angle and centre, without assuming a parametric form for its light (e.g. a Sersic profile):\n", + "\n", + "![ellipse](https://raw.githubusercontent.com/Jammy2211/PyAutoGalaxy/main/docs/overview/images/overview_3/ellipse.png)\n", + "\n", + "This provides complementary information to parametric light profile fitting, for example giving insights on whether\n", + "the ellipticity and position angle are constant with radius or if the galaxy's emission is lopsided. \n", + "\n", + "There are also multipole moment extensions to ellipse fitting, which determine higher order deviations from elliptical \n", + "symmetry providing even more information on the galaxy's structure.\n", + "\n", + "The following paper describes the technique in detail: https://arxiv.org/html/2407.12983v1\n", + "\n", + "Checkout `autogalaxy_workspace/notebooks/features/ellipse_fitting.ipynb` to learn how to use ellipse fitting.\n", + "\n", + "\n", + "__Multi Gaussian Expansion (MGE)__\n", + "\n", + "An MGE decomposes the light of a galaxy into tens or hundreds of two dimensional Gaussians:\n", + "\n", + "![MGE](https://raw.githubusercontent.com/Jammy2211/PyAutoGalaxy/main/docs/overview/images/overview_3/mge.png)\n", + "\n", + "In the image above, 30 Gaussians are shown, where their sizes go from below the pixel scale (in order to resolve\n", + "point emission) to beyond the size of the galaxy (to capture its extended emission).\n", + "\n", + "Scientific Applications include capturing departures from elliptical symmetry in the light of galaxies, providing a \n", + "flexible model to deblend the emission of point sources (e.g. quasars) from the emission of their host galaxy and \n", + "deprojecting the light of a galaxy from 2D to 3D.\n", + "\n", + "Checkout `autogalaxy_workspace/notebooks/features/multi_gaussian_expansion.ipynb` to learn how to use an MGE.\n", + "\n", + "\n", + "__Shapelets__\n", + "\n", + "Shapelets are a set of orthogonal basis functions that can be combined the represent galaxy structures:\n", + "\n", + "Scientific Applications include capturing symmetric structures in a galaxy which are more complex than a Sersic profile,\n", + "irregular and asymmetric structures in a galaxy like spiral arms and providing a flexible model to deblend the emission \n", + "of point sources (e.g. quasars) from the emission of their host galaxy.\n", + "\n", + "Checkout `autogalaxy_workspace/notebooks/features/shapelets.ipynb` to learn how to use shapelets.\n", + "\n", + "\n", + "__Sky Background__\n", + "\n", + "When an image of a galaxy is observed, the background sky contributes light to the image and adds noise:\n", + "\n", + "For detailed studies of the outskirts of galaxies (e.g. stellar halos, faint extended disks), the sky background must be\n", + "accounted for in the model to ensure robust and accurate fits.\n", + "\n", + "Checkout `autogalaxy_workspace/notebooks/features/sky_background.ipynb` to learn how to use include the sky\n", + "background in your model.\n", + "\n", + "\n", + "__Operated Light Profiles__\n", + "\n", + "An operated light profile is one where it is assumed to already be convolved with the PSF of the data, with the \n", + "`Moffat` and `Gaussian` profiles common choices:\n", + "\n", + "They are used for certain scientific applications where the PSF convolution is known to be a significant effect and\n", + "the knowledge of the PSF allows for detailed modeling abd deblending of the galaxy's light.\n", + "\n", + "Checkout `autogalaxy_workspace/notebooks/features/operated_light_profiles.ipynb` to learn how to use operated profiles.\n", + "\n", + "\n", + "__Pixelizations__\n", + "\n", + "A pixelization reconstructs a galaxy's light on a mesh of pixels, for example a rectangular mesh, Delaunay \n", + "triangulation or Voronoi grid. \n", + "\n", + "These models are highly flexible and can capture complex structures in a galaxy's light that parametric models\n", + "like a Sersic profile cannot, for example spiral arms or asymmetric merging features.\n", + "\n", + "The image below shows a non parametric of a galaxy observed in the Hubble Ultra Deep Field. Its bulge and disk are\n", + "fitted accurately using light profiles, whereas its asymmetric and irregular spiral arm features are accurately\n", + "captured using a rectangular mesh:\n", + "\n", + "![HST Image](https://raw.githubusercontent.com/Jammy2211/PyAutoGalaxy/main/paper/hstcombined.png)\n", + "\n", + "Checkout `autogalaxy_workspace/notebooks/features/pixelizations.ipynb` to learn how to use a pixelization, however\n", + "this is a more advanced feature and it is recommended you first get to grips with the core API.\n", + "\n", + "\n", + "__Other:__\n", + "\n", + "- Automated pipelines / database tools.\n", + "- Graphical models." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [], + "outputs": [], + "execution_count": null + } + ], + "metadata": { + "anaconda-cloud": {}, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.1" + } + }, + "nbformat": 4, + "nbformat_minor": 4 } \ No newline at end of file diff --git a/start_here.py b/start_here.py index 43d64292..3dc39fc9 100644 --- a/start_here.py +++ b/start_here.py @@ -310,18 +310,11 @@ __HowToGalaxy Lectures__ -For experienced scientists, the run through above will have been a breeze. Concepts surrounding galaxy structure and -morphology were already familiar and the statistical techniques used for fitting and modeling already understood. +If you are new to galaxy modeling or want a slower, first-principles walk-through of the statistical techniques +behind it, the **HowToGalaxy** lecture series takes you from the basics through to modeling real galaxy imaging +data. It now lives in its own repository: -For those less familiar with these concepts (e.g. undergraduate students, new PhD students or interested members of the -public), things may have been less clear and a slower more detailed explanation of each concept would be beneficial. - -The **HowToGalaxy** Jupyter Notebook lectures are provide exactly this They are a 3+ chapter guide which thoroughly -take you through the core concepts of galaxy light profiles, teach you the principles ofthe statistical techniques -used in modeling and ultimately will allow you to undertake scientific research like a professional astronomer. - -If this sounds like it suits you, checkout the `autogalaxy_workspace/notebooks/howtogalaxy` package now, its it -recommended you go here before anywhere else! +https://github.com/PyAutoLabs/HowToGalaxy __Features__