-
Notifications
You must be signed in to change notification settings - Fork 56
Add skeleton fab script #246
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Joerg Henrichs (hiker)
wants to merge
47
commits into
MetOffice:main
Choose a base branch
from
hiker:240_add_skeleton_fab_script
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
47 commits
Select commit
Hold shift + click to select a range
4dbac71
#240 Added new base class for Fab scripts.
hiker 2254b31
#240 Removed support for transmute step to keep this PR smaller.
hiker c1e87d1
#240 Add all skeleton fab script, helper classes and site-configurati…
hiker a350ef3
#240 Updated tests to pass all linters, fixed licence statement.
hiker 962ce0a
#240 Updated skeleton build script.
hiker 54491da
#240 Fixed incorrect typing in test.
hiker 22c2ef9
#240 Added README file.
hiker 25c26fb
#240 Removed fab and vernier support from NCI site.
hiker 72192df
#240 Fixed typos.
hiker f7a839c
#240 Allow import of psyclone_tools without setting PYTHONPATH.
hiker 170cab3
#240 Fixed flake8 errors.
hiker d7aa9f1
#240 Added myself to contributors list.
hiker 2315007
Merge branch 'main' into 240_add_skeleton_fab_script
hiker 9ee0b64
Fixed rose picker etc issues raised in review.
hiker 6dc4bd8
Renamed test directory to tests (to avoid clash with .gitignore in ro…
hiker 00b7ec9
#240 Renamed rose_picker_tool.py to rose_picker.py etc.
hiker 6a25195
#240 Addressed issues raised in review.
hiker 1d4c878
Merge remote-tracking branch 'upstream/main' into 240_add_skeleton_fa…
hiker 4872ec9
#240 Removed get_additional_psyclone_options.
hiker 495d651
#240 Fixed configurator to use the new names of the tools.
hiker fa84662
#240 Fixed building current LFRic skeleton due to changes in build sy…
hiker bf87398
#240 Updated tests to work with changes to LFRic build system.
hiker 463faae
#240 Removed unnecessary path for rose picker.
hiker 0c9a599
#240 Set a default as project name.
hiker ac0999c
#240 Fixed incorrect names in templaterator.
hiker 8421891
Merge remote-tracking branch 'upstream/main' into 240_add_skeleton_fa…
hiker 0f42ad1
Merge remote-tracking branch 'upstream/main' into 240_add_skeleton_fa…
hiker c0fc24f
#240 Removed support for old-style environment variables for precision.
hiker 434c394
#240 Removed unnecessary import.
hiker 890a725
#240 Marked the new steps as steps so they get measured.
hiker d98699b
#240 Clarified docstring.
hiker 5f2c892
Updated comments.
hiker 3c3ba13
#240 Handle warning in new steps.
hiker d16a46b
#240 Moved build files into lfric_build directory.
hiker d3d4673
#240 Renamed default compiler setup scripts to 'setup_script_XXX.py'.
hiker a5309c1
#240 Fixed failing tests, handled warning.
hiker 40b1863
Merge branch 'main' into 240_add_skeleton_fab_script
MatthewHambley 9731fa1
#240 Remove unit-test as compilation profile.
hiker c54edab
#240 Removed setting up unit-test for gnu (since unit-test was removed).
hiker 558d9c9
Merge branch 'main' into 240_add_skeleton_fab_script
yaswant 1c7a85f
#240 Updated precision handling as requested in review.
hiker 620cdae
Merge branch '240_add_skeleton_fab_script' of github.com:hiker/lfric_…
hiker b4b7672
Merge branch 'main' into 240_add_skeleton_fab_script
hiker 5ae6376
Merge branch 'main' into 240_add_skeleton_fab_script
hiker 280268b
#240 Addressed issues raised in review.
hiker ecdff1d
#240 Check that NetCDF is available before starting to compile.
hiker cf5b686
Merge remote-tracking branch 'origin/main' into 240_add_skeleton_fab_…
hiker File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,27 +1,28 @@ | ||
| # Contributors | ||
|
|
||
| | GitHub user | Real Name | Affiliation | Date | | ||
| | ---------------- | ---------------------- | ----------- | ---------- | | ||
| | andrewcoughtrie | Andrew Coughtrie | Met Office | 2025.12.12 | | ||
| | james-bruten-mo | James Bruten | Met Office | 2025-12-09 | | ||
| | jedbakerMO | Jed Baker | Met Office | 2025-12-29 | | ||
| | jennyhickson | Jenny Hickson | Met Office | 2025-12-10 | | ||
| | mo-marqh | Mark Hedley | Met Office | 2025-12-11 | | ||
| | mo-rickywong | Ricky Wong | Met Office | 2025-01-30 | | ||
| | mike-hobson | Mike Hobson | Met Office | 2025-12-17 | | ||
| | MatthewHambley | Matthew Hambley | Met Office | 2025-12-15 | | ||
| | mo-lottieturner | Lottie Turner | Met Office | 2025-12-16 | | ||
| | tommbendall | Thomas Bendall | Met Office | 2026-01-23 | | ||
| | yaswant | Yaswant Pradhan | Met Office | 2025-12-16 | | ||
| | stevemullerworth | Steve Mullerworth | Met Office | 2026-01-08 | | ||
| | harry-shepherd | Harry Shepherd | Met Office | 2026-01-08 | | ||
| | EdHone | Ed Hone | Met Office | 2026-01-09 | | ||
| | tom-j-h | Tom Hill | Met Office | 2026-01-19 | | ||
| | mo-alistairp | Alistair Pirrie | Met Office | 2026-01-12 | | ||
| | t00sa | Sam Clarke-Green | Met Office | 2026-01-27 | | ||
| | MetBenjaminWent | Benjamin Went | Met Office | 2026-01-30 | | ||
| | jcsmeto | James Cunningham-Smith | Met Office | 2026-02-06 | | ||
| | thomasmelvin | Thomas Melvin | Met Office | 2026-01-15 | | ||
| | ericaneininger | Erica Neininger | Met Office | 2026-03-02 | | ||
| | mo-lucy-gordon | Lucy Gordon | Met Office | 2026-03-18 | | ||
| | shreybh1 | Shrey Bhardwaj | Met Office | 2026-03-26 | | ||
| | GitHub user | Real Name | Affiliation | Date | | ||
| | ---------------- | ----------------- | --------------------- | ---------- | | ||
| | andrewcoughtrie | Andrew Coughtrie | Met Office | 2025.12.12 | | ||
| | james-bruten-mo | James Bruten | Met Office | 2025-12-09 | | ||
| | jedbakerMO | Jed Baker | Met Office | 2025-12-29 | | ||
| | jennyhickson | Jenny Hickson | Met Office | 2025-12-10 | | ||
| | mo-marqh | Mark Hedley | Met Office | 2025-12-11 | | ||
| | mo-rickywong | Ricky Wong | Met Office | 2025-01-30 | | ||
| | mike-hobson | Mike Hobson | Met Office | 2025-12-17 | | ||
| | MatthewHambley | Matthew Hambley | Met Office | 2025-12-15 | | ||
| | mo-lottieturner | Lottie Turner | Met Office | 2025-12-16 | | ||
| | tommbendall | Thomas Bendall | Met Office | 2026-01-23 | | ||
| | yaswant | Yaswant Pradhan | Met Office | 2025-12-16 | | ||
| | stevemullerworth | Steve Mullerworth | Met Office | 2026-01-08 | | ||
| | harry-shepherd | Harry Shepherd | Met Office | 2026-01-08 | | ||
| | EdHone | Ed Hone | Met Office | 2026-01-09 | | ||
| | tom-j-h | Tom Hill | Met Office | 2026-01-19 | | ||
| | mo-alistairp | Alistair Pirrie | Met Office | 2026-01-12 | | ||
| | t00sa | Sam Clarke-Green | Met Office | 2026-01-27 | | ||
| | MetBenjaminWent | Benjamin Went | Met Office | 2026-01-30 | | ||
| | jcsmeto | James Cunningham-Smith | Met Office | 2026-02-06 | | ||
| | thomasmelvin | Thomas Melvin | Met Office | 2026-01-15 | | ||
| | ericaneininger | Erica Neininger | Met Office | 2026-03-02 | | ||
| | mo-lucy-gordon | Lucy Gordon | Met Office | 2026-03-18 | | ||
| | shreybh1 | Shrey Bhardwaj | Met Office | 2026-03-26 | | ||
| | hiker | Joerg Henrichs | Bureau of Meteorology | 2026-02-11 | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| #!/usr/bin/env python3 | ||
|
|
||
| ############################################################################## | ||
| # (c) Crown copyright Met Office. All rights reserved. | ||
| # The file LICENCE, distributed with this code, contains details of the terms | ||
| # under which the code may be used. | ||
| ############################################################################## | ||
| # Author: J. Henrichs, Bureau of Meteorology | ||
| # Author: J. Lyu, Bureau of Meteorology | ||
|
|
||
| """ | ||
| A FAB build script for applications/skeleton. It relies on | ||
| the LFRicBase class contained in the infrastructure directory. | ||
| """ | ||
|
|
||
| import logging | ||
| from pathlib import Path | ||
| import sys | ||
|
|
||
| from fab.steps.grab.folder import grab_folder | ||
|
|
||
| # We need to import the base class: | ||
| sys.path.insert(0, str(Path(__file__).parents[2] / "lfric_build")) | ||
|
|
||
| from lfric_base import LFRicBase # noqa: E402 | ||
|
|
||
|
|
||
| class FabSkeleton(LFRicBase): | ||
| """ | ||
| A Fab-based build script for skeleton. It relies on the LFRicBase class | ||
| to implement the actual functionality, and only provides the required | ||
| source files. | ||
|
|
||
| :param name: The name of the application. | ||
| """ | ||
|
|
||
| def __init__(self, name: str = "skeleton") -> None: | ||
| super().__init__(name=name) | ||
| # Store the root of this apps for later | ||
| this_file = Path(__file__).resolve() | ||
| self._this_root = this_file.parent | ||
|
|
||
| def grab_files_step(self) -> None: | ||
| """ | ||
| Grabs the required source files and optimisation scripts. | ||
| """ | ||
| super().grab_files_step() | ||
| dirs = ['applications/skeleton/source/'] | ||
|
|
||
| # pylint: disable=redefined-builtin | ||
| for dir in dirs: | ||
| grab_folder(self.config, src=self.lfric_core_root / dir, | ||
| dst_label='') | ||
|
|
||
| # Copy the optimisation scripts into a separate directory | ||
| grab_folder(self.config, src=self._this_root / "optimisation", | ||
| dst_label='optimisation') | ||
|
|
||
| def get_rose_meta(self) -> Path: | ||
| """ | ||
| :returns: the rose-meta.conf path. | ||
| """ | ||
| return (self._this_root / 'rose-meta' / 'lfric-skeleton' / 'HEAD' / | ||
| 'rose-meta.conf') | ||
|
|
||
|
|
||
| # ----------------------------------------------------------------------------- | ||
| if __name__ == '__main__': | ||
|
|
||
| logger = logging.getLogger('fab') | ||
| logger.setLevel(logging.DEBUG) | ||
| fab_skeleton = FabSkeleton() | ||
| fab_skeleton.build() |
|
MatthewHambley marked this conversation as resolved.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| # LFRic Core Fab Build Scripts | ||
|
|
||
| Make sure you have Fab version 2.0.1 or later installed (in addition to all | ||
| LFRic core requirements of course). | ||
|
|
||
| ## Setting up Site- and Platform-specific Settings | ||
| Site- and platform-specific settings are contained in | ||
| ```$LFRIC_CORE/infrastructure/build/site_specific/${SITE}-${PLATFORM}``` | ||
| The default settings are in ```site_specific/default``` (and at this | ||
| stage each other site-specific setup inherits the values set in the default, | ||
| and then adds or modifies settings). The Fab build system provides various | ||
| callbacks to the ```config.py``` file in the corresponding directory (details | ||
| are in the [Fab documentation](https://metoffice.github.io/fab/fab_base/config.html). | ||
|
|
||
| If there is no existing site-specific setup, it is recommended to copy an existing | ||
| configuration file (e.g. from ```nci_gadi/config.py```). This act as a template | ||
| to indicate where you can specify linker information, select a default compiler | ||
| suite etc. | ||
|
|
||
| The default setup contains compiler flags for Cray, GNU, Intel-classic (ifort), | ||
| Intel-LLVM (ifx), and NVIDIA. For modularity's sake (and to keep the file length | ||
| shorter), the default configuration will get the settings from the corresponding | ||
| ```setup_...py``` script. There is no need for a site to replicate this structure, | ||
| existing ```config.py``` scripts show how this can be done. | ||
|
|
||
|
|
||
| ## Building the Skeleton Apps | ||
|
|
||
| In order to build the skeleton apps, change into the directory | ||
| ```$LFRIC_CORE/applications/skeleton```, | ||
| and use the following command: | ||
|
|
||
| ``` | ||
| ./fab_skeleton.py --nprocs 4 --suite gnu | ||
| ``` | ||
| Select an appropriate number of processes to run in parallel, and a compiler | ||
| suite, e.g. one of ```gnu```, ```cray```, ```intel-classic```, ```intel-llvm``` | ||
| or ```nvidia```. Once the process is finished, you should have a binary in the | ||
| directory ```./fab-workspace/skeleton-full-debug-COMPILER``` (where | ||
| ```COMPILER``` is the compiler used, e.g. ```mpif90-gfortran```). | ||
|
|
||
| Likely, you will have to setup corresponding compiler and linker options, e.g. | ||
| include paths, library paths and libraries to link. The directory ```site_specific``` | ||
| contains site-specific setup files, and one called ```default``` (which is | ||
| used in the above example if no site is selected). If your site does not exist, | ||
| create a corresponding directory for your site and platform in the format | ||
| ```site_platform``` by using an existing site as template. See also the | ||
| [Fab documentation](https://metoffice.github.io/fab/fab_base/config.html) | ||
| for details about specifying compiler and linker options). | ||
| You can then use the command line options ```--site``` and ```--platform``` | ||
| to pick your setup, e.g.: | ||
|
|
||
| ``` | ||
| ./fab_skeleton.py --nprocs 4 --site nci --platform gadi --suite intel-llvm | ||
| ``` | ||
|
|
||
| This would use the file ```site_specific/nci_gadi/config.py```, and all additional | ||
| compiler and linker options defined there. | ||
|
|
||
| Using ```./fab_skeleton.py -h``` will show a help message with all supported command line | ||
| options (and their default value). If a default value is listed using an environment | ||
| variables (```(default: $SITE or 'default')```), the corresponding environment variable | ||
| is used if no command line option has been specified. | ||
|
|
||
| A different compilation profile can be specified using ```--profile``` option. Note | ||
| that the available compilation profiles can vary from site to site (see | ||
| [Fab documentation](https://metoffice.github.io/fab/fab_base/config.html) for details). | ||
|
|
||
| If Fab has issues finding a compiler, you can use the Fab debug command line option | ||
| ```--available-compilers```, which will list all compilers and linkers Fab has | ||
| identified as being available. |
|
MatthewHambley marked this conversation as resolved.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,103 @@ | ||
| ############################################################################## | ||
| # (c) Crown copyright Met Office. All rights reserved. | ||
| # The file LICENCE, distributed with this code, contains details of the terms | ||
| # under which the code may be used. | ||
| ############################################################################## | ||
| # Author J. Henrichs, Bureau of Meteorology | ||
| # Author J. Lyu, Bureau of Meteorology | ||
|
|
||
| """ | ||
| This file defines the configurator script sequence for LFRic. | ||
| """ | ||
|
|
||
| import logging | ||
| from pathlib import Path | ||
| from typing import cast, Optional | ||
|
|
||
| from fab.api import BuildConfig, find_source_files, Category | ||
| from fab.tools.shell import Shell | ||
|
|
||
| from rose_picker import RosePicker | ||
|
|
||
| logger = logging.getLogger('fab') | ||
|
|
||
|
|
||
| def configurator(config: BuildConfig, | ||
| lfric_core_source: Path, | ||
| rose_meta_conf: Path, | ||
| include_paths: Optional[list[Path]] = None, | ||
| config_dir: Optional[Path] = None) -> None: | ||
| """ | ||
| This method implements the LFRic configurator tool. | ||
|
|
||
| :param config: the Fab build config instance | ||
| :param lfric_core_source: the path to the LFRic core directory | ||
| :param rose_meta_conf: the path to the rose-meta configuration file | ||
| :param include_paths: additional include paths (each path will be added, | ||
| as well as the path with /'rose-meta') | ||
| :param config_dir: the directory for the generated configuration files | ||
| """ | ||
|
|
||
| tools = lfric_core_source / 'infrastructure' / 'build' / 'tools' | ||
| config_dir = config_dir or config.build_output / 'configuration' | ||
| config_dir.mkdir(parents=True, exist_ok=True) | ||
|
|
||
| # rose picker | ||
| # ----------- | ||
| # creates rose-meta.json and config_namelists.txt in | ||
| # gungho/build | ||
| logger.info('rose_picker') | ||
|
|
||
| include_dirs = [lfric_core_source / 'rose-meta'] | ||
| if include_paths: | ||
| for path in include_paths: | ||
| include_dirs.append(path / 'rose-meta') | ||
|
|
||
| rose_picker = RosePicker() | ||
| rose_picker.execute(rose_meta_conf, config_dir, | ||
| include_paths=include_dirs) | ||
| rose_meta = config_dir / 'rose-meta.json' | ||
|
|
||
| shell = config.tool_box.get_tool(Category.SHELL) | ||
| shell = cast(Shell, shell) | ||
|
|
||
| # build_config_loaders | ||
| # -------------------- | ||
| # builds a bunch of f90s from the json | ||
| logger.info('GenerateNamelistLoader') | ||
| shell.exec(f"{tools / 'GenerateNamelistLoader'} -verbose {rose_meta} " | ||
| f"-directory {config_dir}") | ||
|
|
||
| # create configuration_mod.f90 in source root | ||
| # ------------------------------------------- | ||
| logger.info('GenerateConfigLoader') | ||
| with open(config_dir / 'config_namelists.txt', encoding="utf8") as f_in: | ||
| names = [name.strip() for name in f_in.readlines()] | ||
|
|
||
| shell.exec(f"{tools / 'GenerateConfigLoader'} " | ||
| f"{' '.join(names)} " | ||
| f"-o {config_dir}") | ||
|
|
||
| logger.info('GenerateExtendedNamelistType') | ||
| shell.exec(f"{tools / 'GenerateExtendedNamelistType'} {rose_meta} " | ||
| f"-directory {config_dir}") | ||
|
|
||
| duplicates: list[str] = [] | ||
| with open(config_dir / 'duplicate_namelists.txt', encoding="utf8") as f_in: | ||
| for name in f_in.readlines(): | ||
| duplicates.extend(["-duplicate", name.strip()]) | ||
|
|
||
| logger.info('GenerateConfigType') | ||
| shell.exec(f"{tools / 'GenerateConfigType'} " | ||
| f"{' '.join(names)} " | ||
| f"{' '.join(duplicates)} " | ||
| f"-o {config_dir}") | ||
|
|
||
| # create feign_config_mod.f90 in source root | ||
| # ------------------------------------------ | ||
| logger.info('GenerateFeigns') | ||
| feign_config_mod_fpath = config_dir / 'feign_config_mod.f90' | ||
| shell.exec(f"{tools / 'GenerateFeigns'} {rose_meta} " | ||
| f"-output {feign_config_mod_fpath}") | ||
|
|
||
| find_source_files(config, source_root=config_dir) |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.