Skip to content

Test replacing/removing -xCORE-AVX2#138

Draft
dougiesquire wants to merge 19 commits intomainfrom
mavx2
Draft

Test replacing/removing -xCORE-AVX2#138
dougiesquire wants to merge 19 commits intomainfrom
mavx2

Conversation

@dougiesquire
Copy link
Collaborator

@dougiesquire dougiesquire commented Feb 24, 2026

This PR is to test removing/replacing -xCORE-AVX2 across the ACCESS-OM2 stack


🚀 The latest prerelease access-om2/pr138-11 at e44855d is here: #138 (comment) 🚀


🚀 The latest prerelease access-om2/pr138-19 at 89234e2 is here: #138 (comment) 🚀

@dougiesquire dougiesquire self-assigned this Feb 24, 2026
@github-actions
Copy link
Contributor

🚀 Attempted to deploy access-om2 Prerelease pr138-1 with commit 329cf77

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om2, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.02.000 as a Release (when merged).
  • pr138-1 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om2/pr138-1

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om2-pr138-1 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to config/versions.json in this PR, or propose changes to build-cds config/settings.json.

@dougiesquire
Copy link
Collaborator Author

!update-configs

@github-actions
Copy link
Contributor

🔧 Opening/Updating Model Configuration PRs in ACCESS-NRI/access-om2-configs

Configurations Requested

Configurations requested from profile default:

  • dev-1deg_jra55_ryf
Pull Requests Opened/Updated

The following PRs were opened/updated:

More details can be found in the workflow run: https://github.com/ACCESS-NRI/ACCESS-OM2/actions/runs/22376965677

@dougiesquire
Copy link
Collaborator Author

dougiesquire commented Feb 25, 2026

From the linked PR above, with pr138-1:

✅ test_repro_historical
✅ test_repro_determinism
✅ test_repro_restart

@dougiesquire
Copy link
Collaborator Author

Hmm, this does appear to have a reasonable negative impact on performance. I ran this dev-1deg_jra55_ryf configuration with the following changes:

Exp 1

  • Use access-om2/2026.02.000
  • Set restart_format = 'pio' in cice_in.nml

Exp 2

  • Use access-om2/pr138-1 (this PR)
  • Set restart_format = 'pio' in cice_in.nml

I changed the run length to 1 year and ran payu run -n 3. These are the run durations:

Exp 1

$ grep "payu_run_duration_seconds" archive/payu_jobs/*/run/*.gadi-pbs.json
archive/payu_jobs/0/run/161644452.gadi-pbs.json:        "payu_run_duration_seconds": 839.5187113920692,
archive/payu_jobs/1/run/161645931.gadi-pbs.json:        "payu_run_duration_seconds": 843.7589967059903,
archive/payu_jobs/2/run/161647049.gadi-pbs.json:        "payu_run_duration_seconds": 838.253176500788,

Exp 2

$ grep "payu_run_duration_seconds" archive/payu_jobs/*/run/*.gadi-pbs.json
archive/payu_jobs/0/run/161646511.gadi-pbs.json:        "payu_run_duration_seconds": 945.3608389219735,
archive/payu_jobs/1/run/161647985.gadi-pbs.json:        "payu_run_duration_seconds": 945.074181515025,
archive/payu_jobs/2/run/161649727.gadi-pbs.json:        "payu_run_duration_seconds": 944.211514512077,

So ~12% slower with -mavx2 than with -xCORE-AVX2

@github-actions
Copy link
Contributor

github-actions bot commented Mar 1, 2026

🚀 Attempted to deploy access-om2 Prerelease pr138-2 with commit d7fdf52

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om2, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.02.000 as a Release (when merged).
  • pr138-2 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om2/pr138-2

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om2-pr138-2 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to config/versions.json in this PR, or propose changes to build-cds config/settings.json.

@dougiesquire
Copy link
Collaborator Author

!update-configs

@github-actions
Copy link
Contributor

github-actions bot commented Mar 1, 2026

🔧 Opening/Updating Model Configuration PRs in ACCESS-NRI/access-om2-configs

Configurations Requested

Configurations requested from profile default:

  • dev-1deg_jra55_ryf
Pull Requests Opened/Updated

The following PRs were opened/updated:

More details can be found in the workflow run: https://github.com/ACCESS-NRI/ACCESS-OM2/actions/runs/22555440525

@dougiesquire
Copy link
Collaborator Author

With pr138-2:
✅ test_repro_historical
✅ test_repro_determinism
✅ test_repro_restart

@dougiesquire dougiesquire changed the title Test replacing -xCORE-AVX2 with -mavx2 Test replacing/removing -xCORE-AVX2 Mar 1, 2026
@manodeep
Copy link
Contributor

manodeep commented Mar 2, 2026

@dougiesquire To use the cpu name specific targets, you will likely need to set the concretizer config concretizer->targets->granularity=microarchitectures. It's set to generic by default.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 2, 2026

🚀 Attempted to deploy access-om2 Prerelease pr138-3 with commit d22ab30

🖥️ Gadi Deployment ❌

(@dougiesquire cancelled this due to #138 (comment))

@dougiesquire
Copy link
Collaborator Author

Performance with -mavx2 removed from all packages (so no -xCORE-avx2 or -mavx2)


I ran this dev-1deg_jra55_ryf configuration with the following changes:

Exp 1

  • Use access-om2/2026.02.000
  • Set restart_format = 'pio' in cice_in.nml

Exp 2

  • Use access-om2/pr138-2 (this PR)
  • Set restart_format = 'pio' in cice_in.nml

I changed the run length to 1 year and ran payu run -n 3. These are the run durations:

Exp 1

$ grep "payu_run_duration_seconds" archive/payu_jobs/*/run/*.gadi-pbs.json
archive/payu_jobs/0/run/162056547.gadi-pbs.json:        "payu_run_duration_seconds": 833.1820627681445,
archive/payu_jobs/1/run/162059170.gadi-pbs.json:        "payu_run_duration_seconds": 830.3039888441563,
archive/payu_jobs/2/run/162061363.gadi-pbs.json:        "payu_run_duration_seconds": 836.3757481910288,

Exp 2

$ grep "payu_run_duration_seconds" archive/payu_jobs/*/run/*.gadi-pbs.json
archive/payu_jobs/0/run/162056595.gadi-pbs.json:        "payu_run_duration_seconds": 939.527026204858,
archive/payu_jobs/1/run/162059610.gadi-pbs.json:        "payu_run_duration_seconds": 944.9940079760272,
archive/payu_jobs/2/run/162062341.gadi-pbs.json:        "payu_run_duration_seconds": 948.0654471958987,

So still ~12% slower

@github-actions
Copy link
Contributor

github-actions bot commented Mar 2, 2026

🚀 Attempted to deploy access-om2 Prerelease pr138-4 with commit 2a2ec93

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om2, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.02.000 as a Release (when merged).
  • pr138-4 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om2/pr138-4

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om2-pr138-4 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to config/versions.json in this PR, or propose changes to build-cds config/settings.json.

@dougiesquire
Copy link
Collaborator Author

!update-configs

@github-actions
Copy link
Contributor

github-actions bot commented Mar 2, 2026

🔧 Opening/Updating Model Configuration PRs in ACCESS-NRI/access-om2-configs

Configurations Requested

Configurations requested from profile default:

  • dev-1deg_jra55_ryf
Pull Requests Opened/Updated

The following PRs were opened/updated:

More details can be found in the workflow run: https://github.com/ACCESS-NRI/ACCESS-OM2/actions/runs/22557435246

@dougiesquire
Copy link
Collaborator Author

With pr138-4:
✅ test_repro_historical
✅ test_repro_determinism
✅ test_repro_restart

@dougiesquire
Copy link
Collaborator Author

dougiesquire commented Mar 2, 2026

Performance with target=cascadelake


I ran this dev-1deg_jra55_ryf configuration with the following changes:

Exp 1

  • Use access-om2/2026.02.000
  • Set restart_format = 'pio' in cice_in.nml

Exp 2

  • Use access-om2/pr138-4 (this PR)
  • Set restart_format = 'pio' in cice_in.nml

I changed the run length to 1 year and ran payu run. These are the run durations:

Exp 1

$ grep "payu_run_duration_seconds" archive/payu_jobs/*/run/*.gadi-pbs.json
archive/payu_jobs/0/run/162071407.gadi-pbs.json:        "payu_run_duration_seconds": 832.9791051410139,

Exp 2

$ grep "payu_run_duration_seconds" archive/payu_jobs/*/run/*.gadi-pbs.json
archive/payu_jobs/0/run/162071402.gadi-pbs.json:        "payu_run_duration_seconds": 945.2046530928928,

So still ~12% slower

@dougiesquire
Copy link
Collaborator Author

!update-configs

@github-actions
Copy link
Contributor

github-actions bot commented Mar 4, 2026

🔧 Opening/Updating Model Configuration PRs in ACCESS-NRI/access-om2-configs

Configurations Requested

Configurations requested from profile default:

  • dev-01deg_jra55_ryf

  • dev-025deg_jra55_ryf

  • dev-1deg_jra55_ryf

Pull Requests Opened/Updated

The following PRs were opened/updated:

More details can be found in the workflow run: https://github.com/ACCESS-NRI/ACCESS-OM2/actions/runs/22655605101

@dougiesquire
Copy link
Collaborator Author

dougiesquire commented Mar 4, 2026

Performance with no flags and target=x86_64_v2


I ran this dev-1deg_jra55_ryf configuration with the following changes:

Exp 1

  • Use access-om2/2026.02.000
  • Set restart_format = 'pio' in cice_in.nml

Exp 2

  • Use access-om2/pr138-13 (this PR)
  • Set restart_format = 'pio' in cice_in.nml

I changed the run length to 1 year and ran payu run. These are the run durations:

Exp 1

$ grep "payu_run_duration_seconds" archive/payu_jobs/*/run/*.gadi-pbs.json
        "payu_run_duration_seconds": 837.9251451641321,

Exp 2

$ grep "payu_run_duration_seconds" archive/payu_jobs/*/run/*.gadi-pbs.json
        "payu_run_duration_seconds": 860.9682809540536,

So ~3% slower

@github-actions
Copy link
Contributor

github-actions bot commented Mar 4, 2026

🚀 Attempted to deploy access-om2 Prerelease pr138-14 with commit d1ee94d

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om2, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.02.001 as a Release (when merged).
  • pr138-14 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om2/pr138-14

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om2-pr138-14 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to config/versions.json in this PR, or propose changes to build-cds config/settings.json.

@dougiesquire
Copy link
Collaborator Author

!update-configs

@github-actions
Copy link
Contributor

github-actions bot commented Mar 4, 2026

🔧 Opening/Updating Model Configuration PRs in ACCESS-NRI/access-om2-configs

Configurations Requested

Configurations requested from profile default:

  • dev-01deg_jra55_ryf

  • dev-025deg_jra55_ryf

  • dev-1deg_jra55_ryf

Pull Requests Opened/Updated

The following PRs were opened/updated:

More details can be found in the workflow run: https://github.com/ACCESS-NRI/ACCESS-OM2/actions/runs/22657954237

@github-actions
Copy link
Contributor

github-actions bot commented Mar 4, 2026

🚀 Attempted to deploy access-om2 Prerelease pr138-15 with commit c2075cf

🖥️ Gadi Deployment ❌

@github-actions
Copy link
Contributor

github-actions bot commented Mar 4, 2026

🚀 Attempted to deploy access-om2 Prerelease pr138-16 with commit f9f50bf

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om2, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.02.001 as a Release (when merged).
  • pr138-16 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om2/pr138-16

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om2-pr138-16 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to config/versions.json in this PR, or propose changes to build-cds config/settings.json.

@dougiesquire
Copy link
Collaborator Author

!update-configs

@github-actions
Copy link
Contributor

github-actions bot commented Mar 4, 2026

🔧 Opening/Updating Model Configuration PRs in ACCESS-NRI/access-om2-configs

Configurations Requested

Configurations requested from profile default:

  • dev-01deg_jra55_ryf

  • dev-025deg_jra55_ryf

  • dev-1deg_jra55_ryf

Pull Requests Opened/Updated

The following PRs were opened/updated:

More details can be found in the workflow run: https://github.com/ACCESS-NRI/ACCESS-OM2/actions/runs/22660011408

@dougiesquire
Copy link
Collaborator Author

Performance with no flags and target=x86_64


I ran this dev-1deg_jra55_ryf configuration with the following changes:

Exp 1

  • Use access-om2/2026.02.000
  • Set restart_format = 'pio' in cice_in.nml

Exp 2

  • Use access-om2/pr138-14 (this PR)
  • Set restart_format = 'pio' in cice_in.nml

I changed the run length to 1 year and ran payu run. These are the run durations:

Exp 1

$ grep "payu_run_duration_seconds" archive/payu_jobs/*/run/*.gadi-pbs.json
        "payu_run_duration_seconds": 843.5014327599201,

Exp 2

grep "payu_run_duration_seconds" archive/payu_jobs/*/run/*.gadi-pbs.json
        "payu_run_duration_seconds": 874.2103985729627,

So ~4% slower

@github-actions
Copy link
Contributor

github-actions bot commented Mar 4, 2026

🚀 Attempted to deploy access-om2 Prerelease pr138-17 with commit efd6395

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om2, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.02.001 as a Release (when merged).
  • pr138-17 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om2/pr138-17

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om2-pr138-17 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to config/versions.json in this PR, or propose changes to build-cds config/settings.json.

@dougiesquire
Copy link
Collaborator Author

Performance with no flags and target=haswell


I ran this dev-1deg_jra55_ryf configuration with the following changes:

Exp 1

  • Use access-om2/2026.02.000
  • Set restart_format = 'pio' in cice_in.nml

Exp 2

  • Use access-om2/pr138-16 (this PR)
  • Set restart_format = 'pio' in cice_in.nml

I changed the run length to 1 year and ran payu run -n 3. These are the run durations:

Exp 1

$ grep "payu_run_duration_seconds" archive/payu_jobs/*/run/*.gadi-pbs.json
archive/payu_jobs/0/run/162270387.gadi-pbs.json:        "payu_run_duration_seconds": 838.8389234167989,
archive/payu_jobs/1/run/162274169.gadi-pbs.json:        "payu_run_duration_seconds": 841.6124492289964,
archive/payu_jobs/2/run/162276958.gadi-pbs.json:        "payu_run_duration_seconds": 842.9676855828147,

Exp 2

$ grep "payu_run_duration_seconds" archive/payu_jobs/*/run/*.gadi-pbs.json
archive/payu_jobs/0/run/162270386.gadi-pbs.json:        "payu_run_duration_seconds": 892.1444520510267,
archive/payu_jobs/1/run/162274170.gadi-pbs.json:        "payu_run_duration_seconds": 894.877702570986,
archive/payu_jobs/2/run/162276957.gadi-pbs.json:        "payu_run_duration_seconds": 890.5481417709962,

So ~6% slower

@github-actions
Copy link
Contributor

github-actions bot commented Mar 4, 2026

🚀 Attempted to deploy access-om2 Prerelease pr138-18 with commit 4ad5704

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om2, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.02.001 as a Release (when merged).
  • pr138-18 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om2/pr138-18

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om2-pr138-18 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to config/versions.json in this PR, or propose changes to build-cds config/settings.json.

@dougiesquire
Copy link
Collaborator Author

Performance with -march=core-avx2 flags in spack.packages.all.prefer


I ran this dev-1deg_jra55_ryf configuration with the following changes:

Exp 1

  • Use access-om2/2026.02.000
  • Set restart_format = 'pio' in cice_in.nml

Exp 2

  • Use access-om2/pr138-18 (this PR)
  • Set restart_format = 'pio' in cice_in.nml

I changed the run length to 1 year and ran payu run -n 2. These are the run durations:

Exp 1

$ grep "payu_run_duration_seconds" archive/payu_jobs/*/run/*.gadi-pbs.json
archive/payu_jobs/0/run/162306425.gadi-pbs.json:        "payu_run_duration_seconds": 834.740109429229,
archive/payu_jobs/1/run/162307246.gadi-pbs.json:        "payu_run_duration_seconds": 842.9836442731321,

Exp 2

$ grep "payu_run_duration_seconds" archive/payu_jobs/*/run/*.gadi-pbs.json
archive/payu_jobs/0/run/162306426.gadi-pbs.json:        "payu_run_duration_seconds": 921.1132797608152,
archive/payu_jobs/1/run/162307314.gadi-pbs.json:        "payu_run_duration_seconds": 889.0010685380548,

So ~6% slower

@github-actions
Copy link
Contributor

github-actions bot commented Mar 4, 2026

🚀 Attempted to deploy access-om2 Prerelease pr138-19 with commit 89234e2

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om2, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.02.001 as a Release (when merged).
  • pr138-19 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om2/pr138-19

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om2-pr138-19 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to config/versions.json in this PR, or propose changes to build-cds config/settings.json.

@dougiesquire
Copy link
Collaborator Author

Performance with -march=core-avx2 only for key dependencies (cice5, mom5, libaccessom2, oasis3-mct, access-fms, access-generic-tracers, access-fms)


I ran this dev-1deg_jra55_ryf configuration with the following changes:

Exp 1

  • Use access-om2/2026.02.000
  • Set restart_format = 'pio' in cice_in.nml

Exp 2

  • Use access-om2/pr138-19 (this PR)
  • Set restart_format = 'pio' in cice_in.nml

I changed the run length to 1 year and ran payu run -n 2. These are the run durations:

Exp 1

$ grep "payu_run_duration_seconds" archive/payu_jobs/*/run/*.gadi-pbs.json
archive/payu_jobs/0/run/162310337.gadi-pbs.json:        "payu_run_duration_seconds": 842.1666844459251,
archive/payu_jobs/1/run/162311232.gadi-pbs.json:        "payu_run_duration_seconds": 840.5417929301038,

Exp 2

$ grep "payu_run_duration_seconds" archive/payu_jobs/*/run/*.gadi-pbs.json
archive/payu_jobs/0/run/162310338.gadi-pbs.json:        "payu_run_duration_seconds": 891.6344424730632,
archive/payu_jobs/1/run/162311416.gadi-pbs.json:        "payu_run_duration_seconds": 895.2898426400498,

So ~6% slower

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants