Skip to content

add 1-hour and 2-hour retries for all tasks except pp-starter #243

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ceblanton merged 2 commits into from
Jun 11, 2026
Merged

add 1-hour and 2-hour retries for all tasks except pp-starter #243

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ce275a2
#240 #242 add 1-hour and 2-hour retries for all tasks except pp-star…
ceblanton Jun 11, 2026
2383713
#242 Set default retries to be a jinja variable that the test pipeli…
ceblanton Jun 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions flow.cylc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,12 @@
{% set STALL_TIMEOUT = "P1W" %}
{% endif %}

{# Set default retries unless it is already set by the test pipeline #}
{# Retry all tasks twice- once after an hour, second after two hours #}
{% if DEFAULT_RETRIES is not defined %}
{% set DEFAULT_RETRIES = "PT1H, PT2H" %}
{% endif %}

{# Set ANALYSIS_START and ANALYSIS_STOP if they do not exist #}
{% if ANALYSIS_START is not defined %}
{% set ANALYSIS_START = PP_START %}
Expand Down Expand Up @@ -109,8 +115,8 @@
# Runahead limit specifies the number of cycle points that will spawn ahead of the oldest incomplete task.
# We don't need it because we specify the final cycle point, and it decreases robustness
# as it prevents future tasks from running due to a previous failed task.
# As runahead limit is required, we will set it to 99999.
runahead limit = P99999
# As runahead limit is required, we will set it to 999.
runahead limit = P999
[[queues]]
# Limit the entire workflow to 100 active tasks. Only allow a single cleaning
# or data-catalog task to run at once, as they can fail if run in parallel.
Expand Down Expand Up @@ -289,11 +295,15 @@

[runtime]
[[root]]
execution retry delays = {{ DEFAULT_RETRIES }}
[[[environment]]]
fre_yaml = $CYLC_WORKFLOW_RUN_DIR/{{ YAML }}

[[pp-starter]]
inherit = PP-STARTER
# Note: pp-starter is a fake task and should be rephrased as a proper trigger.

@singhd789 singhd789 Jun 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait how is it a "fake" task?

@ceblanton ceblanton Jun 11, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does zero productive work. It's merely a trigger for the other tasks. Cylc has proper mechanisms for triggers, and we just never got there yet. You can see the modern triggers in Alex's workflows

@singhd789 singhd789 Jun 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see. I guess I saw this as a task because it does work to resolve the target file. Either way, it's just a comment I suppose

# No task retries for this fake task.
execution retry delays =
# NOTE! script must appear *before* [[[enviroment]]] or else
# the job scripts will have quoting issues
script = """
Expand Down
3 changes: 2 additions & 1 deletion for_gh_runner/runscript.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,8 @@ fre_pp_steps () {

## RUN
echo -e "\nRunning the workflow with cylc play ..."
cylc play --no-detach --debug -s 'STALL_TIMEOUT="PT0S"' ${name}
# set these two jinja variables to disable task retries and set the stall timer to zero
cylc play --no-detach --debug -s 'STALL_TIMEOUT="PT0S"' -s 'DEFAULT_RETRIES=""' ${name}
#check_exit_status "PLAY" # if cylc play fails and this is not commented, log uploading does not work

## SUMMARY
Expand Down
Loading