Only update RTIF for terminal task states

[amoghrajesh]

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[ashb]

Needs some thinking about this -- I'm not sure this is the correct behaviour

[amoghrajesh]

Thanks, I anticipated that too. Let me come up with a design for that

[pankajkoti]

At the moment, we're observing that the rtif endpoint does not exist for such intermediate try_numbers of the task run and the post-processing is blocked (e.g. uploading of logs to remote logging store & the logs are not available for those initial try_numbers of the task run) due to the AirflowRuntimeError that gets raised from the flow here.

Logs from the worker

Assigning outlets with DatasetAlias in Airflow 3
[2026-03-16 15:48:23] ERROR - Task failed with exception
CosmosDbtRunError: error here
File "/usr/local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1112 in run
File "/usr/local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1528 in _execute_task
File "/usr/local/astronomer-cosmos/cosmos/operators/base.py", line 342 in execute
File "/usr/local/astronomer-cosmos/cosmos/operators/local.py", line 895 in build_and_run_cmd
File "/usr/local/astronomer-cosmos/cosmos/operators/local.py", line 692 in run_command
File "/usr/local/astronomer-cosmos/cosmos/operators/local.py", line 273 in handle_exception_dbt_runner
File "/usr/local/astronomer-cosmos/cosmos/dbt/runner.py", line 153 in handle_exception_if_needed

[2026-03-16 15:48:23] ERROR - Failed to set rendered fields during finalization task_id=raw_customers_seed dag_id=basic_cosmos_dag
AirflowRuntimeError: API_SERVER_ERROR: {'status_code': 404, 'message': 'Not Found', 'detail': {'detail': 'Not Found'}}
File "/usr/local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1627 in finalize
File "/usr/local/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py", line 206 in send
File "/usr/local/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py", line 270 in _get_response
File "/usr/local/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py", line 257 in _from_frame

404 errors we're seeing in the API server

2026-03-16T10:22:46.992666Z [info     ] Task instance state updated    [airflow.api_fastapi.execution_api.routes.task_instances] loc=task_instances.py:424 new_state=up_for_retry rows_affected=1 ti_id=019cf62b-1924-76c0-b29b-696f4c9867c0
192.168.97.6:34318 - "PATCH /execution/task-instances/019cf62b-1924-76c0-b29b-696f4c9867c0/state HTTP/1.1" 204
2026-03-16T10:22:47.002805Z [info     ] Updating RenderedTaskInstanceFields [airflow.api_fastapi.execution_api.routes.task_instances] field_count=10 loc=task_instances.py:687 ti_id=019cf62b-1924-76c0-b29b-696f4c9867c0
2026-03-16T10:22:47.003740Z [error    ] Task Instance not found        [airflow.api_fastapi.execution_api.routes.task_instances] loc=task_instances.py:691 ti_id=019cf62b-1924-76c0-b29b-696f4c9867c0
192.168.97.6:34318 - "PUT /execution/task-instances/019cf62b-1924-76c0-b29b-696f4c9867c0/rtif HTTP/1.1" 404
2026-03-16T10:22:47.176682Z [info     ] Task instance state updated    [airflow.api_fastapi.execution_api.routes.task_instances] loc=task_instances.py:424 new_state=up_for_retry rows_affected=1 ti_id=019cf62b-1922-7cc4-900b-ac0a636cb0b0
192.168.97.6:34312 - "PATCH /execution/task-instances/019cf62b-1922-7cc4-900b-ac0a636cb0b0/state HTTP/1.1" 204
2026-03-16T10:22:47.187138Z [info     ] Updating RenderedTaskInstanceFields [airflow.api_fastapi.execution_api.routes.task_instances] field_count=10 loc=task_instances.py:687 ti_id=019cf62b-1922-7cc4-900b-ac0a636cb0b0
2026-03-16T10:22:47.188136Z [error    ] Task Instance not found        [airflow.api_fastapi.execution_api.routes.task_instances] loc=task_instances.py:691 ti_id=019cf62b-1922-7cc4-900b-ac0a636cb0b0
192.168.97.6:34312 - "PUT /execution/task-instances/019cf62b-1922-7cc4-900b-ac0a636cb0b0/rtif HTTP/1.1" 404
2026-03-16T10:22:48.079401Z [info     ] Task instance state updated    [airflow.api_fastapi.execution_api.routes.task_instances] loc=task_instances.py:424 new_state=up_for_retry rows_affected=1 ti_id=019cf62b-1923-70cb-a656-1cc31f3bd788
192.168.97.6:34320 - "PATCH /execution/task-instances/019cf62b-1923-70cb-a656-1cc31f3bd788/state HTTP/1.1" 204
2026-03-16T10:22:48.090230Z [info     ] Updating RenderedTaskInstanceFields [airflow.api_fastapi.execution_api.routes.task_instances] field_count=10 loc=task_instances.py:687 ti_id=019cf62b-1923-70cb-a656-1cc31f3bd788
2026-03-16T10:22:48.091175Z [error    ] Task Instance not found        [airflow.api_fastapi.execution_api.routes.task_instances] loc=task_instances.py:691 ti_id=019cf62b-1923-70cb-a656-1cc31f3bd788
192.168.97.6:34320 - "PUT /execution/task-instances/019cf62b-1923-70cb-a656-1cc31f3bd788/rtif HTTP/1.1" 404

[amoghrajesh]

Thanks for sharing that @pankajkoti, I am thinking of a better way to do things

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[pankajkoti]

hi @amoghrajesh, I just filed a new issue #66416 that lists the bare minimum steps required to reproduce this issue outside of Cosmos. This will help when the fix needs to be tested.

[FrankYang0529]

I create another PR to return 410 in setting rtif API if the id is stale. The supervisor ignores 410 case, but still raises error for 404. So we don't need to rely on task instance state to update rtif. WDYT?

#68902