Skip to content

Executor Synchronous callback workload#61153

Merged
ferruzzi merged 32 commits intoapache:mainfrom
aws-mwaa:ferruzzi/executor-callbacks/sync-callbacks
Feb 28, 2026
Merged

Executor Synchronous callback workload#61153
ferruzzi merged 32 commits intoapache:mainfrom
aws-mwaa:ferruzzi/executor-callbacks/sync-callbacks

Conversation

@ferruzzi
Copy link
Contributor

Add support for the Callback workload to be run in the executors. Other executors will need to be updated before they can support the workload, but I tried to make it as non-invasive as I could.

This is the bulk of the work required to allow synchronous callbacks to be used in DeadlineAlerts. For example, this now works in LocalExecutor:

with DAG(
    dag_id="sync_deadline",
    deadline=DeadlineAlert(
        reference=DeadlineReference.FIXED_DATETIME(datetime(1980, 8, 10, 2)),
        interval=timedelta(0),
        callback=SyncCallback(
            SlackWebhookNotifier,
            {"text": "Sync Callback; Alert should trigger immediately!"},
        )
    )
):
    EmptyOperator(task_id='empty_task')

Co-author: Builds on work handed off by @seanghaeli and research from @ramitkataria; if I did this right then they should be getting co-author credits, I think?


Was generative AI tooling used to co-author this PR?
  • [ x ] Yes (please specify the tool below)
    Cline (Claude Sonnet 4.5) was used for debugging and suggesting some unit test edge cases.

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@ferruzzi ferruzzi requested a review from dstandish as a code owner February 4, 2026 22:02
@ferruzzi ferruzzi force-pushed the ferruzzi/executor-callbacks/sync-callbacks branch 6 times, most recently from e39a3b2 to eab0d0f Compare February 9, 2026 17:41
Copy link
Contributor

@o-nikolas o-nikolas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some comments, but overall a large/comprehensive PR that's doing some heavy lifting. Pretty cool what's being setup here, but needs careful review by some more folks I think.

@ramitkataria should have a look through the callbacks stuff

And then maybe @jscheffl for another pair of eyes on the executor/scheduler bits?

@ferruzzi ferruzzi force-pushed the ferruzzi/executor-callbacks/sync-callbacks branch 3 times, most recently from 2e1dbde to a322da1 Compare February 18, 2026 20:42
@ferruzzi ferruzzi force-pushed the ferruzzi/executor-callbacks/sync-callbacks branch from dbf3a52 to e7622f1 Compare February 28, 2026 16:22
@ferruzzi ferruzzi merged commit 9fa13c8 into apache:main Feb 28, 2026
129 checks passed
@ferruzzi ferruzzi deleted the ferruzzi/executor-callbacks/sync-callbacks branch February 28, 2026 17:59
@ferruzzi ferruzzi mentioned this pull request Mar 4, 2026
2 tasks
dominikhei pushed a commit to dominikhei/airflow that referenced this pull request Mar 11, 2026
* Synchronous callback support for BaseExecutor, LocalExecutor, and CeleryExecutor

Add support for the Callback workload to be run in the executors.  Other executors will need to be updated before the can support the workload, but I tried to make it as non-invasive as I could.


Co-authored-by: Sean Ghaeli <seanghaeli@gmail.com>
Co-authored-by: Ramit Kataria <ramitkat@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:deadline-alerts AIP-86 (former AIP-57) area:Executors-core LocalExecutor & SequentialExecutor area:providers area:Scheduler including HA (high availability) scheduler area:task-sdk provider:celery

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants