Add async connection testing via workers for security isolation

[anishgirianish]

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

Summary

Follows the direction proposed by @potiuk in #59643 to move connection testing off the API server and onto workers.

Connection testing has been disabled by default since Airflow 2.7.0 because executing user-supplied driver code (ODBC/JDBC) on the API server poses security risks, and workers typically have network access to external systems that API servers don't.

This moves the whole thing onto workers. A dedicated TestConnection workload goes through the scheduler, gets dispatched to a supporting executor, and the worker runs test_connection()` with a proper timeout. Results come back through the Execution API. Design was discussed on dev@ : "[DISCUSS] Move connection testing to workers" (Feb 2026).

Demo

breeze-e2e-rundown-compressed.mp4

Overview

• Dedicated workload type : not piggybacking on ExecuteCallback, so connection tests never compete with correctness-critical callbacks
• Scheduler dispatch + reaper: PENDING tests get dispatched to a supporting executor, capped by max_connection_test_concurrency (default 4). A reaper catches stuck tests after timeout + grace period
• Worker-side timeout : signal.alarm enforcement in LocalExecutor, results reported back via Execution API
• Request Buffer Mechanism: <TODO:add details>
• Queue parameter: optional queue field on the API, wired through to scheduler dispatch.
• Fail-fast: supports_connection_test flag on BaseExecutor, immediate FAILED if no executor supports it

Config

• [core] connection_test_timeout: worker timeout, default 60s
• [core] max_connection_test_concurrency: dispatch budget, default 4
• [scheduler] connection_test_reaper_interval: reaper frequency, default 30s

Not in this PR

• UI changes (will create separate pr for this)

References

• Dev mailing list discussion: DISCUSS Move connection testing to workers

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[jason810496]

Nice! LGTM overall.

[anishgirianish]

@jason810496 Thanks for the thorough review! Addressed your feedback in the latest push:

• Removed result_status column — state is sufficient
• Moved _ImportPathCallbackDef to connection_test.py with a create_callback() factory method

Could you please take another look when you get a chance? Thanks!

[potiuk]

@anishgirianish This PR has a few issues that need to be addressed before it can be reviewed — please see our Pull Request quality criteria.

Issues found:

• ⚠️ Unresolved review comments: This PR has 2 unresolved review threads from maintainers: @jscheffl (MEMBER): 2 unresolved threads. Please review and resolve all inline review comments before requesting another review. You can resolve a conversation by clicking 'Resolve conversation' on each thread after addressing the feedback. See pull request guidelines.

What to do next:

• The comment informs you what you need to do.
• Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
• There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates.
• Maintainers will then proceed with a normal review.

There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the Airflow Slack.