SDK conf does not resolve provider metadata config defaults (has_option returns False)

[nailo2c]

Apache Airflow version

3.1.8

If "Other Airflow 3 version" selected, which one?

No response

What happened?

airflow.sdk.configuration.conf does not resolve config options defined in provider metadata (provider.yaml). For example, conf.has_option("celery", "pool") returns False on the SDK conf, while the core conf correctly returns True.

This was discovered during the #60000 conf import migration. celery_command.py uses conf.has_option("celery", "pool") to decide whether to pass --pool to the Celery worker. After switching to the compat SDK conf, this returns False and the --pool prefork argument is silently dropped.

From reading the code, this may be caused by a circular dependency in the lookup sequence. When has_option() triggers _get_option_from_provider_metadata_config_fallbacks (parser.py:318), the cached property _provider_metadata_config_fallback_default_values (parser.py:342) calls ProvidersManagerTaskRuntime().provider_configs, which in turn calls conf.load_providers_configuration(). load_providers_configuration() uses already_initialized_provider_configs (parser.py:1202) which may return an empty dict if initialization hasn't completed yet.

What you think should happen instead?

sdk_conf.has_option("celery", "pool") should return True, consistent with core_conf.has_option("celery", "pool").

How to reproduce

from airflow.configuration import conf as core_conf
from airflow.sdk.configuration import conf as sdk_conf

print("core:", core_conf.has_option("celery", "pool"))  # True
print("sdk:", sdk_conf.has_option("celery", "pool"))    # False — expected True

celery.pool is defined in providers/celery/provider.yaml:287 with default prefork.

Operating System

Breeze (Debian 12)

Versions of Apache Airflow Providers

Apache Airflow main branch.

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Mishra-coder]

Hi! I’d like to work on this issue. Could you please assign it to me? I’ll make sure to follow the contribution guidelines. Thanks!

[jason810496]

I just tested it out from my end, it seems both Core and SDK conf.has_option didn't work even after the PR merged.
It seems like regression when resolving the review comments ...

I will take a closer look now, assign myself just to track the progress, as it might impact the 3.2 release.

[nailo2c]

Hi @jason810496, Core conf works on my side, what command are you using for the test?

Here is mine: breeze --backend postgres run pytest providers/celery/tests/unit/celery/cli/test_celery_command.py::TestWorkerStart::test_worker_started_with_required_arguments

[jason810496]

Core conf works on my side, what command are you using for the test?

Actually, I found that there's no problem for conf.get for both Core and TaskSDK, only has_option will not respect the provider metadata for both Core and TaskSDK.