[ISSUE-68912] feat(celery): honor json_logs config in Celery worker startup#68916
Open
hanxdatadog wants to merge 2 commits into
Open
[ISSUE-68912] feat(celery): honor json_logs config in Celery worker startup#68916hanxdatadog wants to merge 2 commits into
hanxdatadog wants to merge 2 commits into
Conversation
Add two-level config lookup for JSON logging in the Celery worker: - Check [celery] json_logs first (celery-specific override) - Fall back to [logging] json_logs (global setting, added in 3.2.0) - Fall back to False if neither is set This mirrors the existing CELERY_LOGGING_LEVEL / LOGGING_LEVEL fallback pattern in the same file. Also adds [celery] json_logs to provider.yaml config schema. Closes apache#68912
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #68912
Problem
The Celery worker ignores
[logging] json_logs/AIRFLOW__LOGGING__JSON_LOGS. It always callsconfigure_logging(output=sys.stdout.buffer)without passingjson_output, so structured JSON logging is never enabled on the worker even when configured globally.Solution
Add a two-level config lookup before calling
configure_logging:[celery] json_logs— worker-specific override, takes precedence.[logging] json_logs— global setting, used when the celery key is absent.False— default when neither key is configured.This mirrors the existing
[celery] CELERY_LOGGING_LEVEL/[logging] LOGGING_LEVELfallback pattern already present in the worker startup code.Changes
providers/celery/src/airflow/providers/celery/cli/celery_command.py: read[celery] json_logs→[logging] json_logs→Falseand passjson_outputtoconfigure_logging.providers/celery/provider.yaml+get_provider_info.py: newjson_logsconfig option under[celery].providers/celery/tests/unit/celery/cli/test_celery_command.py:TestWorkerJsonLogs— 3 tests covering (a) default is False, (b) global[logging]setting is inherited, (c)[celery]override takes precedence.providers/celery/docs/celery_executor.rst: new "Worker logging" section documenting the fallback order and Airflow 3.0/3.2 compatibility note.providers/celery/newsfragments/68912.feature.rst: changelog entry.Notes
AIRFLOW_V_3_0_PLUSguard is unchanged).conf.getboolean(..., fallback=None)is safe on 3.0/3.1 even when the[celery] json_logskey is absent.version_added: ~inprovider.yaml— version not yet decided; to be set by the release manager.