Skip to content

[ISSUE-68912] feat(celery): honor json_logs config in Celery worker startup#68916

Open
hanxdatadog wants to merge 2 commits into
apache:mainfrom
hanxdatadog:hanx/issue-68912-celery-worker-json-logs-rebased
Open

[ISSUE-68912] feat(celery): honor json_logs config in Celery worker startup#68916
hanxdatadog wants to merge 2 commits into
apache:mainfrom
hanxdatadog:hanx/issue-68912-celery-worker-json-logs-rebased

Conversation

@hanxdatadog

Copy link
Copy Markdown
Contributor

Closes #68912

Problem

The Celery worker ignores [logging] json_logs / AIRFLOW__LOGGING__JSON_LOGS. It always calls configure_logging(output=sys.stdout.buffer) without passing json_output, so structured JSON logging is never enabled on the worker even when configured globally.

Solution

Add a two-level config lookup before calling configure_logging:

  1. [celery] json_logs — worker-specific override, takes precedence.
  2. [logging] json_logs — global setting, used when the celery key is absent.
  3. False — default when neither key is configured.

This mirrors the existing [celery] CELERY_LOGGING_LEVEL / [logging] LOGGING_LEVEL fallback pattern already present in the worker startup code.

# Global — affects the API server, scheduler, and workers:
[logging]
json_logs = True

# Or override for the Celery worker only, leaving other components unchanged:
[celery]
json_logs = True

Changes

  • providers/celery/src/airflow/providers/celery/cli/celery_command.py: read [celery] json_logs[logging] json_logsFalse and pass json_output to configure_logging.
  • providers/celery/provider.yaml + get_provider_info.py: new json_logs config option under [celery].
  • providers/celery/tests/unit/celery/cli/test_celery_command.py: TestWorkerJsonLogs — 3 tests covering (a) default is False, (b) global [logging] setting is inherited, (c) [celery] override takes precedence.
  • providers/celery/docs/celery_executor.rst: new "Worker logging" section documenting the fallback order and Airflow 3.0/3.2 compatibility note.
  • providers/celery/newsfragments/68912.feature.rst: changelog entry.

Notes

  • Only active on Airflow 3+ (AIRFLOW_V_3_0_PLUS guard is unchanged).
  • conf.getboolean(..., fallback=None) is safe on 3.0/3.1 even when the [celery] json_logs key is absent.
  • version_added: ~ in provider.yaml — version not yet decided; to be set by the release manager.

Add two-level config lookup for JSON logging in the Celery worker:
- Check [celery] json_logs first (celery-specific override)
- Fall back to [logging] json_logs (global setting, added in 3.2.0)
- Fall back to False if neither is set

This mirrors the existing CELERY_LOGGING_LEVEL / LOGGING_LEVEL fallback
pattern in the same file.

Also adds [celery] json_logs to provider.yaml config schema.

Closes apache#68912
@hanxdatadog hanxdatadog changed the title feat(celery): honor json_logs config in Celery worker startup [ISSUE-68912] feat(celery): honor json_logs config in Celery worker startup Jun 23, 2026
@hanxdatadog hanxdatadog marked this pull request as ready for review June 23, 2026 20:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: add json_logs config support for Celery worker stdout

1 participant