Skip to content

Add unified emission_policy config for OpenLineage provider #66992

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mobuchowski merged 3 commits into from
Jun 29, 2026
Merged

Add unified emission_policy config for OpenLineage provider #66992

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b18c5a0
feat: Add detailed and extensible lineage controls
kacpermuda Jun 2, 2026
0c0c3b8
Small cleanup
kacpermuda Jun 17, 2026
932c25d
Few more improvements and tests
kacpermuda Jun 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 14 additions & 77 deletions providers/openlineage/docs/configurations-ref.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,20 @@
Highlighted configurations
===========================


Configuration precedence
-------------------------

Primary, and recommended method of configuring OpenLineage Airflow Provider is Airflow configuration.
As there are multiple possible ways of configuring OpenLineage, it's important to keep in mind the precedence of different configurations.
OpenLineage Airflow Provider looks for the configuration in the following order:

1. Check ``config_conn_id`` in ``airflow.cfg`` under ``openlineage`` section.
2. Check ``config_path`` in ``airflow.cfg`` under ``openlineage`` section (or AIRFLOW__OPENLINEAGE__CONFIG_PATH environment variable)
3. Check ``transport`` in ``airflow.cfg`` under ``openlineage`` section (or AIRFLOW__OPENLINEAGE__TRANSPORT environment variable)
4. If all the above options are missing, the OpenLineage Python client used underneath looks for configuration in the order described in `this <https://openlineage.io/docs/client/python/configuration>`_ documentation. Please note that **using Airflow configuration is encouraged** and is the only future proof solution.


.. _configuration_transport:openlineage:

Transport setup
Expand Down Expand Up @@ -123,83 +137,6 @@ Example content of config YAML file:
Detailed description, together with example config files, can be found `in Python client documentation <https://openlineage.io/docs/client/python/configuration#transports>`_.


Configuration precedence
^^^^^^^^^^^^^^^^^^^^^^^^^

Primary, and recommended method of configuring OpenLineage Airflow Provider is Airflow configuration.
As there are multiple possible ways of configuring OpenLineage, it's important to keep in mind the precedence of different configurations.
OpenLineage Airflow Provider looks for the configuration in the following order:

1. Check ``config_conn_id`` in ``airflow.cfg`` under ``openlineage`` section.
2. Check ``config_path`` in ``airflow.cfg`` under ``openlineage`` section (or AIRFLOW__OPENLINEAGE__CONFIG_PATH environment variable)
3. Check ``transport`` in ``airflow.cfg`` under ``openlineage`` section (or AIRFLOW__OPENLINEAGE__TRANSPORT environment variable)
4. If all the above options are missing, the OpenLineage Python client used underneath looks for configuration in the order described in `this <https://openlineage.io/docs/client/python/configuration>`_ documentation. Please note that **using Airflow configuration is encouraged** and is the only future proof solution.


.. _configuration_selective_enable:openlineage:

Enabling OpenLineage on Dag/task level
---------------------------------------

One can selectively enable OpenLineage for specific Dags and tasks by using the ``selective_enable`` policy.
To enable this policy, set the ``selective_enable`` option to True in the [openlineage] section of your Airflow configuration file:

.. code-block:: ini

[openlineage]
selective_enable = True

``AIRFLOW__OPENLINEAGE__SELECTIVE_ENABLE`` environment variable is an equivalent.

.. code-block:: ini

AIRFLOW__OPENLINEAGE__SELECTIVE_ENABLE=true


While ``selective_enable`` enables selective control, the ``disabled`` option still has precedence.
If you set ``disabled`` to True in the configuration, OpenLineage will be disabled for all Dags and tasks regardless of the ``selective_enable`` setting.

Once the ``selective_enable`` policy is enabled, you can choose to enable OpenLineage
for individual Dags and tasks using the ``enable_lineage`` and ``disable_lineage`` functions.

1. Enabling Lineage on a Dag:

.. code-block:: python

from airflow.providers.openlineage.utils.selective_enable import disable_lineage, enable_lineage

with enable_lineage(Dag(...)):
# Tasks within this Dag will have lineage tracking enabled
MyOperator(...)

AnotherOperator(...)

2. Enabling Lineage on a Task:

While enabling lineage on a Dag implicitly enables it for all tasks within that Dag, you can still selectively disable it for specific tasks:

.. code-block:: python

from airflow.providers.openlineage.utils.selective_enable import disable_lineage, enable_lineage

with DAG(...) as dag:
t1 = MyOperator(...)
t2 = AnotherOperator(...)

# Enable lineage for the entire Dag
enable_lineage(dag)

# Disable lineage for task t1
disable_lineage(t1)

Enabling lineage on the Dag level automatically enables it for all tasks within that Dag unless explicitly disabled per task.

Enabling lineage on the task level implicitly enables lineage on its Dag.
This is because each emitting task sends a `ParentRunFacet <https://openlineage.io/docs/spec/facets/run-facets/parent_run>`_,
which requires the Dag-level lineage to be enabled in some OpenLineage backend systems.
Disabling Dag-level lineage while enabling task-level lineage might cause errors or inconsistencies.


.. _configuration_custom_facets:openlineage:

Custom Facets
Expand Down
Loading
Loading