Skip to content

Let an auth manager filter dags in SQL instead of materializing every id - #71341

Open
1fanwang wants to merge 1 commit into
apache:mainfrom
1fanwang:stewang/authorized-dag-ids-select
Open

Let an auth manager filter dags in SQL instead of materializing every id#71341
1fanwang wants to merge 1 commit into
apache:mainfrom
1fanwang:stewang/authorized-dag-ids-select

Conversation

@1fanwang

@1fanwang 1fanwang commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Closes: #71309

Why

get_authorized_dag_ids is where every list endpoint starts, and its default implementation reads
the whole dag table before any manager is consulted:

stmt = (
    select(DagModel.dag_id, dag_bundle_team_association_table.c.team_name)
    .join(DagBundleModel, DagModel.bundle_name == DagBundleModel.name)
    .join(dag_bundle_team_association_table, ..., isouter=True)
)
rows = session.execute(stmt).all()

It then groups by team and calls filter_authorized_dag_ids once per team. So a request for 50
dags loads every dag id in the deployment, and the cost scales with the deployment rather than the
page. That is core's own path, not one provider's: FabAuthManager overrides it and still reads
every row, and Keycloak inherits it and then fires a call per dag, which is what
#69041 and
#61686 report as ten and twenty-five second
pages.

The docs point managers at Dag tags and bundles for attribute-based access control, and
get_db_manager lets a manager add tables of its own. Tags, bundles and teams are all rows in
this database. The set[str] return type is what stops any of them being answered as a query.

What changed

BaseAuthManager gains get_authorized_dag_ids_select, returning a select of dag ids or None.
None is the default and keeps today's behaviour, so managers backed by an external policy service
(Keycloak, Amazon Verified Permissions) are unaffected.

A returned select is applied as dag_id IN (subquery). Endpoints that need the ids in memory, the
dependency graph services and the run-state counts endpoint, still get them: PermittedDagFilter
materializes on first read, so nothing is loaded unless something asks.

A select replaces get_authorized_dag_ids outright, including its per-team grouping, so a
multi-team manager scopes the select itself or returns None and keeps the fan-out.

FabAuthManager implements it with the grant query it already knows how to write.

Testing Done

Measured, no FAB. A tag-based manager, which is what the docs recommend. MySQL 8 with 41,606
dags, each carrying an environment tag and a team tag. Page of 50, median of 15 rounds, client
inside the cluster.

Tag the manager authorizes on Matching dags Materialize (today) Subquery
Two team tags 418 5.4 ms 1.4 ms
An environment tag 37,446 316.1 ms 0.8 ms

Measured, FAB. Same database at 41,606 dags, 1,610 roles, 66,529 per-dag edit grants.

Grant Materialize (today) Subquery
Per-dag edit on 228 dags 3.7 ms 2.3 ms
Authorized on all 41,606 dags 351.8 ms 0.6 ms

The expensive row in both is the ordinary one: a tag most dags carry, or a role with a global
can_read, which is Viewer and up.

Red. A select has no truth value, so a filter falling back with or raises. With that
fallback restored:

$ pytest airflow-core/tests/unit/api_fastapi/core_api/test_security.py \
    -k PermittedDagFilterSubquery -q
E   TypeError: Boolean value of this clause is not defined
4 failed, 2 passed, 111 deselected

Green.

$ pytest airflow-core/tests/unit/api_fastapi/core_api/test_security.py \
    -k PermittedDagFilterSubquery -q
8 passed, 111 deselected

$ pytest airflow-core/tests/unit/api_fastapi/auth/managers/test_base_auth_manager.py -q
61 passed

$ pytest airflow-core/tests/unit/api_fastapi/common -q
162 passed, 6 skipped

The eight cover the None default, a select reaching the SQL as a subquery, an empty select
honoured as "nothing is permitted" rather than collapsing to no filter, a subclass filter
inheriting it, a tag-based select needing no FAB, a team-scoped select staying one statement, and the set
materializing once and only when read.

Ruff and mypy are clean over the six modules the change touches.

@1fanwang
1fanwang force-pushed the stewang/authorized-dag-ids-select branch 3 times, most recently from cb4476c to 6c06640 Compare August 9, 2026 08:37
@1fanwang 1fanwang changed the title Let an auth manager push dag authorization into SQL Let an auth manager filter dags in SQL instead of materializing every id Aug 9, 2026
get_authorized_dag_ids returns a set, so every authorized dag id is loaded
into memory before pagination is applied. FabAuthManager keeps its grants in
the metadata database and still does this: a user authorized on all dags gets
select(DagModel.dag_id) materialized on every list request.

get_authorized_dag_ids_select lets a manager return a select instead, which
the permitted-dag filters apply as a subquery. Returning None, the default,
keeps the existing behaviour, and every permitted-* filter inherits it because
they all build the clause with in_().

Signed-off-by: 1fanwang <1fannnw@gmail.com>
@1fanwang
1fanwang force-pushed the stewang/authorized-dag-ids-select branch from 6c06640 to fe5c60f Compare August 9, 2026 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Auth manager cannot push dag authorization into SQL, so list views enumerate every dag

1 participant