Skip to content

Fix Grid view failing to load large Dags on MySQL - #71342

Open
WonYong-Jang wants to merge 1 commit into
apache:mainfrom
WonYong-Jang:fix-grid-out-of-sort-memory-mysql
Open

Fix Grid view failing to load large Dags on MySQL#71342
WonYong-Jang wants to merge 1 commit into
apache:mainfrom
WonYong-Jang:fix-grid-out-of-sort-memory-mysql

Conversation

@WonYong-Jang

@WonYong-Jang WonYong-Jang commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Grid view showed no tasks for large Dags when the metadata backend is MySQL.
The API server logged:

ERROR 1038: Out of sort memory, consider increasing server sort buffer size

Root cause

The grid endpoint used _get_latest_serdag to load the latest serialized Dag.
That query did select(SerializedDagModel) ... ORDER BY ... LIMIT 1, which loads the whole row - including the big data / data_compressed blob columns.

On MySQL, ORDER BY uses filesort, and filesort copies each candidate row (the blob too) into the sort buffer. The sort buffer has a fixed size (sort_buffer_size, 256 KB by default). When a single serialized Dag is bigger than that, the sort fails with "Out of sort memory", so the whole request fails and the Grid view renders nothing.

PostgreSQL is not affected because it does not copy large values into the sort buffer.

Fix

Use the existing SerializedDagModel.latest_item_select_object helper.
On MySQL it first finds the latest row's id (ORDER BY on id only, no blob), and then loads that single row by primary key. The blob is never put into the sort buffer, so the error is gone.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code (Opus 4.8) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg boring-cyborg Bot added the area:API Airflow's REST/HTTP API label Aug 9, 2026
@WonYong-Jang
WonYong-Jang marked this pull request as ready for review August 9, 2026 07:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant