fix(scheduler): defer loading dag_run.conf to avoid memory spike with large run config (#71267) [fj4WqyCCw3C5ShR1RfB7MoBPTpkRrBFYP1uT35g3MvT]

[waterWang]

Problem

When a DAG with dynamic task mapping is triggered with a large run config (512KB–2MB JSON), the scheduler memory spikes by 5–6× during scheduling of the dynamic tasks. Once the dagrun completes, the memory returns to normal.

Reported in #71267.

Root Cause

The scheduler's critical section query at _executable_task_instances_to_queued uses joinedload(TI.dag_run) (line 808), which eagerly loads the full dag_run row — including the conf column — for every task instance in the SQL result set.

When 500+ mapped task instances all share the same dag_run, the joined SQL result set carries the 2MiB conf column 500+ times. This results in approximately 1 GiB of redundant conf data being transferred from the database and materialized in Python memory, causing the memory spike.

Fix

Add .defer(DagRun.conf) to the eager-load chain. The scheduler never reads dag_run.conf in the critical section, so deferring it eliminates the bloat while keeping all other columns and the secondary selectinload of created_dag_version available.

Verification

• SQLAlchemy standalone test confirms defer correctly excludes the conf column from the SELECT statement
• The existing test test_executable_task_instances_no_per_ti_queries still passes (no N+1 regression)
• No query count changes — the defer only affects which columns are loaded, not how many queries are issued

Closes #71267

[Prab-27]

Thanks @waterWang !
would you please fix the title of this PR - you don't need to add any issue number or hash here
and you could remove fix ():