Skip to content

Adds a new public API endpoint for backfill dag run entries - ` GET /backfills/{backfill_id}/dag_runs#67381

Draft
shivaam wants to merge 1 commit into
apache:mainfrom
shivaam:pr/backfill-dag-runs-api
Draft

Adds a new public API endpoint for backfill dag run entries - ` GET /backfills/{backfill_id}/dag_runs#67381
shivaam wants to merge 1 commit into
apache:mainfrom
shivaam:pr/backfill-dag-runs-api

Conversation

@shivaam

@shivaam shivaam commented May 23, 2026

Copy link
Copy Markdown
Contributor

Adds a new public API endpoint GET /backfills/{backfill_id}/dag_runs that returns
the BackfillDagRun entries for a given backfill with joined DagRun state. Users
can see what happened in a backfill: which dates ran, their states, and which slots
were skipped (with reason).

Previous PR: #64089 explored enriching BackfillResponse directly. Based on
feedback from @uranusjr, this PR takes the cleaner approach of a dedicated
sub-resource endpoint.

Changes

  • BackfillDagRunResponse / BackfillDagRunCollectionResponse Pydantic models
    with AliasPath for joined DagRun fields (matching existing BackfillResponse pattern)
  • list_backfill_dag_runs() route using joinedload(BackfillDagRun.dag_run) — LEFT OUTER JOIN
    includes skipped slots where dag_run_id is NULL
  • Pagination via limit/offset, default ordering by sort_ordinal
  • 404 when backfill doesn't exist
  • 8 unit tests: happy path, skipped slots, 404, pagination, empty backfill, ordering contract

Response shape

{
  "backfill_dag_runs": [
    {
      "id": 1,
      "backfill_id": 1,
      "dag_run_id": 42,
      "logical_date": "2024-01-01T00:00:00Z",
      "partition_key": null,
      "sort_ordinal": 1,
      "exception_reason": null,
      "dag_run_state": "success",
      "dag_run_run_id": "backfill__2024-01-01T00:00:00+00:00"
    },
    {
      "id": 2,
      "backfill_id": 1,
      "dag_run_id": null,
      "logical_date": "2024-01-02T00:00:00Z",
      "partition_key": null,
      "sort_ordinal": 2,
      "exception_reason": "already exists",
      "dag_run_state": null,
      "dag_run_run_id": null
    }
  ],
  "total_entries": 10
}

Follow-up PRs

  • UI: Progress bar on backfills table/banner + backfill detail page with Dag runs table
  • CLI: airflow backfill list and airflow backfill list-runs commands

closes: #46250


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (claude-opus-4-6)

Generated-by: Claude Code (claude-opus-4-6) following the guidelines


Important

🛠️ Maintainer triage note for @shivaam · by @potiuk · 2026-06-17 14:51 UTC

Helpful heads-up from the maintainers — please address before this PR can be reviewed:

  • Static / docs checks failing (CI image checks / Static checks). Run them locally with prek run --all-files (or pre-commit run --all-files) and push the fixes.
  • See the Pull Request quality criteria.

The ball is in your court — you've been assigned to this PR. Fix the above, then mark it Ready for review.

Automated triage — may be imperfect; a maintainer takes the next look.

@shivaam

shivaam commented May 23, 2026

Copy link
Copy Markdown
Contributor Author

@vatsrahul1001 tagging you since you mentioned you were planning to work on it.

select_stmt, total_entries = paginated_select(
statement=select(BackfillDagRun)
.where(BackfillDagRun.backfill_id == backfill_id)
.options(joinedload(BackfillDagRun.dag_run)),

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

joinedload issues a LEFT OUTER JOIN, preserving rows where dag_run_id is NULL (skipped slots).

@potiuk

potiuk commented May 24, 2026

Copy link
Copy Markdown
Member

@shivaam A few things need addressing before review — see our Pull Request quality criteria.

  • Static checks fail: CI image checks / Static checks — lint/type issue.
  • Basic tests fail: Basic tests / Breeze integration tests, Basic tests / Test git clone on Windows. See the Checks tab.

No rush.


Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.


Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting

Adds a new public API endpoint that returns the BackfillDagRun entries
for a given backfill with joined DagRun state. Users can see what
happened in a backfill: which dates ran, their states (queued, running,
success, failed), and which slots were skipped (with reason).

- BackfillDagRunResponse / BackfillDagRunCollectionResponse models
- LEFT OUTER JOIN via joinedload includes skipped slots (null dag_run_id)
- Pagination via limit/offset, default ordering by sort_ordinal
- 404 when backfill doesn't exist
- 8 unit tests covering happy path, skipped slots, 404, pagination,
  empty backfill, and ordering contract

closes: apache#46250
@shivaam shivaam force-pushed the pr/backfill-dag-runs-api branch from a7f9bca to 376f5f5 Compare May 26, 2026 20:42
@eladkal eladkal added this to the Airflow 3.3.0 milestone Jun 7, 2026
@potiuk

potiuk commented Jun 9, 2026

Copy link
Copy Markdown
Member

@shivaam A few things need addressing before review — see our Pull Request quality criteria.

  • Merge conflicts with main. See docs.

No rush.

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

@potiuk potiuk marked this pull request as draft June 18, 2026 21:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add API endpoint that lets you see what happened in the backfill

3 participants