Add partition clear support to REST API matching the CLI#68702
Merged
Conversation
Lee-W
force-pushed
the
partition-clear-rest-api
branch
2 times, most recently
from
d5b88c4 to
a49ae18
Compare
Lee-W
changed the title
Lee-W
requested review from
bbovenzi,
bugraoz93,
choo121600,
ephraimbuddy,
guan404ming,
jason810496,
pierrejeambrun,
rawwar,
ryanahamilton,
shubhamraj-git and
vatsrahul1001
as code owners
Lee-W
force-pushed
the
partition-clear-rest-api
branch
3 times, most recently
from
aa09898 to
292ce16
Compare
Member
There was a problem hiding this comment.
A few suggestions / fix.
Also the date-window semantics here don't actually match airflow partitions clear, which the PR aims to mirror. The CLI compares raw values (partition_date >= start / <= end, partition_command.py:100-103), while this resolves calendar-day bounds (>= resolve_day_bound(start) / < resolve_day_bound(end+1day)). For the same inputs the two clear different runs at the end boundary (and across timezones). Your calendar-day version actually matches the documented "inclusive, calendar-day granular" intent better than the CLI's <= end does — so I think the right fix is to make the CLI use the same resolve_day_bound logic (or share a helper), rather than leave the two interfaces inconsistent. Worth confirming the intended boundary behavior either way.
Lee-W
force-pushed
the
partition-clear-rest-api
branch
from
93ad895 to
0f3a84b
Compare
Lee-W
force-pushed
the
partition-clear-rest-api
branch
2 times, most recently
from
17da6bf to
104ee6d
Compare
Lee-W
changed the title
Lee-W
force-pushed
the
partition-clear-rest-api
branch
from
5a0e85e to
4ba184e
Compare
uranusjr
reviewed
Jun 23, 2026
uranusjr
reviewed
Jun 23, 2026
Lee-W
force-pushed
the
partition-clear-rest-api
branch
from
c34c93a to
d9fe531
Compare
Lee-W
force-pushed
the
partition-clear-rest-api
branch
from
723456d to
bcca769
Compare
clearDagRuns now accepts partition_key / partition_date window selectors
as an alternative to an explicit run list. Add POST /dags/{dag_id}/clearPartitions
to reset partition_key/partition_date on matching runs, with optional
task-instance clear — REST parity with `airflow dags clear` / `airflow partitions clear`.
Extract the shared partition_key / partition_date window fields and their date-order check into a PartitionSelectorMixin reused by BulkDAGRunClearBody and ClearPartitionsBody, and replace the repeated partition-selector presence checks with a has_partition_selectors property. No behavior change.
Add a dag_id filter to the task-instance lookups in both the REST clear_partition_fields service and the airflow partitions clear CLI so a run_id shared across dags no longer clears another dag's task instances, and collapse the per-run dry-run task-instance lookups into a single batched count query.
Extract the resolve_day_bound partition_date window resolution duplicated across the REST clear_dag_runs route, the clear_partition_fields service, and the airflow partitions clear CLI into a single DagRun.apply_partition_date_window helper so the three cannot drift.
Extract the partition column-reset, task-instance batching, and dry-run counting into a single DagRun.clear_partition_runs helper reused by the clearPartitions REST endpoint and the airflow partitions clear CLI, replacing the two parallel implementations. The CLI keeps its per-run output through an optional callback. No behavior change.
`dags clear` and `partitions clear` passed user-supplied datetimes through `resolve_day_bound(.date())`, which stripped the time component and expanded any sub-day bound to local midnight. On an hourly partitioned Dag, `--partition-date-start 08:00 --partition-date-end 08:00` cleared all 24 partitions instead of just the 08:00 one. Adds `localize_partition_datetime` to the `Timetable` protocol (base: UTC pass-through; CronMixin: wall-clock re-interpreted in the timetable's local timezone, same logic as apache#68718). Removes the now-redundant private `_localize_wall_clock_to_timetable_timezone` from `CronPartitionTimetable`. Updates `apply_partition_date_window` to use the new method with an inclusive `<=` end bound instead of the old half-open `< next_midnight` form.
BulkDAGRunClearBody and ClearPartitionsBody duplicated the same "exactly one selection mode" rule, including the partition-window definition and the selector-enumeration error message, which would drift independently. Move the shared check onto PartitionSelectorMixin so the partition-selector semantics live in one place.
The "sub-day precision is preserved" phrasing framed the help against a since-fixed truncation bug, which is meaningless to a reader seeing the text fresh. The timezone re-interpretation note plus the date-only -> midnight rule already convey that the time of day is honoured.
Lee-W
force-pushed
the
partition-clear-rest-api
branch
from
bcca769 to
b482bd0
Compare
A loop variable shadowed the imported `task` decorator (ruff F402) and a British-spelled word in a new docstring tripped the en_US docs spell-check.
Contributor
Backport successfully created: v3-3-testNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
|
github-actions Bot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Jun 23, 2026
…pache#68702) * API: Add partition clear support to REST API to match the CLI clearDagRuns now accepts partition_key / partition_date window selectors as an alternative to an explicit run list. Add POST /dags/{dag_id}/clearPartitions to reset partition_key/partition_date on matching runs, with optional task-instance clear — REST parity with `airflow dags clear` / `airflow partitions clear`. * API: Deduplicate partition selector fields across clear request bodies Extract the shared partition_key / partition_date window fields and their date-order check into a PartitionSelectorMixin reused by BulkDAGRunClearBody and ClearPartitionsBody, and replace the repeated partition-selector presence checks with a has_partition_selectors property. No behavior change. * Scope partition-clear task instance queries to the target dag Add a dag_id filter to the task-instance lookups in both the REST clear_partition_fields service and the airflow partitions clear CLI so a run_id shared across dags no longer clears another dag's task instances, and collapse the per-run dry-run task-instance lookups into a single batched count query. * Share the partition date-window filter across clear paths Extract the resolve_day_bound partition_date window resolution duplicated across the REST clear_dag_runs route, the clear_partition_fields service, and the airflow partitions clear CLI into a single DagRun.apply_partition_date_window helper so the three cannot drift. * Share the partition-clear core between the REST API and the CLI Extract the partition column-reset, task-instance batching, and dry-run counting into a single DagRun.clear_partition_runs helper reused by the clearPartitions REST endpoint and the airflow partitions clear CLI, replacing the two parallel implementations. The CLI keeps its per-run output through an optional callback. No behavior change. * Fix partition clear commands widening sub-day windows to the whole day `dags clear` and `partitions clear` passed user-supplied datetimes through `resolve_day_bound(.date())`, which stripped the time component and expanded any sub-day bound to local midnight. On an hourly partitioned Dag, `--partition-date-start 08:00 --partition-date-end 08:00` cleared all 24 partitions instead of just the 08:00 one. Adds `localize_partition_datetime` to the `Timetable` protocol (base: UTC pass-through; CronMixin: wall-clock re-interpreted in the timetable's local timezone, same logic as apache#68718). Removes the now-redundant private `_localize_wall_clock_to_timetable_timezone` from `CronPartitionTimetable`. Updates `apply_partition_date_window` to use the new method with an inclusive `<=` end bound instead of the old half-open `< next_midnight` form. * Update REST datamodel descriptions to reflect sub-day precision * Share partition selection-mode validation across clear request bodies BulkDAGRunClearBody and ClearPartitionsBody duplicated the same "exactly one selection mode" rule, including the partition-window definition and the selector-enumeration error message, which would drift independently. Move the shared check onto PartitionSelectorMixin so the partition-selector semantics live in one place. * Drop sub-day-precision wording from partition clear CLI help The "sub-day precision is preserved" phrasing framed the help against a since-fixed truncation bug, which is meaningless to a reader seeing the text fresh. The timezone re-interpretation note plus the date-only -> midnight rule already convey that the time of day is honoured. * Remove unused resolve_day_bound * Refactor tests * regen docs * Fix test failure * Fix ruff F402 and docs spelling failing CI on partition-clear branch A loop variable shadowed the imported `task` decorator (ruff F402) and a British-spelled word in a new docstring tripped the en_US docs spell-check. --------- (cherry picked from commit a0805a8) Co-authored-by: Wei Lee <weilee.rx@gmail.com> Co-authored-by: Rahul Vats <rah.sharma11@gmail.com>
aws-airflow-bot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Jun 23, 2026
…pache#68702) * API: Add partition clear support to REST API to match the CLI clearDagRuns now accepts partition_key / partition_date window selectors as an alternative to an explicit run list. Add POST /dags/{dag_id}/clearPartitions to reset partition_key/partition_date on matching runs, with optional task-instance clear — REST parity with `airflow dags clear` / `airflow partitions clear`. * API: Deduplicate partition selector fields across clear request bodies Extract the shared partition_key / partition_date window fields and their date-order check into a PartitionSelectorMixin reused by BulkDAGRunClearBody and ClearPartitionsBody, and replace the repeated partition-selector presence checks with a has_partition_selectors property. No behavior change. * Scope partition-clear task instance queries to the target dag Add a dag_id filter to the task-instance lookups in both the REST clear_partition_fields service and the airflow partitions clear CLI so a run_id shared across dags no longer clears another dag's task instances, and collapse the per-run dry-run task-instance lookups into a single batched count query. * Share the partition date-window filter across clear paths Extract the resolve_day_bound partition_date window resolution duplicated across the REST clear_dag_runs route, the clear_partition_fields service, and the airflow partitions clear CLI into a single DagRun.apply_partition_date_window helper so the three cannot drift. * Share the partition-clear core between the REST API and the CLI Extract the partition column-reset, task-instance batching, and dry-run counting into a single DagRun.clear_partition_runs helper reused by the clearPartitions REST endpoint and the airflow partitions clear CLI, replacing the two parallel implementations. The CLI keeps its per-run output through an optional callback. No behavior change. * Fix partition clear commands widening sub-day windows to the whole day `dags clear` and `partitions clear` passed user-supplied datetimes through `resolve_day_bound(.date())`, which stripped the time component and expanded any sub-day bound to local midnight. On an hourly partitioned Dag, `--partition-date-start 08:00 --partition-date-end 08:00` cleared all 24 partitions instead of just the 08:00 one. Adds `localize_partition_datetime` to the `Timetable` protocol (base: UTC pass-through; CronMixin: wall-clock re-interpreted in the timetable's local timezone, same logic as apache#68718). Removes the now-redundant private `_localize_wall_clock_to_timetable_timezone` from `CronPartitionTimetable`. Updates `apply_partition_date_window` to use the new method with an inclusive `<=` end bound instead of the old half-open `< next_midnight` form. * Update REST datamodel descriptions to reflect sub-day precision * Share partition selection-mode validation across clear request bodies BulkDAGRunClearBody and ClearPartitionsBody duplicated the same "exactly one selection mode" rule, including the partition-window definition and the selector-enumeration error message, which would drift independently. Move the shared check onto PartitionSelectorMixin so the partition-selector semantics live in one place. * Drop sub-day-precision wording from partition clear CLI help The "sub-day precision is preserved" phrasing framed the help against a since-fixed truncation bug, which is meaningless to a reader seeing the text fresh. The timezone re-interpretation note plus the date-only -> midnight rule already convey that the time of day is honoured. * Remove unused resolve_day_bound * Refactor tests * regen docs * Fix test failure * Fix ruff F402 and docs spelling failing CI on partition-clear branch A loop variable shadowed the imported `task` decorator (ruff F402) and a British-spelled word in a new docstring tripped the en_US docs spell-check. --------- (cherry picked from commit a0805a8) Co-authored-by: Wei Lee <weilee.rx@gmail.com> Co-authored-by: Rahul Vats <rah.sharma11@gmail.com>
cetingokhan
pushed a commit
to cetingokhan/airflow
that referenced
this pull request
Jun 24, 2026
* API: Add partition clear support to REST API to match the CLI
clearDagRuns now accepts partition_key / partition_date window selectors
as an alternative to an explicit run list. Add POST /dags/{dag_id}/clearPartitions
to reset partition_key/partition_date on matching runs, with optional
task-instance clear — REST parity with `airflow dags clear` / `airflow partitions clear`.
* API: Deduplicate partition selector fields across clear request bodies
Extract the shared partition_key / partition_date window fields and their date-order check into a PartitionSelectorMixin reused by BulkDAGRunClearBody and ClearPartitionsBody, and replace the repeated partition-selector presence checks with a has_partition_selectors property. No behavior change.
* Scope partition-clear task instance queries to the target dag
Add a dag_id filter to the task-instance lookups in both the REST clear_partition_fields service and the airflow partitions clear CLI so a run_id shared across dags no longer clears another dag's task instances, and collapse the per-run dry-run task-instance lookups into a single batched count query.
* Share the partition date-window filter across clear paths
Extract the resolve_day_bound partition_date window resolution duplicated across the REST clear_dag_runs route, the clear_partition_fields service, and the airflow partitions clear CLI into a single DagRun.apply_partition_date_window helper so the three cannot drift.
* Share the partition-clear core between the REST API and the CLI
Extract the partition column-reset, task-instance batching, and dry-run counting into a single DagRun.clear_partition_runs helper reused by the clearPartitions REST endpoint and the airflow partitions clear CLI, replacing the two parallel implementations. The CLI keeps its per-run output through an optional callback. No behavior change.
* Fix partition clear commands widening sub-day windows to the whole day
`dags clear` and `partitions clear` passed user-supplied datetimes
through `resolve_day_bound(.date())`, which stripped the time component
and expanded any sub-day bound to local midnight. On an hourly
partitioned Dag, `--partition-date-start 08:00 --partition-date-end
08:00` cleared all 24 partitions instead of just the 08:00 one.
Adds `localize_partition_datetime` to the `Timetable` protocol (base:
UTC pass-through; CronMixin: wall-clock re-interpreted in the
timetable's local timezone, same logic as apache#68718). Removes the
now-redundant private `_localize_wall_clock_to_timetable_timezone` from
`CronPartitionTimetable`. Updates `apply_partition_date_window` to use
the new method with an inclusive `<=` end bound instead of the old
half-open `< next_midnight` form.
* Update REST datamodel descriptions to reflect sub-day precision
* Share partition selection-mode validation across clear request bodies
BulkDAGRunClearBody and ClearPartitionsBody duplicated the same
"exactly one selection mode" rule, including the partition-window
definition and the selector-enumeration error message, which would
drift independently. Move the shared check onto PartitionSelectorMixin
so the partition-selector semantics live in one place.
* Drop sub-day-precision wording from partition clear CLI help
The "sub-day precision is preserved" phrasing framed the help against a
since-fixed truncation bug, which is meaningless to a reader seeing the
text fresh. The timezone re-interpretation note plus the date-only ->
midnight rule already convey that the time of day is honoured.
* Remove unused resolve_day_bound
* Refactor tests
* regen docs
* Fix test failure
* Fix ruff F402 and docs spelling failing CI on partition-clear branch
A loop variable shadowed the imported `task` decorator (ruff F402) and a
British-spelled word in a new docstring tripped the en_US docs spell-check.
---------
Co-authored-by: Rahul Vats <rah.sharma11@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the
airflow dags clear/airflow partitions clearpartition-clear capability to the REST API, consolidates the now-shared core between CLI and REST so they can't drift, and fixes a sub-day window bug found along the way.Why
run_idshared across Dags let a partition clear touch another Dag's task instances.dags clear/partitions clearwidened any sub-day partition bound to the whole local day — on an hourly Dag,--partition-date-start 08:00 --partition-date-end 08:00cleared all 24 partitions instead of one.What
clearDagRunsacceptspartition_key/partition_datewindow selectors as an alternative to an explicit run list.POST /dags/{dag_id}/clearPartitionsresetspartition_key/partition_dateon matching runs, with optional task-instance clear.DagRun.clear_partition_runs(column reset + TI batching + dry-run count) andDagRun.apply_partition_date_window(window filter) now back both REST and CLI.PartitionSelectorMixindedups the partition selector fields + date-order check acrossBulkDAGRunClearBodyandClearPartitionsBody.dag_id.PUTauthorization request, so it is gated by the same access check as an explicit run list.Timetable.localize_partition_datetime(base: UTC pass-through;CronMixin: wall-clock in the timetable's timezone) replaces the day-bound widening; window end is now an inclusive<=instead of half-open< next_midnight. It also subsumes the partition backfill iteration's private localization helper, so backfill and clear now share one wall-clock conversion.Was generative AI tooling used to co-author this PR?
Generated-by: [Claude] following the guidelines
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.