Reapply "Add ExecutionPlan::apply_expressions() (apache#20337)" (apache#22437) - #24018
Open
jayshrivastava wants to merge 3 commits into
Open
Reapply "Add ExecutionPlan::apply_expressions() (apache#20337)" (apache#22437)#24018jayshrivastava wants to merge 3 commits into
jayshrivastava wants to merge 3 commits into
Conversation
github-actions
Bot
added
documentation
jayshrivastava
changed the title
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #24018 +/- ##
==========================================
- Coverage 80.84% 80.76% -0.08%
==========================================
Files 1096 1099 +3
Lines 373936 375205 +1269
Branches 373936 375205 +1269
==========================================
+ Hits 302313 303041 +728
- Misses 53584 54084 +500
- Partials 18039 18080 +41 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
jayshrivastava
force-pushed
the
js/dynamic-filters-method
branch
from
July 30, 2026 22:12
8eff543 to
b8794d2
Compare
jayshrivastava
force-pushed
the
js/dynamic-filters-method
branch
from
July 30, 2026 23:26
b8794d2 to
5514194
Compare
jayshrivastava
commented
Jul 30, 2026
| /// `apply_expressions` + `downcast_ref::<DynamicFilterPhysicalExpr>` and | ||
| /// counting nodes. Neither API is observable from SQL. | ||
| #[tokio::test] | ||
| async fn test_discover_dynamic_filters_via_expressions_api() { |
Contributor
Author
There was a problem hiding this comment.
This test explicitly covers that dynamic filters are discoverable post-pushdown. This is what we want in datafusion-distributed and I assume other projects want this as well
| aggregate.with_dynamic_filter_expr(dynamic_filter)? | ||
| } else { | ||
| let mut aggregate = aggregate; | ||
| aggregate.dynamic_filter = None; |
Contributor
Author
There was a problem hiding this comment.
Semi-related. I think this was a bug. We kept the default dynamic filter created by try_new around even there's no pushed down filter. Now, we explicitly remove it. The test I added in datafusion/proto/tests/cases/roundtrip_physical_plan.rs covers this.
jayshrivastava
commented
Jul 30, 2026
| .flatten() | ||
| .map(|sort_expr| &sort_expr.expr), | ||
| f, | ||
| ) |
Contributor
Author
There was a problem hiding this comment.
Question for reviewers - this pattern repeats a lot. Is it worth adding a helper?
apply_expression_roots(
properties
.output_ordering()
.into_iter()
.flatten()
.map(|sort_expr| &sort_expr.expr),
f
)The projection pattern also repeats a bit
crate::apply_expression_roots(
self.projector
.projection()
.as_ref()
.iter()
.map(|proj_expr| &proj_expr.expr),
f,
)
Contributor
Author
|
I see the code coverage is isn't very high. I don't see any good candidates to call |
jayshrivastava
marked this pull request as ready for review
July 31, 2026 00:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
ExecutionPlannodes discoverable #23814Rationale for this change
See the issue above.
What changes are included in this PR?
There's 3 commits in this PR:
Firstly, commit 1 re-applies the changes in #20337 (reverted in #22437).
Some of the reasons for why the original PR was reverted include
(a)
apply_expressionsis too complicated to implement and there's no concrete need to justify this complexity(b) there was no usage of
apply_expressionsinside this repoTo address (a), justification for this method is provided in #23814. Additionally, commit 2 in this PR updates the
apply_expressionsAPI to be more ergonomic:apply_expressionsnow yields&Arc<dyn PhysicalExpr>instead of&dyn PhysicalExpr. It should be easier for callers to use the new API without worrying about the lifetime restriction on a non Arc referenceapply_expression_rootswhich makes abstracts away theTreeNodeRecursioncomplexity from implementors. Now, it's very trivial to implement ex.To address (b), commit 3 adds a usage of
apply_expressionsinphysical-plan/src/aggregates/mod.rs. Previously, there was a hack that checked if a filter was pushed down usingArc::strong_count(dyn_filter) > 1. Now it usesapply_expressions.Are these changes tested?
Yes.
Are there any user-facing changes?
There's a new mandatory method
ExecutionPlan::apply_expressions(). See the upgrading guide and documentation for details.