[SPARK-58562][SQL][DOCS] Document subplan merging in the SQL performance tuning guide - #57765
Open
peter-toth wants to merge 1 commit into
Open
[SPARK-58562][SQL][DOCS] Document subplan merging in the SQL performance tuning guide#57765peter-toth wants to merge 1 commit into
peter-toth wants to merge 1 commit into
Conversation
…nce tuning guide Adds a "Merging Subplans" section to docs/sql-performance-tuning.md describing what MergeSubplans does, which subplans it merges, the 4 public spark.sql.optimizer.mergeSubplans.filterPropagation.* configs with their trade-offs, and how to disable the rule.
dongjoon-hyun
approved these changes
Aug 4, 2026
dongjoon-hyun
left a comment
Member
There was a problem hiding this comment.
+1, LGTM. Thank you for adding the documentation, @peter-toth .
uros-b
approved these changes
Aug 4, 2026
Member
|
Thank you @peter-toth and @dongjoon-hyun! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR adds a "Merging Subplans" section to
docs/sql-performance-tuning.md, between "Join Strategy Hints" and "Adaptive Query Execution". Docs only, no code change.The section covers:
MergeSubplansdoes and which subplans qualify (non-correlated deterministic scalar subqueries and non-grouping aggregates), with a small example that merges by default,EXPLAIN(a subquery whose single output column ismergedValue, shared viaReusedSubquery),WITHclause is skipped,spark.sql.optimizer.mergeSubplans.filterPropagation.*in a config table, and how to turn the rule off viaspark.sql.optimizer.excludedRules.Why are the changes needed?
The optimization and its configs are not described anywhere in the docs today. The 4 configs are public, so they show up in the generated SQL config table in
configuration.md, but nothing tells users what the optimization does, which query shapes it applies to, or what the trade-offs of the 3 non-default configs are. The only other mention indocs/is the migration-guide note about the rule's package move.This matters most for symmetric filter propagation. It is disabled by default because merging two filtered scans widens the scan filter to
OR(f1, f2), which can lose partition or file pruning. But on queries that compute several differently filtered aggregates over the same table it is a big win: q9 and q28 got about 3.5x faster in TPC-DS runs, and together with propagation through joins q88 got about 7x and q90 about 2x faster. Users have no way to discover that today.The section also makes the plan shape discoverable: a merged subplan shows up as a subquery with a
mergedValuecolumn, which is otherwise hard to trace back to its source.Does this PR introduce any user-facing change?
No, documentation only.
How was this patch tested?
No tests, docs only. I verified every statement against the implementation on master: the config keys, defaults and versions against
SQLConf, the merge conditions and the join nullability rules againstPlanMerger, the skipped-CTE and candidate conditions againstMergeSubplans, theSCAN_MERGINGopt-in againstTableCapabilityand its (test-only) implementors, and the rule name used inexcludedRulesagainstRule.ruleNameand the non-excludable rule list. ThemergedValueoutput and theReusedSubqueryshape are cross-checked against the checked-in TPC-DS q9 golden plan, which also confirms that subplans sharing the same filter merge with the default configs.The added table follows the same
spark-configmarkup as the other tables on the page.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)