Skip to content

[SPARK-58562][SQL][DOCS] Document subplan merging in the SQL performance tuning guide - #57765

Open
peter-toth wants to merge 1 commit into
apache:masterfrom
peter-toth:SPARK-58562-document-plan-merging
Open

[SPARK-58562][SQL][DOCS] Document subplan merging in the SQL performance tuning guide#57765
peter-toth wants to merge 1 commit into
apache:masterfrom
peter-toth:SPARK-58562-document-plan-merging

Conversation

@peter-toth

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This PR adds a "Merging Subplans" section to docs/sql-performance-tuning.md, between "Join Strategy Hints" and "Adaptive Query Execution". Docs only, no code change.

The section covers:

  • what MergeSubplans does and which subplans qualify (non-correlated deterministic scalar subqueries and non-grouping aggregates), with a small example that merges by default,
  • how a merged subplan looks in EXPLAIN (a subquery whose single output column is mergedValue, shared via ReusedSubquery),
  • when two subplans are merged, and that a query still carrying a non-inlined WITH clause is skipped,
  • why the one-sided filter case is always beneficial while the symmetric one is not, with the TPC-DS numbers measured in SPARK-40193 and SPARK-56677 and guidance on when to consider enabling it,
  • the 4 configs under spark.sql.optimizer.mergeSubplans.filterPropagation.* in a config table, and how to turn the rule off via spark.sql.optimizer.excludedRules.

Why are the changes needed?

The optimization and its configs are not described anywhere in the docs today. The 4 configs are public, so they show up in the generated SQL config table in configuration.md, but nothing tells users what the optimization does, which query shapes it applies to, or what the trade-offs of the 3 non-default configs are. The only other mention in docs/ is the migration-guide note about the rule's package move.

This matters most for symmetric filter propagation. It is disabled by default because merging two filtered scans widens the scan filter to OR(f1, f2), which can lose partition or file pruning. But on queries that compute several differently filtered aggregates over the same table it is a big win: q9 and q28 got about 3.5x faster in TPC-DS runs, and together with propagation through joins q88 got about 7x and q90 about 2x faster. Users have no way to discover that today.

The section also makes the plan shape discoverable: a merged subplan shows up as a subquery with a mergedValue column, which is otherwise hard to trace back to its source.

Does this PR introduce any user-facing change?

No, documentation only.

How was this patch tested?

No tests, docs only. I verified every statement against the implementation on master: the config keys, defaults and versions against SQLConf, the merge conditions and the join nullability rules against PlanMerger, the skipped-CTE and candidate conditions against MergeSubplans, the SCAN_MERGING opt-in against TableCapability and its (test-only) implementors, and the rule name used in excludedRules against Rule.ruleName and the non-excludable rule list. The mergedValue output and the ReusedSubquery shape are cross-checked against the checked-in TPC-DS q9 golden plan, which also confirms that subplans sharing the same filter merge with the default configs.

The added table follows the same spark-config markup as the other tables on the page.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)

…nce tuning guide

Adds a "Merging Subplans" section to docs/sql-performance-tuning.md describing
what MergeSubplans does, which subplans it merges, the 4 public
spark.sql.optimizer.mergeSubplans.filterPropagation.* configs with their
trade-offs, and how to disable the rule.

@dongjoon-hyun dongjoon-hyun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Thank you for adding the documentation, @peter-toth .

@uros-b

uros-b commented Aug 4, 2026

Copy link
Copy Markdown
Member

Thank you @peter-toth and @dongjoon-hyun!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants