[SPARK-58561][INFRA] Support backporting a PR merged into a non-default branch

[uros-b]

What changes were proposed in this pull request?

Make dev/merge_spark_pr.py able to backport a PR that was merged into a non-default branch, and default its cherry-pick prompt to a branch that does not already have the change.

Backport mode is entered when the script finds the commit that merged an already-closed PR. That lookup previously read the merge commit off the PR's closed event:

merge_commits = [e for e in pr_events if e["event"] == "closed" and e["commit_id"] is not None]

GitHub only attributes a commit to the closed event when that commit lands on the default branch, because the Closes #N keyword in the commit message is what closes the PR and the keyword is honored only there. A PR merged into any other branch -- e.g. one opened against a rolling branch-M.x -- is instead closed by this script through the API (close_pr), and that closed event carries no commit. The merge then survives only as a referenced event, so merge_commits came up empty, backport mode never engaged, and the script fell through to the normal merge path -- offering to merge the PR a second time.

This adds two helpers:

• find_merge_commit(pr_num, pr_events) prefers the closed event's commit (unchanged behavior, and GitHub's own authoritative link), and only when that is absent falls back to referenced events. Since a referenced event is raised by any commit merely mentioning the PR, each fallback candidate is confirmed against the Closes #N from line that merge_pr writes into every merge commit.
• default_pick_branch(branch_names, already_picked) returns the highest-ranked release branch that has not already received the change. Backport mode previously defaulted the prompt to branch_names[0], which for a PR merged into branch-M.x is that very branch: accepting the default asked git to cherry-pick a commit onto the branch that already had it, which fails with The previous cherry-pick is now empty and lands the committer in the "Would you like to manually fix-up this merge?" prompt. The normal merge path already excluded target_ref from its defaults, so this consolidates both call sites onto the shared helper.

Why are the changes needed?

A PR opened against branch-4.x and merged there cannot currently be backported to branch-4.3 with the merge script at all -- the script's own API-close erases the trail that its backport mode depends on. Concretely, for #57713 (merged into branch-4.x as 881e5a94a15), re-running the script prints:

Start to merge pull request #57713
Pull request 57713 is not mergeable in its current form.
Continue? (experts only!) (y/N):

Answering y there would create a duplicate squash commit on branch-4.x; the cherry-pick prompt is never reached. The only recourse was a manual git cherry-pick -sx outside the tool, which also skips the merge comment the script would post. With this change the same invocation reaches:

Pull request 57713 has already been merged, assuming you want to backport
Found commit 881e5a94a15...
Enter a branch name [branch-4.3]:

Does this PR introduce any user-facing change?

No. dev/merge_spark_pr.py is a committer tool and is not part of any released artifact.

How was this patch tested?

• Added doctests for both new helpers; the script runs doctest.testmod() on startup. The suite goes from 68 to 76 passing tests, 0 failures.

• Replayed the real GitHub event payloads for three PRs with get_json stubbed to read genuine commit messages from a local clone (no network), asserting the resolved merge commit:

  PR	Base	closed / referenced events	Resolved
  [SPARK-58502][SQL] Make in-limit SQL test deterministic #57713	branch-4.x	0 / 1	881e5a94a15	fixed by this PR
  [SPARK-58502][SQL] Make in-limit SQL test deterministic #57696	master	1 / 2	f0e2b19b82d	unchanged
  [SPARK-58510][SQL] Make variant path functions error consistently on malformed constant paths across execution modes #57682	master	1 / 2	bf57ee78058	unchanged

• Verified the fallback rejects false positives: [SPARK-58502][SQL] Make in-limit SQL test deterministic #57696's referenced list also contains 881e5a94a15 ([SPARK-58502][SQL] Make in-limit SQL test deterministic #57713's commit, which mentions it). With [SPARK-58502][SQL] Make in-limit SQL test deterministic #57696's closed commit stripped to force the fallback, the marker check still resolves to f0e2b19b82d.

• Verified a PR closed without merging yields (None, None), so it does not wrongly enter backport mode.

• Verified the merge-path refactor is behavior-preserving by transcribing the previous remaining_branches logic and diffing its prompt-default sequence against default_pick_branch across five multi-pick scenarios, including the Upstream-First two-branch path and a mid-list target_ref; all sequences match.

• Verified passing already_picked=(target_ref,) in backport mode leaves the Upstream-First policy prompt unchanged by comparing _upstream_first_sibling under the old and new arguments for every (target, pick) pair.

Note: ruff could not be run in the authoring environment (no PyPI access); formatting follows the surrounding conventions in the file and is left to CI's lint job to confirm.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)

[cloud-fan]

2 blocking, 0 non-blocking, 0 nits.
The intended backport flow has two correctness gaps: merge-commit identification can select an unrelated commit, and repeated invocations do not detect branches already backported.

Correctness (2)

• dev/merge_spark_pr.py:463: The referenced-event marker can match copied PR-body text and select the wrong commit for backporting. -- see inline
• dev/merge_spark_pr.py:1758: Repeated backport invocations default to a maintenance branch that already contains the change. -- see inline

Verification

I traced merge identification back to merge_pr: PR-body line starts are preserved in a git commit -m paragraph, so the fallback regex cannot identify the generated trailer by line anchoring alone. I also traced repeated backports: the closed-PR path performs one cherry_pick, exits, and on the next invocation rebuilds already_picked as only (target_ref,); it never checks which other release branches contain merge_hash.

PR description suggestions

• Correct the claim that line anchoring distinguishes the generated trailer from copied PR-body text.
• Document and handle repeated backport invocations: the current implementation does not discover maintenance branches that already contain the merge commit.

[uros-b]

Thank you @cloud-fan! PTAL again

[cloud-fan]

1 addressed, 1 remaining, 2 new. (1 newly introduced, 1 late catch, 0 previously raised.)
3 blocking, 0 non-blocking, 0 nits.
The ordinary non-default-branch backport flow is improved, but merge-footer recognition and exhausted-branch selection still permit wrong or empty cherry-picks.

Remaining from prior review (1)

• The new matcher still scans the entire copied PR body, so a body that quotes a complete merge footer (for example while discussing or reverting a commit) satisfies the Closes plus Authored-by regex. A referenced commit for a different PR can therefore still be selected as this PR's merge. Anchor the match to the final generated footer, or parse the final commit-message paragraphs, and add a regression case containing the complete quoted footer. -- existing thread

Correctness (2)

• dev/merge_spark_pr.py:727: This branch scan reintroduces the same false-positive class that has_merge_footer is meant to avoid: git log --grep accepts the fragment anywhere in the commit message. A body merely quoting Closes #N from makes every containing release branch look already backported, so the script can skip the correct default and report a backport that never happened. Filter candidate commit messages with the validated footer matcher before mapping them to branches. -- see inline
• dev/merge_spark_pr.py:771: When every known release branch is already in already_picked, this fallback selects branch_names[0], which is known to contain the change. Backport mode calls cherry_pick once before asking whether another pick is wanted, so rerunning after all branches have received the PR immediately offers an empty cherry-pick. Stop with an 'all branches already contain this change' message instead of returning an already-picked default. -- see inline

Verification

I traced generated commit messages from merge_pr through both referenced-event selection and release-branch discovery, including PR bodies that contain footer-shaped paragraphs. I also traced backport mode when every known release branch is already present in picked_refs; no tests were run as part of this review.

PR description suggestions

• Correct the claim that a PR body cannot fake the complete footer structure; the body is copied verbatim and may quote an entire merge footer.
• Document and cover the case where every known release branch already contains the change.