Skip to content

[python] Preserve manifest order in Data Evolution splits - #9242

Merged
JingsongLi merged 3 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/preserve-de-manifest-order
Aug 16, 2026
Merged

[python] Preserve manifest order in Data Evolution splits#9242
JingsongLi merged 3 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/preserve-de-manifest-order

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Purpose

Native-plan CI exposed different Data Evolution split.files ordering between Python and Rust.

Java preserves the input manifest order within each overlapping row-id group (#6496). This PR aligns PyPaimon with that behavior by:

  • removing the planner-only sequence pre-sort;
  • preserving manifest order through slice/shard filtering.

The reader still applies its own merge ordering, so query results are unchanged.

Tests

  • manifest order within one row-id group;
  • mixed Blob/Parquet order after slice/shard filtering;
  • external files with the same basename.

@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review August 16, 2026 03:35
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as draft August 16, 2026 03:58
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review August 16, 2026 04:12
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as draft August 16, 2026 06:56
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review August 16, 2026 07:32

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit 12635df into apache:master Aug 16, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants