Skip to content

fix(python): Release GIL during catalog I/O - #716

Merged
JingsongLi merged 1 commit into
apache:mainfrom
XiaoHongbo-Hope:codex/release-gil-for-catalog-io
Aug 15, 2026
Merged

fix(python): Release GIL during catalog I/O#716
JingsongLi merged 1 commit into
apache:mainfrom
XiaoHongbo-Hope:codex/release-gil-for-catalog-io

Conversation

@XiaoHongbo-Hope

Copy link
Copy Markdown
Contributor

Purpose

PaimonCatalog performed blocking catalog and REST I/O while holding the Python GIL. This stalls other Python threads and makes an in-process Python REST server time out after 30 seconds.

Changes

  • Release the GIL while creating a catalog.
  • Release the GIL while listing databases/tables and loading a table.
  • Add an in-process REST server regression test covering all four calls.

Verification

  • test_catalog_gil.py: 1 passed in 0.56s.
  • PyPaimon mock REST native planning: 1 split in 0.041s; the previous binding timed out after 30s.
  • cargo check -p pypaimon_rust
  • cargo fmt --all --check
  • git diff --check

@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review August 15, 2026 06:33

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit c7a71f4 into apache:main Aug 15, 2026
13 checks passed
jerry-024 added a commit to jerry-024/paimon-rust that referenced this pull request Aug 17, 2026
* main:
  perf(vindex): size native batches by active indexes (apache#709)
  fix(data-evolution): require row tracking and row IDs (apache#718)
  feat(go): add table write bindings (apache#658)
  fix(scan): preserve Data Evolution file order in row-id groups (apache#717)
  fix(file_index): align file index format with Java V1 (apache#719)
  fix: configure OpenDAL writer chunk size (apache#713)
  fix(python): release GIL during catalog I/O (apache#716)
  feat(write): add fixed-bucket write primitives for postpone tables (apache#659)
  feat: rust examples for creating and querying Paimon tables (apache#648)
  fix(table): reject row ranges for format tables at read construction (apache#700)
  perf(arrow): prune IN predicates with row-group stats (apache#705)
  feat(io): support in-memory local cache (apache#710)
  fix(dlf): refresh expiring credentials (apache#714)
  fix(datafusion): normalize index_type in global index procedures (apache#715)
  fix(auth): fail closed on query-auth tables outside the read boundary (apache#691)

# Conflicts:
#	crates/paimon/src/table/vector_search_builder.rs
#	crates/paimon/src/vindex/reader.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants