Skip to content

Conversation

@kevinjqliu
Copy link
Contributor

@kevinjqliu kevinjqliu commented Jan 20, 2026

Rationale for this change

This PR speeds up CI and reduces amount of resources required
The overall time doesnt change much (12m -> 10m), since make test-integration dominates most of the time.

  • reduces integration tests to run only on Python 3.12 (we dont need to run for each python version anymore now that we're using spark connect)
  • parallelizes integration tests (run s3/gcs/adls integration tests in parallel)
  • add UV dependency caching
  • add fail-fast strategy

Are these changes tested?

Are there any user-facing changes?

@kevinjqliu kevinjqliu changed the title speed up python ci infra: optimize python ci Jan 20, 2026
@kevinjqliu kevinjqliu requested a review from Fokko January 20, 2026 05:11
@kevinjqliu kevinjqliu marked this pull request as ready for review January 20, 2026 05:11
Copy link
Contributor

@Fokko Fokko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! More productive is always great!

@kevinjqliu kevinjqliu merged commit 7dd25a6 into apache:main Jan 20, 2026
11 checks passed
@kevinjqliu kevinjqliu deleted the kevinjqliu/improve-python-ci branch January 20, 2026 15:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants