Skip to content

Commit e224454

Browse files
committed
ci: optimize caching to include componentize-py git checkouts
The main bottleneck in CI was that componentize-py rebuilds CPython from source (~5 minutes) on every run. This was happening because: 1. componentize-py's build.rs builds CPython in its git checkout directory: ~/.cargo/git/checkouts/componentize-py-<hash>/<commit>/cpython/builddir/ 2. We were only caching ~/.cargo/git/db/ (bare repos), not checkouts/ 3. The separate componentize-py-cpython cache with wildcards was redundant and potentially conflicting with the main cargo-deps cache Changes: - Add ~/.cargo/git/checkouts/ to cargo-deps cache (contains CPython build) - Remove redundant componentize-py-cpython cache - Remove sccache CC/CXX wrappers (caused 104 write errors, not needed) - Improve cache key restore pattern for better hit rates - Bump cache version to v3 to bust old caches Expected improvement: - Cold cache: 30-60 min (unchanged) - Warm cache: 8min -> 2-5min (5min savings from cached CPython build)
1 parent f00b95d commit e224454

1 file changed

Lines changed: 9 additions & 21 deletions

File tree

.github/workflows/python-ci.yml

Lines changed: 9 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -62,44 +62,32 @@ jobs:
6262
run: cargo install --git https://github.com/fastly/Viceroy.git --tag "$VICEROY_TAG" viceroy
6363

6464
# Cache Rust dependencies and build artifacts
65+
# CRITICAL: ~/.cargo/git/checkouts/ contains componentize-py source + built CPython (5+ min build)
6566
- name: Cache Rust dependencies
6667
uses: actions/cache@v4
6768
with:
6869
path: |
6970
~/.cargo/registry/index/
7071
~/.cargo/registry/cache/
7172
~/.cargo/git/db/
73+
~/.cargo/git/checkouts/
7274
target/
73-
key: cargo-deps-${{ runner.os }}-${{ hashFiles('**/Cargo.lock') }}
75+
key: cargo-deps-${{ runner.os }}-${{ hashFiles('**/Cargo.lock') }}-v3
7476
restore-keys: |
77+
cargo-deps-${{ runner.os }}-${{ hashFiles('**/Cargo.lock') }}-
7578
cargo-deps-${{ runner.os }}-
76-
77-
# Cache componentize-py CPython build (this is the slow part!)
78-
# The CPython build is deterministic based on the componentize-py version
79-
# componentize-py will download WASI SDK automatically if not found at /opt/wasi-sdk
80-
- name: Cache componentize-py CPython build
81-
uses: actions/cache@v4
82-
with:
83-
path: |
84-
~/.cargo/git/checkouts/componentize-py-*/*/cpython/builddir/
85-
target/*/build/componentize-py-*/out/
86-
key: componentize-py-cpython-${{ runner.os }}-${{ hashFiles('**/Cargo.lock') }}-v2
87-
restore-keys: |
88-
componentize-py-cpython-${{ runner.os }}-
89-
90-
# Setup sccache for C/C++ compilation caching (speeds up CPython build)
79+
80+
# Setup sccache for Rust compilation caching
9181
- name: Run sccache-cache
9282
uses: mozilla-actions/sccache-action@v0.0.8
9383
- name: Configure sccache
9484
run: |
9585
echo "SCCACHE_GHA_ENABLED=true" >> $GITHUB_ENV
9686
echo "RUSTC_WRAPPER=sccache" >> $GITHUB_ENV
97-
echo "CC=sccache clang" >> $GITHUB_ENV
98-
echo "CXX=sccache clang++" >> $GITHUB_ENV
9987
100-
# Build the Rust native extension; this would be triggered by `uv sync` but
101-
# separated out here as it can take quite awhile depending on what is
102-
# cached.
88+
# Build the Rust native extension (this is the slow part - builds componentize-py + CPython)
89+
# First run (cold cache): 30-60 minutes
90+
# Subsequent runs (warm cache): ~2-5 minutes
10391
- name: Build native extension
10492
run: uv run maturin develop --release
10593

0 commit comments

Comments
 (0)