Skip to content

feat: add kiln-benchmarks crate#2

Open
joeycarpinelli wants to merge 1 commit intomainfrom
feat/kiln-benchmarks
Open

feat: add kiln-benchmarks crate#2
joeycarpinelli wants to merge 1 commit intomainfrom
feat/kiln-benchmarks

Conversation

@joeycarpinelli
Copy link
Contributor

Summary

Implements the full benchmark suite described in docs/design/git-kiln-benchmarks.md.

What's included

New crate: crates/kiln-benchmarks

A publish = false workspace member containing 7 Criterion benchmarks that measure Git's object store performance under Kiln's build cache access patterns. All repos are bare, created in tempfile::TempDir, with no remotes configured.

# Benchmark What it measures
1 bench1_cache_lookup Hit/miss ref lookup at N = 100 / 1k / 10k
2 bench2_ingestion Write 4 MB rlib + metadata; compression level 0 vs 6; repo size
3 bench3_materialization Restore cached outputs to FS; odb vs fs timing split
4 bench4_deduplication Object count after two builds, one crate changed
5 bench5_ref_namespace Enumerate refs at 1k / 10k / 100k; lookup hit/miss
6 bench6_fetch_simulation Transfer size estimates + local ingestion timing
7 bench7_gc Repo size before/after git gc, pack files, binary delta exclusion

crates/kiln-benchmarks/README.md — serves as the results document. Contains benchmark descriptions, scale targets, pre-populated results tables (to be filled in after a run), and a findings/recommendations section.

Config changes

  • .config/release-please-config.json — adds kiln-benchmarks with publish: false
  • .config/release-please-manifest.json — adds initial version 0.0.0
  • .github/workflows/CD.yml — publishes only git-kiln (not the bench crate); adds a benchmarks job that runs cargo bench and uploads target/criterion/ as a 90-day artifact

Verification

cargo build --package kiln-benchmarks --benches  # clean, no warnings
cargo test --workspace                            # all tests pass
# Quick smoke test (cache lookup at N=100):
cargo bench --package kiln-benchmarks --bench kiln -- bench1_cache_lookup/hit/100

The smoke test produced ~21 µs mean latency on an M-series Mac — well within the 5 ms target.

Adds a new crates/kiln-benchmarks workspace member implementing the full
benchmark suite described in docs/design/git-kiln-benchmarks.md.

Benchmarks:
  1. Action hash cache lookup (hit/miss at N=100, 1k, 10k refs)
  2. Output tree ingestion (compression level 0 vs 6, repo size at 100/500 crates)
  3. Output tree materialization (N=10, 50, 200 crates, odb vs fs timing)
  4. Deduplication across crate versions (object count, dedup ratio)
  5. Ref namespace scale (enumerate at 1k/10k/100k, lookup hit/miss)
  6. Fetch simulation (transfer size estimates, local ingestion timing)
  7. GC and pack behavior (size before/after, retained refs, delta compression)

All repos are bare, remotes-free, created in tempfile::TempDir.
core.looseCompression=0 is set on repos that store binary artifact blobs.

Also:
- Adds crates/kiln-benchmarks/README.md (results template + analysis guide)
- Adds kiln-benchmarks to release-please config (publish=false)
- Updates CD.yml: publishes only git-kiln, adds a benchmarks job that runs
  cargo bench and uploads results to GitHub Actions artifacts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant