bench(v0.5.1): real-package pip-install chain spawn numbers#228
Merged
Conversation
Same harness shape as Phase 5 (bench/chain-spawn/RESULTS-v0.5.md) but driven against the actual numpy → pandas → scikit-learn chain that v0.5 was designed for. The Phase 5 bench had to use stdlib-only Python source deltas because pip install hung on CRNG starvation (#218); now that #218 + #225 closed in v0.5.1, this is the apples- to-apples follow-up. Headline numbers (10 iters per head, 512 MiB python:3.12-slim base): head depth p50_ms per-link tax L0 demo-pyt 0 75 — L1 +numpy 1 778 +703 L2 +pandas 2 1229 +451 L3 +sklearn 3 1700 +471 Flat (compacted) 0 78 — Confirms the per-link tax model from Phase 5: ~460 ms per link, tracking SHA-256 of the 512 MiB base at ~1.1 GiB/s. The big new data point is the **Flat-equivalent row at 78 ms p50** — produced by `snapshot-compact py-sklearn → py-sklearn-flat`, which actually sets parent_tag=None and restores via the historical Phase 1 fast path. So `forkd snapshot-compact` really does buy back the chain's per-link tax. 22× faster spawn vs the depth-3 chain head. Headline operational guidance for v0.5.1 users: Build with snapshot-diff chains, ship with snapshot-compact. The chain stores lineage compactly while you iterate; compact collapses the per-link tax to zero once the chain stabilizes. Probe correctness: spawn from py-sklearn, `import numpy, pandas, sklearn` succeeds and a fitted LinearRegression model runs to completion. Vmstate-drift question stays closed for real PyPI packages. Raw log included at bench/chain-spawn/bench-v0.5.1-pip-chain.log for reproducibility. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to Phase 5 (`RESULTS-v0.5.md`) now that v0.5.1 closed #218 + #225. Phase 5 had to use stdlib-only Python source deltas because `pip install` hung; this bench drives the actual `numpy → pandas → scikit-learn` chain v0.5 was designed for.
Headline
10 iters per head, 512 MiB `python:3.12-slim` base. Per-link tax model from Phase 5 confirmed: ~460 ms per link, tracking SHA-256 of the base at ~1.1 GiB/s.
The new data point
`snapshot-compact py-sklearn → py-sklearn-flat` produces a flat snapshot (`parent_tag=None`) that spawns at 78 ms p50 — 22× faster than the depth-3 chain head. So compact really does buy back the chain's per-link tax.
Operational guidance: build with chain, ship with compact. The chain stores lineage compactly while you iterate; compact collapses the per-link tax once the chain stabilizes.
Correctness
Spawn from py-sklearn → `import numpy, pandas, sklearn` → fitted `LinearRegression` runs to completion. Vmstate-drift question stays closed for real PyPI packages.
Closes
n/a — pure benchmark documentation, no code changes.
🤖 Generated with Claude Code