feat(scripts): fleet-hash — deterministic build-input hashes over the bake graph - #298
Merged
Conversation
… bake graph (#297) The hash primitive for carried-forward releases (delivery/RULES.md rules 11-14): hash(target) = sha256(context tree hash + sorted recursive base hashes), read off the bake graph that principle 15.d keeps aligned with the Dockerfiles. Three providers behind one entry point: fleet-hash.sh # all 153 static bake targets fleet-hash.sh combo <bench> <agent> # eval + eval-standalone rows fleet-hash.sh per-task <bench> <id> # per-task image row Output decomposes into hash / context-hash / bases-hash / externals so a consumer can tell WHY a target moved (context edit vs base cascade vs upstream drift). External FROM refs are emitted unresolved — digest resolution needs the network and happens at release time (rule 11); the script itself is offline and a pure function of the committed tree at REF. Design notes proven by tests/static/input_hash.rs on a synthetic git fixture: deterministic across runs, a leaf edit moves only that leaf's context component, a base edit cascades through dependents' bases component, REF pins the whole computation to a commit. The real-repo test asserts one row per per-artifact bake file and exercises all three providers. Combos hash from the combination Dockerfiles + parent target hashes, not the over-broad containers/core context. Dir-name lookups are context-based, so dotted model dirs (gpt-5.4) resolve despite their underscored target names. SQL FROM clauses on RUN continuation lines are not image refs. Portable to bash 3.2 (macOS); one awk parse + one batched git rev-parse keeps a full-fleet run ~5s. Part of #292's migration path, rung 2 of 5. Resolves #297. Signed-off-by: Elron Bandel <office.bandel@gmail.com> Signed-off-by: Elron Bandel <elron.bandel@ibm.com>
…ombo-list drift safety Signed-off-by: Elron Bandel <elron.bandel@ibm.com>
hash(target) = sha256 of the sorted tree hashes of the target's context and every transitive base context. Sensitivity-equivalent to the Merkle chain (wiring changes edit bake files, which live inside a hashed context) but the whole computation is three small awk programs, one batched git rev-parse, and one sha256sum invocation: ~12 forks instead of ~460, 24s -> 5s -> 0.3s, and the bash 3.2 eval-map workaround is gone entirely. Combos become a flat union of parent closures + the combination file blobs; standalone folds the eval set in directly. Signed-off-by: Elron Bandel <elron.bandel@ibm.com>
The script reads the bake files directly (its static tests must run without the docker CLI), so pin that reading to bake's own evaluation: a build-lane test compares `fleet-hash.sh graph` (new subcommand, target|context|deps) against one `docker buildx bake --print` over the full -f set, target for target. The awk parse can never silently drift from the canonical consumer of the graph data (RULES.md principle 15). Verified locally: all 153 targets match. Signed-off-by: Elron Bandel <elron.bandel@ibm.com>
…s, ulimit Four adversarial audits on the PR surfaced real correctness bugs; all reproduced, fixed, and pinned by tests: - Combo closure was blind to containers/core/runner/ (and entrypoint/), which the combination Dockerfiles COPY from: a framework-launcher edit changed every combo image and moved zero hashes — the exact false-fresh rule 14 forbids. Reproduced on real commit b072ff1. Both trees now join the combo context inputs. - The graph and Dockerfiles were read from the worktree while trees came from REF, so REF=<old> emitted a hybrid and uncommitted edits leaked in. containers/ is now materialized from REF via git archive: the output is a pure function of the commit (and uncommitted state is invisible rather than fatal). - 15 phantom externals: python 'from X import Y' in heredoc RUN bodies parsed as FROM instructions. Only an unindented uppercase FROM outside a backslash continuation counts now; ARG defaults are expanded so externals are digest-resolvable (${…} left = per-build by design). - The closure splitter held 288 files open — unrunnable at stock macOS ulimit 256. Files close on change (input is sorted). - Fail-loud hardening: second target block in one bake file, missing context line, comment-quoted target: refs, unparsable sums lines, empty/whitespace per-task ids, graph missing from the usage error. - Combo parents derived from combination.docker-bake.hcl's *_IMAGE defaults instead of hardcoded names. Tests upgraded to kill every surviving mutant the audit found: depth-2 diamond fixture (transitivity is now asserted — 91/153 real targets are depth-2), externals shape fixture (AS/platform/scratch/ARG/heredoc), combo agent-axis + standalone sensitivity, per-task id sensitivity, error-path bad examples, real-repo byte-determinism, Fixture Drop cleanup + GIT_* env isolation. The bake --print gate moves from the never-run build lane to tests/static/fleet-hash.sweep.sh, wired into the per-PR static-composition job, and is now bidirectional: bake must also know no target fleet-hash missed. tests/build reverted to main. Signed-off-by: Elron Bandel <elron.bandel@ibm.com>
This was referenced Aug 9, 2026
Closed
elronbandel
added a commit
that referenced
this pull request
Aug 9, 2026
* feat(release): stamp eval.input-hash on every pushed image (#302) Rung 3 of the carried-forward ladder (#292; primitive merged in #298): every push site now records the image's build-input hash in its config (delivery/RULES.md rule 12), the value rungs 4-5 will diff and retag on. Per-target --set at every bake site — never *.labels, because a single invocation bakes targets with different hashes (bases bakes ~19, combos bakes eval + eval-standalone): - bases: one fleet-hash run, one --set per shared dep - leaves: --set "$T.labels.eval.input-hash" from the leaf's row - per-task bake (swe-bench): per-task hash via fleet-hash per-task - per-task script (terminal-bench, skills-bench): build.sh gains an optional EVAL_INPUT_HASH env hook -> --label on the final build (this path has no bake invocation to --set) - combos: fleet-hash combo <b> <a> [task] — the new optional task arg mixes the task id into the hash exactly like per-task does and names rows with the release's <bench>-<tid> convention. No-task hashes are byte-identical to the frozen #298 definition (verified against the merged script across all 153 targets + combos). Label semantics: the hash records the repo-computed inputs at the stamped commit. On a frozen-bases dev dispatch a leaf may build FROM an older base than its hash's base component claims; tagged releases always rebuild bases first, so release labels are exact (noted inline). All three --set shapes verified locally via bake --print: per-target labels land on their own targets only. Signed-off-by: Elron Bandel <elron.bandel@ibm.com> * polish: one with_task helper for the two combo task-mix sites Signed-off-by: Elron Bandel <elron.bandel@ibm.com> --------- Signed-off-by: Elron Bandel <elron.bandel@ibm.com>
This was referenced Aug 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #297. Rung 2 of #292's ratified migration path (rules landed in #293).
What
containers/scripts/fleet-hash.sh— the build-input hash primitive behind delivery rules 11–14, plus the offline static-stage property tests and a per-PR CI gate pinning the graph reading to plain bake. No release-behavior changes — nothing consumes the hash yet (the one workflow change adds a check step to the static-composition job).hash(target) = sha256 of the sorted git tree hashes of the target's context and every transitive in-repo base context, computed from the containers/ tree materialized atREFviagit archive— a pure function of the commit; uncommitted worktree state is invisible. Each row decomposes intohash / context-hash / bases-hash / externalsso a consumer can tell why a target moved. External FROMs are emitted with same-Dockerfile ARG defaults expanded; a ref still carrying${…}is per-build by design.Audited before merge
Four adversarial audit agents (script edge cases / test rigor via mutation analysis / consumer fitness for rungs 3–5 / doctrine & hygiene) ran against this branch; every reproduced finding is fixed and pinned by a test:
containers/core/runner/+entrypoint/(COPY'd by the combination Dockerfiles). Before: a framework-launcher edit changed every combo image and moved zero hashes — reproduced on real commitb072ff1d, the exact false-fresh failure rule 14 forbids.git archived tree.from X import Yparsed as FROM); alias-ordering and--platformbranches pinned by a shape fixture.ulimit -n 256(closure splitter closed files).target:refs, unparsable sums lines, empty/whitespace task ids.combination.docker-bake.hcl's*_IMAGEdefaults, not hardcoded.Gates
tests/static/input_hash.rs(runs on every PR, in "clippy + daemon-free tests"): determinism; transitive cascade over a depth-2 diamond fixture (91/153 real targets are depth-2); component decomposition;REF=HEAD~1re-derivation; worktree-purity; externals across every Dockerfile shape; combo agent-axis + standalone sensitivity; per-task id sensitivity; bad-example error paths. ~4.7s of the 30s static budget.tests/static/fleet-hash.sweep.sh(new step in the static-composition job): onedocker buildx bake --printover the full-fset, compared bidirectionally — every fleet-hash target must match bake's context/deps, and bake must know no target fleet-hash missed.--printis client-side HCL evaluation: no daemon, verified against a deadDOCKER_HOST.Pre-existing red:
task_inspection::inspect_every_existing_fixtureis red onmainand deliberately skipped by CI — untouched here.Rules checked against
.agents/contributing/RULES.md1–3 (resolves drift issue drift: delivery/RULES.md#12,#14 — no build-input hash is computed or recorded anywhere #297; code-only PR; this section)..agents/delivery/RULES.md11–14 (the computed-hash side; recording/comparison are later rungs)..agents/src/RULES.md3, 11 (graph logic incontainers/scripts/, not the CLI).tests/static/RULES.md1–3 (offline, <30s, never ignored; the sweep uses the docker CLI client-side only, same as compose.config.sweep.sh)..agents/RULES.md7–8 (bash + git + awk + jq + sha256; no new static-crate dependencies).