From 6a01a1503a4074bb0209af4e67e51fe02919aba2 Mon Sep 17 00:00:00 2001 From: Elron Bandel Date: Sun, 9 Aug 2026 11:42:06 +0300 Subject: [PATCH] =?UTF-8?q?rules(delivery):=20carried-forward=20releases?= =?UTF-8?q?=20=E2=80=94=20retag=20unchanged=20images=20instead=20of=20rebu?= =?UTF-8?q?ilding=20(#292)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add delivery rules 11-15: an image's build inputs are its context + in-repo base inputs + resolved external-base digests (11); every published image records a build-input hash in its config (12); a released image with unchanged inputs is retagged from the prior release's digest, never rebuilt (13); an absent/unreadable/mismatched hash counts as changed (14); carrying forward exempts nothing from release gates (15). Principle 9 gains the matching carve-out: a release tag MAY point at a digest produced by an earlier release when build inputs are unchanged. This refines, not repeals, 'different tag, different bits' — carrying the byte-identical digest forward is stricter immutability than a non-reproducible rebuild (v0.1.0 shipped ~6,400 silently stale combos, found only by hand: #233). Keying on a content hash with fail-dirty answers the silent-staleness objection that closed #241; #168 lists this change-detection matrix as its blessed follow-up. Folding external-base digests into the inputs makes an upstream base bump a changed input, so CVE refreshes rebuild naturally (#247 interlock via rule 15). Rules-only PR per contributing/RULES.md rule 2; implementation follows in small PRs (hash primitive + static test, label stamping, freshness status, selective retag). Resolves #292 Signed-off-by: Elron Bandel --- .agents/RULES.md | 3 ++- .agents/delivery/RULES.md | 11 +++++++++++ 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/.agents/RULES.md b/.agents/RULES.md index d69884c3..df520eb2 100644 --- a/.agents/RULES.md +++ b/.agents/RULES.md @@ -31,7 +31,7 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S 9. **Pin by default, control via two orthogonal knobs.** Every benchmark, agent, and model image MUST ship with a reproducible default that runs with no environment variables set. Every image MUST expose two independent version controls: - - **Container version** (the Eval Containers-authored wiring) is selected by the **image tag**, set via `EVAL_BENCHMARK_TAG`, `EVAL_AGENT_TAG`, `EVAL_MODEL_TAG`. This is Docker's native versioning mechanism — different tag, different pull, different bits. The fleet-wide default is **one release version**: a SemVer set by the git tag (`latest` on `main`), applied to every image and to the per-benchmark `eval-` compose + `charts/eval` artifacts. `Cargo.toml` and the Helm `Chart.yaml` MUST carry that same version (guard: `tests/static/check.rs`). Bumps: **major** = breaking (a benchmark removed or renamed, the `EVAL_*` contract or `result.json`/output format changed); **minor** = additive (new benchmarks/agents/models, new backward-compatible flags); **patch** = rebuilds, base-image/CVE updates, fixes with no behavior change. The per-component `EVAL_*_TAG` overrides pull a single artifact at a different release version. The tag encodes *our* version, never the upstream software version (that is the Internal version below). + - **Container version** (the Eval Containers-authored wiring) is selected by the **image tag**, set via `EVAL_BENCHMARK_TAG`, `EVAL_AGENT_TAG`, `EVAL_MODEL_TAG`. This is Docker's native versioning mechanism — different tag, different pull, different bits. A release tag MAY point at a digest produced by an earlier release when the image's build inputs are unchanged ([delivery/RULES.md](delivery/RULES.md) rules 11–15). The fleet-wide default is **one release version**: a SemVer set by the git tag (`latest` on `main`), applied to every image and to the per-benchmark `eval-` compose + `charts/eval` artifacts. `Cargo.toml` and the Helm `Chart.yaml` MUST carry that same version (guard: `tests/static/check.rs`). Bumps: **major** = breaking (a benchmark removed or renamed, the `EVAL_*` contract or `result.json`/output format changed); **minor** = additive (new benchmarks/agents/models, new backward-compatible flags); **patch** = rebuilds, base-image/CVE updates, fixes with no behavior change. The per-component `EVAL_*_TAG` overrides pull a single artifact at a different release version. The tag encodes *our* version, never the upstream software version (that is the Internal version below). - **Internal version** (the upstream software baked or installed inside) is selected at runtime via `EVAL_BENCHMARK_VERSION`, `EVAL_AGENT_VERSION`, `EVAL_LITELLM_VERSION`. The framework launcher (`/usr/local/bin/run`) MUST read these env vars, install or activate the requested version, and write the resolved version to the run output directory so every run record is self-describing. @@ -159,3 +159,4 @@ not as issues. The issue tracker is for tracked work only. | 2026-06-11 | Principle 15.f: clarified that provenance labels (`org.opencontainers.image.*`) are out of scope for the per-artifact bake files — they're stamped at build time (src/RULES.md principle 11), keeping these files to targets/contexts/args/tags. No change to bake-file content. | | 2026-06-14 | Principle 9: the fleet-version default now spans the per-benchmark `eval-` compose artifacts (one self-contained compose per benchmark, flattened at publish) rather than a single shared `evaluate` artifact. A published OCI compose can't carry a dynamic per-benchmark `include:` — `docker compose publish` flattens includes — so per-benchmark sidecars are baked in at publish and `run --mode compose` consumes one artifact with a single `-f`. See [delivery/RULES.md](delivery/RULES.md) rule 3. | | 2026-06-14 | Added the `contributing/` topic: the normative core of the root `CONTRIBUTING.md` moved into [`contributing/RULES.md`](contributing/RULES.md), and the References pointer now targets that doctrine. `CONTRIBUTING.md` becomes the human-facing guide; the issue taxonomy ("Issue vocabulary") stays here. No principle in this file changed. | +| 2026-08-09 | Principle 9: a release tag MAY carry forward an earlier release's digest when the image's build inputs are unchanged. Refines — does not repeal — "different tag, different pull, different bits": that phrase constrains what a *different* tag means, not how many tags one digest may carry; carrying the byte-identical digest forward is stricter immutability than a non-reproducible rebuild. Mechanics in [delivery/RULES.md](delivery/RULES.md) rules 11–15. | diff --git a/.agents/delivery/RULES.md b/.agents/delivery/RULES.md index 95bfdca3..956abf73 100644 --- a/.agents/delivery/RULES.md +++ b/.agents/delivery/RULES.md @@ -41,6 +41,16 @@ interpreted as described in [RFC 2119](https://www.rfc-editor.org/rfc/rfc2119). 10. **Consumer-visible entries only.** A `CHANGELOG.md` entry MUST record a change visible to a consumer of a release. +11. **Build inputs.** An image's build inputs MUST comprise its build context, the build inputs of every in-repo base image, and the resolved digests of its external base images. + +12. **Recorded inputs.** Every published image MUST record a hash of its build inputs in its image configuration. + +13. **Carried-forward images.** A released image whose build inputs are unchanged from a prior release MUST be retagged from that release's digest rather than rebuilt. + +14. **Fail dirty.** An image whose recorded build-input hash is absent, unreadable, or different from the repository's computed hash MUST be treated as changed. + +15. **Gate parity.** A carried-forward image MUST pass every release gate that a freshly built image passes. + ## References - [Process](../RULES.md) — principle 9 (the one-version policy and version knobs); principle 13 (self-contained repo). @@ -55,4 +65,5 @@ interpreted as described in [RFC 2119](https://www.rfc-editor.org/rfc/rfc2119). | 2026-06-11 | Initial version. Lifts the unified fleet + CLI release outcomes out of the root `RELEASE.md` into the delivery topic, which had skills but no `RULES.md`. | | 2026-06-11 | Rule 5: permit an explicit `workflow_dispatch` (manual re-run with a version input) alongside a tag push — the fleet workflow's escape hatch; still forbids branch-push publishes. | | 2026-06-14 | Added principles 8–10: the changelog is edited only when cutting a release tag, restricted to the Keep a Changelog sections, and limited to consumer-visible changes. | +| 2026-08-09 | Added rules 11–15 (build inputs, recorded inputs, carried-forward images, fail dirty, gate parity): a release carries an unchanged image's digest forward instead of rebuilding it, keyed on a recorded build-input hash that fails dirty and exempts nothing from release gates. Defining inputs to include resolved external-base digests makes an upstream base bump a *changed* input, so CVE refreshes rebuild naturally. Supersedes the judgment-based `skip_published` dispatch knob (#227); answers the silent-staleness objection that closed #241 (content hash, not path mapping) and implements the change-detection follow-up blessed in #168. | | 2026-06-14 | Rule 3 + Abstract: the single shared `evaluate` compose artifact is replaced by per-benchmark `eval-` artifacts (one self-contained compose per benchmark, flattened at publish). A published artifact can't carry a dynamic per-benchmark `include:` (publish flattens includes), so per-benchmark sidecars (EnterpriseOps-Gym, WebArena, …) are baked in at publish and consumed with a single `-f`. |