Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .agents/RULES.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S

9. **Pin by default, control via two orthogonal knobs.** Every benchmark, agent, and model image MUST ship with a reproducible default that runs with no environment variables set. Every image MUST expose two independent version controls:

- **Container version** (the Eval Containers-authored wiring) is selected by the **image tag**, set via `EVAL_BENCHMARK_TAG`, `EVAL_AGENT_TAG`, `EVAL_MODEL_TAG`. This is Docker's native versioning mechanism — different tag, different pull, different bits. The fleet-wide default is **one release version**: a SemVer set by the git tag (`latest` on `main`), applied to every image and to the per-benchmark `eval-<benchmark>` compose + `charts/eval` artifacts. `Cargo.toml` and the Helm `Chart.yaml` MUST carry that same version (guard: `tests/static/check.rs`). Bumps: **major** = breaking (a benchmark removed or renamed, the `EVAL_*` contract or `result.json`/output format changed); **minor** = additive (new benchmarks/agents/models, new backward-compatible flags); **patch** = rebuilds, base-image/CVE updates, fixes with no behavior change. The per-component `EVAL_*_TAG` overrides pull a single artifact at a different release version. The tag encodes *our* version, never the upstream software version (that is the Internal version below).
- **Container version** (the Eval Containers-authored wiring) is selected by the **image tag**, set via `EVAL_BENCHMARK_TAG`, `EVAL_AGENT_TAG`, `EVAL_MODEL_TAG`. This is Docker's native versioning mechanism — different tag, different pull, different bits. A release tag MAY point at a digest produced by an earlier release when the image's build inputs are unchanged ([delivery/RULES.md](delivery/RULES.md) rules 11–15). The fleet-wide default is **one release version**: a SemVer set by the git tag (`latest` on `main`), applied to every image and to the per-benchmark `eval-<benchmark>` compose + `charts/eval` artifacts. `Cargo.toml` and the Helm `Chart.yaml` MUST carry that same version (guard: `tests/static/check.rs`). Bumps: **major** = breaking (a benchmark removed or renamed, the `EVAL_*` contract or `result.json`/output format changed); **minor** = additive (new benchmarks/agents/models, new backward-compatible flags); **patch** = rebuilds, base-image/CVE updates, fixes with no behavior change. The per-component `EVAL_*_TAG` overrides pull a single artifact at a different release version. The tag encodes *our* version, never the upstream software version (that is the Internal version below).

- **Internal version** (the upstream software baked or installed inside) is selected at runtime via `EVAL_BENCHMARK_VERSION`, `EVAL_AGENT_VERSION`, `EVAL_LITELLM_VERSION`. The framework launcher (`/usr/local/bin/run`) MUST read these env vars, install or activate the requested version, and write the resolved version to the run output directory so every run record is self-describing.

Expand Down Expand Up @@ -159,3 +159,4 @@ not as issues. The issue tracker is for tracked work only.
| 2026-06-11 | Principle 15.f: clarified that provenance labels (`org.opencontainers.image.*`) are out of scope for the per-artifact bake files — they're stamped at build time (src/RULES.md principle 11), keeping these files to targets/contexts/args/tags. No change to bake-file content. |
| 2026-06-14 | Principle 9: the fleet-version default now spans the per-benchmark `eval-<benchmark>` compose artifacts (one self-contained compose per benchmark, flattened at publish) rather than a single shared `evaluate` artifact. A published OCI compose can't carry a dynamic per-benchmark `include:` — `docker compose publish` flattens includes — so per-benchmark sidecars are baked in at publish and `run --mode compose` consumes one artifact with a single `-f`. See [delivery/RULES.md](delivery/RULES.md) rule 3. |
| 2026-06-14 | Added the `contributing/` topic: the normative core of the root `CONTRIBUTING.md` moved into [`contributing/RULES.md`](contributing/RULES.md), and the References pointer now targets that doctrine. `CONTRIBUTING.md` becomes the human-facing guide; the issue taxonomy ("Issue vocabulary") stays here. No principle in this file changed. |
| 2026-08-09 | Principle 9: a release tag MAY carry forward an earlier release's digest when the image's build inputs are unchanged. Refines — does not repeal — "different tag, different pull, different bits": that phrase constrains what a *different* tag means, not how many tags one digest may carry; carrying the byte-identical digest forward is stricter immutability than a non-reproducible rebuild. Mechanics in [delivery/RULES.md](delivery/RULES.md) rules 11–15. |
11 changes: 11 additions & 0 deletions .agents/delivery/RULES.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,16 @@ interpreted as described in [RFC 2119](https://www.rfc-editor.org/rfc/rfc2119).

10. **Consumer-visible entries only.** A `CHANGELOG.md` entry MUST record a change visible to a consumer of a release.

11. **Build inputs.** An image's build inputs MUST comprise its build context, the build inputs of every in-repo base image, and the resolved digests of its external base images.

12. **Recorded inputs.** Every published image MUST record a hash of its build inputs in its image configuration.

13. **Carried-forward images.** A released image whose build inputs are unchanged from a prior release MUST be retagged from that release's digest rather than rebuilt.

14. **Fail dirty.** An image whose recorded build-input hash is absent, unreadable, or different from the repository's computed hash MUST be treated as changed.

15. **Gate parity.** A carried-forward image MUST pass every release gate that a freshly built image passes.

## References

- [Process](../RULES.md) — principle 9 (the one-version policy and version knobs); principle 13 (self-contained repo).
Expand All @@ -55,4 +65,5 @@ interpreted as described in [RFC 2119](https://www.rfc-editor.org/rfc/rfc2119).
| 2026-06-11 | Initial version. Lifts the unified fleet + CLI release outcomes out of the root `RELEASE.md` into the delivery topic, which had skills but no `RULES.md`. |
| 2026-06-11 | Rule 5: permit an explicit `workflow_dispatch` (manual re-run with a version input) alongside a tag push — the fleet workflow's escape hatch; still forbids branch-push publishes. |
| 2026-06-14 | Added principles 8–10: the changelog is edited only when cutting a release tag, restricted to the Keep a Changelog sections, and limited to consumer-visible changes. |
| 2026-08-09 | Added rules 11–15 (build inputs, recorded inputs, carried-forward images, fail dirty, gate parity): a release carries an unchanged image's digest forward instead of rebuilding it, keyed on a recorded build-input hash that fails dirty and exempts nothing from release gates. Defining inputs to include resolved external-base digests makes an upstream base bump a *changed* input, so CVE refreshes rebuild naturally. Supersedes the judgment-based `skip_published` dispatch knob (#227); answers the silent-staleness objection that closed #241 (content hash, not path mapping) and implements the change-detection follow-up blessed in #168. |
| 2026-06-14 | Rule 3 + Abstract: the single shared `evaluate` compose artifact is replaced by per-benchmark `eval-<benchmark>` artifacts (one self-contained compose per benchmark, flattened at publish). A published artifact can't carry a dynamic per-benchmark `include:` (publish flattens includes), so per-benchmark sidecars (EnterpriseOps-Gym, WebArena, …) are baked in at publish and consumed with a single `-f`. |