feat: deny releases using experimental Hermeto backends

[jsmid1]

Builds whose SBOM contains components fetched by an experimental Hermeto backend (hermeto:backend:experimental:*) now violate the release policy. Covers both SPDX and CycloneDX SBOM formats.

Issue: EC-1799

Co-Authored-By: Claude

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

This PR adds a new experimental_hermeto_backend validation rule for both CycloneDX and SPDX SBOM formats. The rules detect when SBOM components or packages were fetched via experimental Hermeto backends by scanning for specific annotation markers. Both formats include comprehensive Rego policy rules, test coverage, and integrated documentation across three standard policy rulesets.

Changes

Experimental Hermeto backend detection

Layer / File(s)	Summary
CycloneDX backend rule and test coverage policy/release/sbom_cyclonedx/sbom_cyclonedx.rego, policy/release/sbom_cyclonedx/sbom_cyclonedx_test.rego	Implements experimental_hermeto_backend deny rule scanning top-level annotations for hermeto:backend:experimental: prefix, matching subjects to component bom-ref values, and emitting denials keyed by component PURL or name. Tests assert experimental backends are denied while stable hermeto:backend:gomod annotations pass, covering edge cases for missing PURL and mixed annotations.
SPDX backend rule, helper, and test coverage policy/release/sbom_spdx/sbom_spdx.rego, policy/release/sbom_spdx/sbom_spdx_test.rego	Implements experimental_hermeto_backend deny rule scanning package annotations for Tool: hermeto:backend annotator with comments starting with hermeto:backend:experimental:, emitting denials keyed by package PURL or name. Adds _spdx_package_id helper to resolve identifiers. Tests mirror CycloneDX scenarios covering PURL variants and mixed annotations.
Rule documentation and ruleset integration antora/docs/modules/ROOT/pages/packages/release_sbom_cyclonedx.adoc, antora/docs/modules/ROOT/pages/packages/release_sbom_spdx.adoc, antora/docs/modules/ROOT/pages/release_policy.adoc, antora/docs/modules/ROOT/partials/release_policy_nav.adoc	Documents the new experimental_hermeto_backend rules for both CycloneDX and SPDX formats describing detection criteria and failure messages. Registers both rules in minimal, redhat, and redhat_rpms policy rulesets. Updates navigation partial with references to both rule documentation pages.

---

🎯 2 (Simple) | ⏱️ ~12 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: denying releases that use experimental Hermeto backends.
Description check	✅ Passed	The description is directly related to the change and correctly mentions both SBOM formats and the experimental Hermeto backend rule.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@policy/release/sbom_spdx/sbom_spdx.rego`:
- Around line 370-386: The rule deny contains result currently requires finding
an externalRef with referenceType "purl" (variable externalref and id :=
externalref.referenceLocator), which causes annotated packages without a purl to
skip this deny; modify the logic in the deny rule so that if no purl externalRef
exists you still produce a result by deriving id from a fallback identifier
(e.g., pkg.name or pkg.SPDXID) instead of requiring externalref; update the
variables used by metadata.result_helper_with_term (the id passed to
metadata.result_helper_with_term and any uses of pkg.externalRefs) to use the
purl when present and the fallback when absent so annotated SPDX packages are
denied consistently.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Enterprise

Run ID: b580bfad-756c-4f64-9a1b-317e1dbf7c92

📥 Commits

Reviewing files that changed from the base of the PR and between 9da3421 and ca05deb.

📒 Files selected for processing (8)

• antora/docs/modules/ROOT/pages/packages/release_sbom_cyclonedx.adoc
• antora/docs/modules/ROOT/pages/packages/release_sbom_spdx.adoc
• antora/docs/modules/ROOT/pages/release_policy.adoc
• antora/docs/modules/ROOT/partials/release_policy_nav.adoc
• policy/release/sbom_cyclonedx/sbom_cyclonedx.rego
• policy/release/sbom_cyclonedx/sbom_cyclonedx_test.rego
• policy/release/sbom_spdx/sbom_spdx.rego
• policy/release/sbom_spdx/sbom_spdx_test.rego

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

Flag	Coverage Δ
unit-tests	100.00% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
policy/release/sbom_cyclonedx/sbom_cyclonedx.rego	100.00% <100.00%> (ø)
...cy/release/sbom_cyclonedx/sbom_cyclonedx_test.rego	100.00% <100.00%> (ø)
policy/release/sbom_spdx/sbom_spdx.rego	100.00% <100.00%> (ø)
policy/release/sbom_spdx/sbom_spdx_test.rego	100.00% <100.00%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[simonbaird]

I think it's good. Can you think of a way to test it on a real image with a real sbom that has an experimental Hermeto? The reporter might be able to share an image like that.

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 7:19 AM UTC · Completed 7:29 AM UTC
Commit: 47d3320 · View workflow run →

[fullsend-ai-review]

Looks good to me

Previous run

Review

Findings

Low

• [edge-case] policy/release/sbom_cyclonedx/sbom_cyclonedx.rego:384 — If the same component is referenced by multiple experimental-prefixed annotations, separate deny results are produced for each distinct annotation.text value. This is correct Rego set semantics (each result is genuinely distinct), not a bug. Noting for awareness.

• [edge-case] policy/release/sbom_cyclonedx/sbom_cyclonedx.rego:389 — The rule accesses top-level s.annotations, which is a CycloneDX 1.5+ feature. For 1.4 SBOMs, the field is undefined and the rule silently produces no results. This is safe (Rego handles undefined gracefully) and is already covered by the existing test_valid_cdx_1_4 test.

Info

• [api-contract] policy/release/sbom_spdx/sbom_spdx.rego:395 — The new _spdx_package_id helper resolves package identity via _package_purl (first purl-typed ref, falling back to pkg.name), which differs from the disallowed_package_attributes rule's per-ref iteration. This is intentional and correct for this context.

• [missing-authorization] — No linked GitHub issue; the PR references an external Jira ticket (EC-1799). Authorization inferred from the Jira reference.

Previous run (2)

Looks good to me

Findings

Low

• [logic-error] policy/release/sbom_spdx/sbom_spdx.rego:382 — The new SPDX deny rule checks annotation.annotator == "Tool: hermeto:backend", while all existing SPDX Hermeto annotations use "Tool: hermeto:jsonencoded". This appears to be an intentionally new and distinct annotation type for backend identification, and the PR's tests, docs, and metadata are internally consistent. However, confirm with the Hermeto team that annotations for the experimental-backend signal use this exact annotator string.

• [test-inadequate] policy/release/sbom_cyclonedx/sbom_cyclonedx_test.rego — The CycloneDX experimental-backend tests do not include a test case where annotation.subjects contains a bom-ref that does not match any component. Standard Rego join semantics handle this correctly (zero results), but an explicit test would document this invariant.

• [pattern-violation] policy/release/sbom_spdx/sbom_spdx.rego — The new _spdx_package_id helper duplicates the PURL extraction logic of the existing _package_purl helper but with a different fallback (pkg.name instead of ""). Consider consolidating to reduce maintenance hazard.

• [edge-case] policy/release/sbom_cyclonedx/sbom_cyclonedx.rego — object.get(component, "purl", component.name) will be undefined if a component lacks both purl and name fields, silently skipping that component. Unlikely per the CycloneDX schema where name is required.

Previous run (3)

Review

Findings

Medium

• [edge-case] policy/release/sbom_spdx/sbom_spdx.rego:370 — The new _spdx_package_id function uses some externalref in pkg.externalRefs to find a purl-type ref and returns its referenceLocator. If a package has multiple externalRefs with referenceType == "purl" and different referenceLocator values, OPA will raise a complete rule conflict error because the function would produce multiple distinct return values. The existing _package_purl function in the same file avoids this by collecting into a list comprehension and taking index 0.
  Remediation: Use the same list-comprehension pattern as _package_purl: _spdx_package_id(pkg) := id if { purls := [ref.referenceLocator | some ref in pkg.externalRefs; ref.referenceType == "purl"]; id := purls[0] } else := pkg.name

Low

• [test-adequacy] policy/release/sbom_cyclonedx/sbom_cyclonedx_test.rego:1198 — The CycloneDX experimental_hermeto_backend rule uses object.get(component, "purl", component.name) to fall back to component.name when no purl is present. The SPDX side has a dedicated test_experimental_hermeto_backend_spdx_no_purl_denied test exercising the equivalent fallback in _spdx_package_id, but the CycloneDX side has no corresponding test for the no-purl fallback path.

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 10:35 AM UTC · Completed 10:45 AM UTC
Commit: 47d3320 · View workflow run →

[st3penta]

Beware: these new rules don't have an effective_on date, meaning they will take effect immediately on deployment.
If no production builds currently use experimental Hermeto backends, that's fine. Otherwise, an effective_on date, and maybe a konflux announce would give teams a migration window.

Is the immediate enforcement intentional?

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 11:13 AM UTC · Completed 11:24 AM UTC
Commit: 47d3320 · View workflow run →

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 2:11 PM UTC · Ended 2:14 PM UTC
Commit: 47d3320 · View workflow run →

[jsmid1]

Beware: these new rules don't have an effective_on date, meaning they will take effect immediately on deployment.
If no production builds currently use experimental Hermeto backends, that's fine. Otherwise, an effective_on date, and maybe a konflux announce would give teams a migration window.

@st3penta I set the effective_on date to 1.8.2026. Would that be a long enough time window?
I will send the announcement once the PR gets merged.

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 2:16 PM UTC · Completed 2:27 PM UTC
Commit: 47d3320 · View workflow run →

[simonbaird]

Lgtm, nice PR.

[simonbaird]

About the effective date, you could ask in Jira what the requestor and/or stakeholders think is the best choice. Perhaps they'd prefer something shorter. (A good technique just in case no-one cares, you can say "I'm planning to have this rule become affective on X, let me know if you prefer another date.)

One more note: FYI, the way we're deploying policy changes now, it takes up to two weeks to go from merged to deployed, which might be a reason to choose a later effective date.

[jsmid1]

Tested with SPDX SBOM https://redhat.atlassian.net/browse/EC-1799?focusedCommentId=17374185.

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 2:27 PM UTC · Completed 2:34 PM UTC
Commit: 47d3320 · View workflow run →

[fullsend-ai-retro]

Retro: PR #1736 — feat: deny releases using experimental Hermeto backends

Timeline: Human-authored PR by jsmid1, created June 2, merged June 24 (22 days). The fullsend-ai-review bot first ran on June 19 (17-day gap from PR creation — unclear why). The bot ran 5 times on the same commit (47d3320), submitting 3 separate APPROVED reviews. Human reviewers (simonbaird, st3penta) provided substantive feedback.

Review quality: The bot flagged legitimate issues (annotator string mismatch, PURL extraction duplication, OPA complete-rule-conflict risk, test gaps). However, it missed a significant release safety concern: new deny rules lacked effective_on dates, meaning they would enforce immediately on deployment with no migration window. Human reviewer st3penta caught this on June 22. The PR author subsequently added an effective_on date of 2026-08-01.

Redundant dispatches: The bot reviewed the same commit 5 times and approved 3 times. This is already tracked by existing issues in fullsend-ai/fullsend: #963 (skip dispatch when HEAD SHA already reviewed) and #1452 (deduplicate dispatches for same SHA). No new proposal needed.

1 proposal filed to add review guidance about effective_on dates to the conforma/policy AGENTS.md.

Proposals filed

• Add effective_on review checklist item to AGENTS.md for new deny rules (in conforma/policy)