diff --git a/docs/auto-release-workflow.md b/docs/auto-release-workflow.md new file mode 100644 index 000000000..b303cde03 --- /dev/null +++ b/docs/auto-release-workflow.md @@ -0,0 +1,1288 @@ +# Automated Konflux Release Workflow + +This document describes the **automated release process** for the Multiarch Tuning Operator using Konflux CI/CD. This replaces the manual release process with automated pipelines that handle validation, release, and catalog updates. + +> **Note**: For the previous manual release process (still useful for understanding release mechanics), see [docs/ocp-release.md](./ocp-release.md). + +--- + +## Table of Contents + +- [Overview](#overview) +- [Key Differences from Manual Process](#key-differences-from-manual-process) +- [Prerequisites](#prerequisites) +- [Release Workflow Stages](#release-workflow-stages) + - [1. Pre-Merge: Development and Configuration](#1-pre-merge-development-and-configuration) + - [2. Merge Trigger and Nudge PR Workflow](#2-merge-trigger-and-nudge-pr-workflow) + - [3. Automated Pipeline Execution](#3-automated-pipeline-execution) + - [4. Post-Release: FBC Catalog Update](#4-post-release-fbc-catalog-update) +- [Configuration](#configuration) + - [ReleasePlan Configuration](#releaseplan-configuration) + - [ReleasePlanAdmission Configuration](#releaseplanadmission-configuration) + - [Service Account Setup](#service-account-setup) +- [Enterprise Contract (EC) Validation](#enterprise-contract-ec-validation) +- [Snapshot Validation](#snapshot-validation) +- [FBC Automation](#fbc-automation) +- [Monitoring and Troubleshooting](#monitoring-and-troubleshooting) +- [Emergency Procedures](#emergency-procedures) +- [References](#references) + +--- + +## Overview + +The automated release workflow orchestrates the complete release lifecycle from code merge to catalog publication: + +```mermaid +flowchart TD + Start[PR Merged to Release Branch] --> OnPush1[On-Push Pipeline 1] + OnPush1 --> Build1[Build Operator and Bundle] + Build1 --> Snapshot1[Snapshot 1 Created] + Snapshot1 --> EC1{EC Tests} + EC1 -->|Failed| ECFail[Fix and Retry] + EC1 -->|Passed| ValCheck1{Snapshot Validation} + ValCheck1 -->|Expected Fail| NudgePR[Renovate Creates Nudge PR] + + NudgePR --> NudgeMerge[Nudge PR Auto-Merges] + NudgeMerge --> OnPush2[On-Push Pipeline 2] + OnPush2 --> Build2[Rebuild Bundle with Correct Image] + Build2 --> Snapshot2[Snapshot 2 Created] + Snapshot2 --> EC2{EC Tests} + EC2 -->|Passed| AutoRelease[Auto Release Triggered] + + AutoRelease --> Collectors[Collectors Pipeline] + Collectors --> SnapshotVal[Snapshot Validation] + SnapshotVal --> ValCheck2{SHA Match?} + ValCheck2 -->|Yes| TenantConfig[Tenant Config Pipeline] + ValCheck2 -->|No| ValFail[Should Not Happen] + + TenantConfig --> ReleasePipeline[Release Pipeline] + ReleasePipeline --> PushImages[Push to Production] + PushImages --> EC3{EC Release Check} + EC3 -->|Passed| FinalPipeline[Final Pipeline] + + FinalPipeline --> GitTag[Create Git Tag] + GitTag --> FBCScripts[Run FBC Scripts] + FBCScripts --> CreateFBCPR[Create FBC PR] + CreateFBCPR --> ReviewFBC[Review and Merge FBC PR] + ReviewFBC --> FBCRelease[FBC Release] + FBCRelease --> Complete[Complete] + + ECFail -.->|Fix Code| Start + EC2 -->|Failed| ECFail + EC3 -->|Failed| ECFail + ValFail -.->|Investigate| Start + + classDef pipeline fill:#e1f5ff,stroke:#0066cc,stroke-width:2px + classDef decision fill:#fff4e1,stroke:#ff9900,stroke-width:2px + classDef action fill:#e8f5e9,stroke:#4caf50,stroke-width:2px + classDef automated fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px + + class OnPush1,OnPush2,Collectors,SnapshotVal,TenantConfig,ReleasePipeline,FinalPipeline pipeline + class EC1,EC2,EC3,ValCheck1,ValCheck2 decision + class Build1,Build2,Snapshot1,Snapshot2,PushImages,GitTag,FBCScripts,CreateFBCPR,FBCRelease,Complete action + class NudgePR,NudgeMerge,AutoRelease,ReviewFBC automated +``` + +> **⚠️ IMPORTANT: Two-Snapshot Workflow** +> +> When you merge your release PR, Konflux creates **TWO snapshots**: +> 1. **First snapshot** (from your PR merge) - Will **FAIL** validation (bundle has old operator image) +> 2. **Second snapshot** (from Renovate nudge PR) - Will **PASS** validation (bundle has correct operator image) +> +> **Always release the SECOND snapshot** created after the nudge PR merges. The first snapshot validation failure is expected and normal. + +**Key Benefits:** +- **Consistency**: Eliminates manual steps and human error +- **Speed**: Release completes in minutes, not hours +- **Validation**: Built-in snapshot consistency checks and Enterprise Contract enforcement +- **Traceability**: Automated git tags and PR creation +- **Security**: EC tests verify policy compliance before release +- **Automated Image Updates**: Renovate nudge PR ensures bundle references match operator images +- **Rollback**: Clear artifacts for quick rollback if needed + +--- + +## Key Differences from Manual Process + +| Aspect | Manual Process ([ocp-release.md](./ocp-release.md)) | Automated Process | +|--------|------------------------------------------------------|-------------------| +| **Snapshot Selection** | Manually select and validate snapshots | Automatic snapshot created on merge | +| **Image Verification** | Manual pull and inspection with `podman` | Automated validation pipeline | +| **Release Creation** | Manual `Release` CR creation and YAML editing | Triggered automatically by ReleasePlan | +| **Git Tagging** | Manual `git tag` and `git push` | Automated by final pipeline | +| **FBC Updates** | Manual script execution and PR creation | Automated PR creation with pre-filled content | +| **Comet Updates** | Manual tag addition | Still requires manual verification | +| **Release Notes** | Manual YAML editing | Still requires manual content (automation creates structure) | +| **Validation** | Manual verification of image SHAs | Automated SHA comparison pipeline | + +**What's Still Manual:** +- Writing release notes content +- Comet content stream updates +- Reviewing and merging the FBC PR +- Emergency rollbacks (follow [Emergency Procedures](#emergency-procedures)) + +--- + +## Prerequisites + +### Repository Setup + +1. **Branch Structure** + - Main branch: `main` + - Release branches: `v1.0`, `v1.1`, etc. + - FBC branch: `fbc` (separate branch for File-Based Catalogs) + + +3. **Development Stream Configuration** + - Konflux Application and Components configured + - ProjectDevelopmentStream created for the release branch + - See [ocp-release.md - Branching](./ocp-release.md#branching-and-creation-of-a-new-development-stream) for initial setup + +### Access Requirements + +- **Konflux Access**: View snapshots, releases, and pipeline runs in `multiarch-tuning-ope-tenant` namespace +- **GitHub Permissions**: Token with `repo` scope for automated PR creation +- **Git Push Access**: Ability to create tags on the release branch + +--- + +## Release Workflow Stages + +### 1. Pre-Merge: Development and Configuration + +**Version Bumping** +Version bumped using `make version` and `hack/bump-version.sh`. Merge in PR to release branch or add to branching PR. + + ```bash + # Before creating the release PR + make version VERSION=1.0.1 + ./hack/bump-version.sh + ``` + +**Pre-Merge Configuration (REQUIRED before creating the branching PR):** + +Before creating the release PR, you must update the Konflux release configuration in the konflux-release-data repository. + +#### Update ReleasePlanAdmission in Konflux Release Data + +**Location:** [GitLab - konflux-release-data](https://gitlab.cee.redhat.com/releng/konflux-release-data) + +**File to update:** `config/stone-prd-rh01.pg1f.p1/product/ReleasePlanAdmission/multiarch-tuning-ope/multiarch-tuning-operator-.yaml` + +**Example:** For v1.x releases, edit `multiarch-tuning-operator-v1-x.yaml` + +**Verify current configuration:** +```bash +# View the current ReleasePlanAdmission (deployed in rhtap-releng-tenant) +oc describe releasePlanAdmission multiarch-tuning-operator-v1-x -n rhtap-releng-tenant +``` + +**Steps:** + +1. **Add new version to allowed tags:** + ```yaml + # multiarch-tuning-operator-v1-x.yaml + spec: + defaults: + tags: + - "1.0.0" + - "1.0.0-{{ timestamp }}" + - "1.0.1" # ← ADD NEW VERSION + - "1.0.1-{{ timestamp }}" # ← ADD TIMESTAMPED VERSION + ``` + +2. **Validate OCP version targets in FBC applications** (if applicable): + ```yaml + # multiarch-tuning-operator-fbc-prod-index.yaml + spec: + applications: + - fbc-v4-17 + - fbc-v4-18 + - fbc-v4-19 # ← Verify all supported OCP versions are listed + ``` + +3. **Update release notes template** (if needed): + ```yaml + data: + releaseNotes: + type: RHEA + synopsis: "Red Hat Multiarch Tuning {{ version }}" + description: | + # Update with actual features/fixes for this release + Enhancements: + * Feature A + + Bug fixes: + * Fix B + ``` + +4. **Run manifest generation:** + ```bash + cd konflux-release-data + ./build-manifests.sh + ``` + +5. **Create Merge Request** in GitLab with changes + +6. **Wait for MR approval and merge** + + + +#### Verify ProjectDevelopmentStream Configuration + +**Location:** [GitLab - konflux-release-data](https://gitlab.cee.redhat.com/releng/konflux-release-data) + +**File:** `tenants-config/cluster/stone-prd-rh01/tenants/multiarch-tuning-ope-tenant/_base/projectl.konflux.dev/projectdevelopmentstreams/multiarch-tuning-operator-/projectdevelopmentstream.yaml` + +**Verify:** + +1. **Template values are correct:** + ```yaml + spec: + project: multiarch-tuning-operator + template: + name: multiarch-tuning-operator + values: + - name: version + value: "1.0" # ← Verify matches release branch (without 'v' prefix) + ``` + +2. **Update collectors fixVersion in template release section:** + + In the ProjectDevelopmentStream template, find the collectors JIRA query configuration and update the fixVersion: + + **Location in PDST YAML:** Look for the collectors parameters section + + ```yaml + template: + # ... other template configuration ... + release: + collectors: + - name: url + value: https://issues.redhat.com + - name: query + value: 'project = "MULTIARCH" AND fixVersion = "1.3" AND component = "Multiarch-Tuning-Operator"' + # ↑ UPDATE THIS fixVersion to match your release + - name: secretName + value: jira-secret + ``` + + **Critical steps:** + - Find the `fixVersion = "X.Y"` or `fixVersion = "X.Y.Z"` in the JIRA query string + - Update it to match your target release version + - Format can be either: + - Minor version: `"1.3"` for all v1.3.x releases + - Specific version: `"1.3.0"` for exact version targeting + - This controls which JIRA tickets are collected for release notes + - Wrong fixVersion will collect incorrect issues or fail to find release-related tickets + + **Examples:** + - For v1.4.x stream: Change `fixVersion = "1.3"` to `fixVersion = "1.4"` + - For specific v1.4.1 release: Change to `fixVersion = "1.4.1"` + - For v2.0.0 release: Change to `fixVersion = "2.0.0"` or `fixVersion = "2.0"` + +#### Update Comet Content Streams (Preparation) + +**Location:** [Comet](https://comet.engineering.redhat.com/containers/repositories/6616582895a35187a06ba2ce) + +**Action:** Prepare to add new tag to content streams (will be finalized after release) +- Make note of the new version tag: `v1.0.1` +- Content stream update happens after release pipeline completes + +#### Pre-Merge Checklist + +**Step 1: Update Konflux Configuration (in konflux-release-data GitLab repo)** + +- [ ] Clone konflux-release-data repository +- [ ] Update ReleasePlanAdmission YAML file + - [ ] Add new version to `defaults.tags` (e.g., `"1.0.1"`, `"1.0.1-{{ timestamp }}"`) + - [ ] Validate OCP version targets in FBC ReleasePlanAdmission (if applicable) + - [ ] Update release notes template (if needed) +- [ ] Update ProjectDevelopmentStream YAML file + - [ ] Verify `version` value matches release branch (e.g., `"1.0"` for v1.0 branch) + - [ ] Update collectors JIRA `fixVersion` query (e.g., change `"1.3"` to `"1.4"`) +- [ ] Run `./build-manifests.sh` to regenerate manifests +- [ ] Create GitLab MR with configuration changes +- [ ] Get MR reviewed and merged +- [ ] Verify changes deployed: `oc describe releasePlanAdmission multiarch-tuning-operator-v1-x -n rhtap-releng-tenant` + +**Step 2: Prepare Operator Repository PR** + +- [ ] Version bump completed: `make version VERSION=x.y.z && ./hack/bump-version.sh` +- [ ] All version references updated in codebase +- [ ] Create PR against release branch (e.g., `v1.0`) +- [ ] PR title/description includes release version and changes +- [ ] All pre-submit checks pass (unit tests, linters, gosec, goimports) + +**Step 3: Get PR Approved and Merge** + +- [ ] PR reviewed and approved +- [ ] No merge conflicts +- [ ] **Merge PR** (triggers automated release workflow) + +--- + +### 2. Merge Trigger and Nudge PR Workflow + +**What happens on initial PR merge:** + +1. **First merge detected by Konflux** + - On-push pipeline builds operator and bundle images + - Tekton pipeline: `.tekton/multiarch-tuning-operator--push.yaml` + - Bundle is built with **old/placeholder** operator image reference + - New snapshot created with component images + +2. **First snapshot EXPECTED to FAIL validation** + - Snapshot validation pipeline runs + - **Will likely fail** because bundle references old operator image digest + - This is normal and expected behavior + - The bundle's CSV still has the previous operator image SHA + +3. **Renovate "nudge" PR automatically created** + - Renovate bot detects the new operator image in the snapshot + - Creates automated PR to update bundle's operator image reference + - PR updates the `multiarch.openshift.io/image` annotation in CSV + - PR title: `[konflux] Update image references` or similar + +4. **Nudge PR merge creates final snapshot** + - Nudge PR auto-merges (if checks pass) + - Second on-push pipeline runs + - Bundle built with **correct** operator image digest + - Second snapshot created - **this is the snapshot to release** + - Snapshot validation now passes (operator SHA matches) + +**Monitoring the workflow:** + +```bash +# Watch for snapshots (you'll see TWO: initial + nudge) +oc get snapshots --sort-by .metadata.creationTimestamp \ + -l pac.test.appstudio.openshift.io/event-type=push,\ +appstudio.openshift.io/application=multiarch-tuning-operator- + +# Watch for the nudge PR in GitHub +gh pr list --repo openshift/multiarch-tuning-operator --head konflux + +# Check EC test results +oc get pipelineruns -l pipelinesascode.tekton.dev/event-type=push \ + --sort-by .metadata.creationTimestamp | grep enterprise-contract +``` + +**What to verify:** + +1. **First snapshot (from main release PR):** + - [ ] On-push pipeline completes successfully + - [ ] Enterprise Contract tests pass + - [ ] Snapshot created (will likely fail validation - this is OK) + +2. **Nudge PR:** + - [ ] Renovate creates nudge PR automatically (usually within minutes) + - [ ] Nudge PR updates bundle operator image reference + - [ ] Nudge PR checks pass + - [ ] Nudge PR merges automatically + +3. **Second snapshot (from nudge PR - RELEASE THIS ONE):** + - [ ] On-push pipeline completes + - [ ] Enterprise Contract tests pass + - [ ] **Snapshot validation passes** (SHA match) + - [ ] Validate: `./hack/check-snapshots.sh -v v1- -s ` + - [ ] Document this snapshot name for the release + +**Important:** +- **Do NOT release the first snapshot** - it will fail validation +- **Wait for the nudge PR** to merge and create the second snapshot +- **Release the snapshot from the nudge PR merge** - this has the correct image references + +--- + +### 3. Automated Pipeline Execution + +Once the snapshot passes Enterprise Contract validation, the ReleasePlan automatically triggers a series of release pipelines (executed in `rhtap-releng-tenant` namespace): + +#### Stage 1: Collectors Pipeline +- **Purpose**: Gather metadata and prerequisites for release +- **Duration**: ~1-2 minutes +- **What it does**: + - Collects tenant configuration + - Gathers release metadata from snapshot + - Prepares inputs for downstream pipelines +- **Outputs**: Metadata for downstream pipelines +- **Pipeline**: Part of ReleasePlanAdmission pipeline chain + +#### Stage 2: Snapshot Validation Pipeline (Pre-Release) +- **Purpose**: Verify operator image SHA in bundle matches snapshot +- **Duration**: ~2-3 minutes +- **Pipeline**: `.tekton/snapshot-validation-pipeline.yaml` +- **What it checks**: + - Extracts bundle image from snapshot components + - Pulls and extracts bundle to read ClusterServiceVersion + - Reads `multiarch.openshift.io/image` annotation from CSV deployment spec + - Extracts SHA256 from operator image reference + - Compares operator image SHA with all snapshot component SHAs + - **Fails if**: Bundle references wrong operator image version +- **Critical**: This prevents releasing inconsistent bundles + +See [Snapshot Validation](#snapshot-validation) for details. + +#### Stage 3: Tenant Config Pipeline +- **Purpose**: Validate and apply tenant-specific release configuration +- **Duration**: ~1-2 minutes +- **What it does**: + - Applies tenant release configuration + - Validates release parameters + - Ensures release target is configured correctly + +#### Stage 4: Release Pipeline +- **Purpose**: Publish images and bundles to production registries +- **Duration**: ~5-10 minutes +- **Pipeline**: `rh-push-to-registry` from release-service-catalog +- **What it does**: + - Pushes operator image to production Quay registry + - Pushes bundle image to production registry + - Creates registry tags based on ReleasePlanAdmission configuration + - Triggers IIB (Index Image Builder) for bundle integration + - Generates release metadata +- **Outputs**: + - Operator image: `registry.redhat.io/multiarch-tuning/multiarch-tuning-rhel9-operator:v1.0.1` + - Bundle image: `registry.redhat.io/multiarch-tuning/multiarch-tuning-operator-bundle:v1.0.1` + - Index image: `brew.registry.redhat.io/rh-osbs/iib:XXXXXX` + - Release notes structure +- **Enterprise Contract Release Check**: Additional EC validation runs on released images + +#### Stage 5: Post-Release (Final) Pipeline +- **Purpose**: Git tagging and FBC PR automation +- **Duration**: ~3-5 minutes +- **Pipeline**: `.tekton/fbc-update-final-pipeline.yaml` +- **What it does**: + 1. Extracts version from snapshot metadata + 2. Finds the release commit SHA from snapshot source + 3. Creates git tag (e.g., `v1.0.1`) on release commit + 4. Pushes tag to repository (creates GitHub release) + 5. Clones FBC branch + 6. Runs `hack/build-indexs.sh ` to append bundle to all OCP catalogs + 7. Runs `hack/update-graph.sh ` to update channel graphs + 8. Creates PR against FBC branch with changes +- **Pipeline results**: + ``` + VERSION: v1.0.1 + TAG_URL: https://github.com/openshift/multiarch-tuning-operator/releases/tag/v1.0.1 + FBC_PR_URL: https://github.com/openshift/multiarch-tuning-operator/pull/123 + ``` + +**Monitoring all stages:** +```bash +# View all release pipeline runs +tkn pipelinerun list -n rhtap-releng-tenant | grep multiarch-tuning + +# Follow a specific pipeline run +tkn pipelinerun logs -f -n rhtap-releng-tenant + +# Check release status +oc get release -o jsonpath='{.status.conditions}' | jq +``` + +### 4. Post-Release: FBC Catalog Update + +**What happens automatically:** +- PR created in FBC branch with: + - Updated `fbc-v4-*/catalog/multiarch-tuning-operator/index.yaml` files + - New bundle appended to all supported OCP versions + - Channel graph updated with new version as head + +**Manual actions required:** +1. Review the FBC PR +2. Verify catalog changes are correct +3. Merge the FBC PR (triggers FBC release pipeline) + +**FBC Release:** +```bash +# After FBC PR is merged, new snapshot is created +oc get snapshots --sort-by .metadata.creationTimestamp \ + -l pac.test.appstudio.openshift.io/event-type=push,\ +appstudio.openshift.io/application=fbc-v4- + +# Create release for each OCP version catalog +oc create -f - < +EOF +``` + +--- + +## Configuration + +### ReleasePlan Configuration + +#### Operator ReleasePlan + +The ReleasePlan for the operator defines the automated release behavior: + +```yaml +apiVersion: appstudio.redhat.com/v1alpha1 +kind: ReleasePlan +metadata: + name: multiarch-tuning-operator-1-0-release-as-operator + namespace: multiarch-tuning-ope-tenant +spec: + application: multiarch-tuning-operator-1-0 + target: multiarch-tuning-ope-prod + + # Collectors - gather JIRA issues for release notes + data: + collectors: + - name: url + value: https://issues.redhat.com + - name: query + value: 'project = "MULTIARCH" AND fixVersion = "mto-1.3" AND component = "Multiarch-Tuning-Operator"' + - name: secretName + value: jira-token + fixVersion: "1.0" # Must match the release version + + # Pre-release validation pipeline + pipeline: + pipelineRef: + resolver: git + params: + - name: url + value: https://github.com/openshift/multiarch-tuning-operator.git + - name: revision + value: main + - name: pathInRepo + value: .tekton/snapshot-validation-pipeline.yaml + serviceAccountName: release-service-account + timeout: "30m" + + # Post-release automation (git tag + FBC PR) + finalPipeline: + pipelineRef: + resolver: git + params: + - name: url + value: https://github.com/openshift/multiarch-tuning-operator.git + - name: revision + value: main + - name: pathInRepo + value: .tekton/fbc-update-final-pipeline.yaml + params: + - name: fbc-branch + value: fbc + - name: git-url + value: https://github.com/openshift/multiarch-tuning-operator + serviceAccountName: release-service-account + timeout: "1h" +``` + +**Important fields to update for each release:** +- `data.fixVersion`: Set to match the release version (e.g., `"1.0"` for v1.0.x releases or `"1.0.1"` for specific release) +- This ensures the collectors query picks up the correct JIRA tickets + +#### FBC ReleasePlan + +The ReleasePlan for File-Based Catalog releases includes a final pipeline that tags the FBC commit: + +```yaml +apiVersion: appstudio.redhat.com/v1alpha1 +kind: ReleasePlan +metadata: + name: fbc-4-18-release-as-fbc + namespace: multiarch-tuning-ope-tenant +spec: + application: fbc-v4-18 + target: multiarch-tuning-ope-prod + + # FBC has no pre-release validation pipeline (catalog validation done during build) + + # Post-release automation (tag the FBC commit in GitHub) + finalPipeline: + pipelineRef: + resolver: git + params: + - name: url + value: https://github.com/openshift/multiarch-tuning-operator.git + - name: revision + value: main + - name: pathInRepo + value: .tekton/fbc-tag-final-pipeline.yaml + params: + - name: git-url + value: https://github.com/openshift/multiarch-tuning-operator + - name: git-branch + value: fbc + serviceAccountName: release-service-account + timeout: "30m" +``` + +**FBC Final Pipeline Actions:** +1. Extracts the FBC snapshot's commit SHA from the `fbc` branch +2. Creates a git tag on that commit: `fbc-v4-18--` +3. Pushes the tag to GitHub +4. Provides traceability for which FBC catalog was released + +**Example FBC tag:** `fbc-v4-18-v1.0.1-20260511` +- `fbc-v4-18`: Catalog for OCP 4.18 +- `v1.0.1`: Operator version included in this catalog +- `20260511`: Release date + +### ReleasePlanAdmission Configuration + +Location: `konflux-release-data/config/stone-prd-rh01.pg1f.p1/product/ReleasePlanAdmission/multiarch-tuning-ope/` + +```yaml +apiVersion: appstudio.redhat.com/v1alpha1 +kind: ReleasePlanAdmission +metadata: + name: multiarch-tuning-operator-1-0 + namespace: multiarch-tuning-ope-tenant +spec: + applications: + - multiarch-tuning-operator-1-0 + origin: multiarch-tuning-ope-tenant + + # Allowed version tags (UPDATE BEFORE EACH RELEASE) + defaults: + tags: + - "1.0.0" + - "1.0.1" + - "1.0.1-{{ timestamp }}" + - "1.0.2" # Add new versions here + - "1.0.2-{{ timestamp }}" + + # Release notes template + data: + releaseNotes: + type: RHEA + synopsis: "Red Hat Multiarch Tuning {{ version }}" + # ... (see example in ocp-release.md) + + # Pipeline references + pipeline: + pipelineRef: + resolver: git + params: + - name: url + value: https://github.com/redhat-appstudio/release-service-catalog.git + - name: revision + value: production + - name: pathInRepo + value: pipelines/rh-push-to-registry/rh-push-to-registry.yaml + serviceAccountName: release-service-account +``` + +**Before each release:** +1. Update `defaults.tags` array with new version +2. Update in [GitLab repo](https://gitlab.cee.redhat.com/releng/konflux-release-data) +3. Run `build-manifests.sh` and create MR + +### Service Account Setup + +The release service account needs: +1. **GitHub token** for PR creation +2. **Registry credentials** for image pushes +3. **Git push permissions** for tagging + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: github-token + namespace: multiarch-tuning-ope-tenant +type: Opaque +stringData: + token: "ghp_your_github_personal_access_token" +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: release-service-account + namespace: multiarch-tuning-ope-tenant +secrets: + - name: github-token + - name: registry-credentials +``` + +**GitHub token requirements:** +- Scope: `repo` (full repository access) +- Permissions: Create PRs, push tags + +--- + +## Enterprise Contract (EC) Validation + +### Purpose + +Enterprise Contract tests enforce Red Hat's security and compliance policies before and during the release process. These automated checks ensure that only compliant images are released to production. + +### When EC Tests Run + +1. **Post-Build (On-Push)**: After snapshot creation from merged PRs + - Validates newly built images meet policy requirements + - Blocks auto-release if tests fail + - Run automatically by Konflux on every snapshot + +2. **Pre-Release (During Release Pipeline)**: Before pushing to production registry + - Additional validation on release artifacts + - Ensures no policy violations were introduced + - Part of the release pipeline execution + +### What EC Tests Validate + +```mermaid +flowchart LR + Snapshot[Snapshot Created] --> EC[Enterprise Contract Tests] + EC --> Sig[Image Signatures] + EC --> SBOM[SBOM Present] + EC --> CVE[CVE Scanning] + EC --> Policy[Red Hat Policies] + EC --> Provenance[Build Provenance] + + Sig --> Result{All Checks Pass?} + SBOM --> Result + CVE --> Result + Policy --> Result + Provenance --> Result + + Result -->|Yes| Release[Proceed to Release] + Result -->|No| Block[Block Release] +``` + +**Specific checks include:** +- **Image signatures**: All images are properly signed +- **SBOM (Software Bill of Materials)**: Complete dependency inventory present +- **CVE scanning**: No critical/high vulnerabilities or proper exceptions documented +- **Policy compliance**: Meets Red Hat image policies (base images, labels, etc.) +- **Build provenance**: Verified build pipeline and source repository +- **License compliance**: No GPL violations or unapproved licenses + +### Monitoring EC Tests + +```bash +# View EC test pipeline runs for a snapshot +oc get pipelineruns -l appstudio.openshift.io/snapshot= \ + | grep enterprise-contract + +# Check EC test results +tkn pipelinerun logs -n multiarch-tuning-ope-tenant + +# View EC policy violations in Konflux UI +# Navigate to: Application > Snapshot > Enterprise Contract tab +``` + +### Handling EC Failures + +**Common failure scenarios:** + +1. **Image not signed** + ``` + Error: Image signature verification failed + ``` + **Fix**: Ensure signing pipeline completed successfully + +2. **SBOM missing** + ``` + Error: No SBOM found for image + ``` + **Fix**: SBOM generation should be automatic; check build pipeline + +3. **CVE violations** + ``` + Error: Critical CVE-2024-XXXXX found in base image + ``` + **Fix**: + - Update base image to patched version + - Or file security exception with ProdSec team + +4. **Policy violation** + ``` + Error: Image does not use approved base image + ``` + **Fix**: Use Red Hat UBI or approved base image + +**If EC fails:** +- Review the EC pipeline logs for specific violations +- Address the policy violations in code/configuration +- Create new PR with fixes +- New snapshot will trigger EC tests again +- **Do not bypass EC tests** - they enforce critical security requirements + +### EC Test Configuration + +EC policies are configured in: +- **Konflux Enterprise Contract policy bundle**: Centrally managed by Red Hat +- **Tenant-specific exceptions**: Can be configured in ReleasePlanAdmission + +Exceptions should be rare and require ProdSec approval. + +--- + +## Snapshot Validation + +### Purpose + +Prevents releasing inconsistent bundles where the bundle's CSV references an operator image not included in the snapshot. + +> **📋 Note on Two-Snapshot Workflow** +> +> In the normal release workflow, you will see TWO snapshots and TWO validation attempts: +> +> **Snapshot 1** (from your release PR merge): +> - Bundle still references the OLD operator image digest +> - Validation will **FAIL** - this is **EXPECTED** and **NORMAL** +> - Do not try to fix this - wait for the nudge PR +> +> **Snapshot 2** (from Renovate nudge PR merge): +> - Bundle updated to reference the NEW operator image digest +> - Validation will **PASS** - this is the snapshot to release +> +> The first validation failure is part of the normal workflow and triggers the Renovate nudge PR to update the bundle. + +### How It Works + +1. **Extract bundle** from snapshot components +2. **Pull bundle image** and extract CSV +3. **Read annotation** `multiarch.openshift.io/image` from CSV deployment spec +4. **Extract SHA256** from the operator image reference +5. **Compare** with all component image SHAs in snapshot +6. **Pass/Fail** based on match + +### Validation Flow + +```mermaid +graph TD + A[Snapshot JSON] --> B[Find Bundle Component] + B --> C[Pull Bundle Image] + C --> D[Extract CSV File] + D --> E[Read multiarch.openshift.io/image] + E --> F[Extract Operator SHA] + F --> G[Get All Snapshot Component SHAs] + G --> H{SHA Match?} + H -->|Yes| I[✅ PASS - Proceed with Release] + H -->|No| J[❌ FAIL - Block Release] + J --> K[Display Mismatch Details] +``` + +### Manual Validation (Pre-Release Check) + +```bash +# Quick validation using hack script +./hack/check-snapshots.sh -v v1-0 -s multiarch-tuning-operator-v1-0-abc123 + +# Interactive mode (lists recent snapshots) +./hack/check-snapshots.sh -v v1-0 +``` + +### Common Failures + +**Stale Bundle** +``` +❌ VALIDATION FAILED +The bundle references sha256:old123... but snapshot contains sha256:new456... +``` + +**Fix:** +```bash +# Ensure bundle was built after operator image +# Check .tekton pipeline dependencies +# Verify Renovate nudging is working +``` + +**Missing Annotation** +``` +❌ Could not extract internal image from CSV annotations +``` + +**Fix:** Add annotation to CSV template: +```yaml +# config/manifests/bases/multiarch-tuning-operator.clusterserviceversion.yaml +spec: + install: + spec: + deployments: + - spec: + template: + metadata: + annotations: + multiarch.openshift.io/image: PLACEHOLDER +``` + +See [docs/snapshot-validation.md](./snapshot-validation.md) for detailed troubleshooting. + +--- + +## FBC Automation + +### What Gets Automated + +After the release pipeline completes, the final pipeline: + +1. **Tags the release commit** + ```bash + git tag v1.0.1 + git push origin v1.0.1 + ``` + +2. **Clones FBC branch** and runs: + ```bash + ./hack/build-indexs.sh + # Appends bundle to all fbc-v4-*/catalog/*/index.yaml + + ./hack/update-graph.sh v1.0.1 + # Updates channel graphs with new version as head + ``` + +3. **Creates FBC PR** with: + - Title: `Add v1.0.1 to FBC catalogs` + - Body: Includes snapshot details and component images + - Changes: All `fbc-v4-*/catalog/multiarch-tuning-operator/index.yaml` files + +### FBC Branch Structure + +``` +fbc/ +├── fbc-v4-17/ +│ └── catalog/ +│ └── multiarch-tuning-operator/ +│ └── index.yaml +├── fbc-v4-18/ +│ └── catalog/ +│ └── multiarch-tuning-operator/ +│ └── index.yaml +├── fbc-v4-19/ +│ └── catalog/ +│ └── multiarch-tuning-operator/ +│ └── index.yaml +└── hack/ + ├── build-indexs.sh # Append bundle to all catalogs + └── update-graph.sh # Update channel upgrade graphs +``` + +### Manual FBC Testing + +```bash +# Clone FBC branch +git checkout fbc + +# Get bundle from snapshot +BUNDLE_IMAGE=$(oc get snapshot my-snapshot \ + -o jsonpath='{.spec.components[?(@.name=="multiarch-tuning-operator-bundle-1-0")].containerImage}') + +# Run automation scripts +./hack/build-indexs.sh "$BUNDLE_IMAGE" +./hack/update-graph.sh v1.0.1 + +# Validate catalogs +for dir in fbc-v*/catalog/multiarch-tuning-operator; do + opm validate "$dir" +done + +# Review changes +git diff +``` + +See [docs/fbc-release-automation.md](./fbc-release-automation.md) for detailed documentation. + +--- + +## Monitoring and Troubleshooting + +### Monitoring Release Progress + +```bash +# 1. Watch for new snapshot after merge +oc get snapshots --sort-by .metadata.creationTimestamp \ + -l appstudio.openshift.io/application=multiarch-tuning-operator-1-0 \ + --watch + +# 2. Check release creation +oc get releases -n multiarch-tuning-ope-tenant --watch + +# 3. View pipeline runs (executed in rhtap-releng-tenant) +tkn pipelinerun list -n rhtap-releng-tenant + +# 4. View specific pipeline logs +tkn pipelinerun logs -f -n rhtap-releng-tenant +``` + +### Common Issues + +#### Snapshot Validation Fails + +**Symptom:** Release blocked with validation failure + +**Diagnosis:** +```bash +./hack/check-snapshots.sh -v v1-0 -s +``` + +**Fix:** +- If bundle is stale: Wait for Renovate to update bundle reference +- If annotation missing: Update CSV template and rebuild +- If SHA mismatch: Investigate build ordering issue + +#### Final Pipeline Fails to Create PR + +**Symptom:** Release completes but no FBC PR created + +**Diagnosis:** +```bash +tkn pipelinerun logs -n rhtap-releng-tenant | grep -A20 "create-pr" +``` + +**Common causes:** +- GitHub token expired +- Token lacks `repo` scope +- Network issue accessing GitHub API + +**Fix:** +```bash +# Verify token +oc get secret github-token -o jsonpath='{.data.token}' | base64 -d | cut -c1-10 +# Should show: ghp_... + +# Check token scopes at https://github.com/settings/tokens +# Ensure 'repo' scope is enabled + +# Recreate secret if needed +oc delete secret github-token +oc create secret generic github-token --from-literal=token=ghp_new_token +``` + +#### Git Tag Already Exists + +**Symptom:** Final pipeline fails: "tag already exists" + +**Diagnosis:** +```bash +git ls-remote --tags origin | grep v1.0.1 +``` + +**Fix:** +```bash +# If tag is on wrong commit +git push --delete origin v1.0.1 +# Re-run final pipeline + +# If tag is correct +# Skip final pipeline, manually create FBC PR +``` + +#### Release Notes Missing in Advisory + +**Symptom:** Release succeeds but RHEA advisory has no description + +**Fix:** Release notes are templated in ReleasePlanAdmission but content must be provided in the Release CR: + +```yaml +apiVersion: appstudio.redhat.com/v1alpha1 +kind: Release +metadata: + generateName: release-1-0-1- +spec: + # ... + data: + releaseNotes: + description: | + Enhancements: + * Feature A added + * Feature B improved + + Bug fixes: + * Fixed issue #123 + * Resolved CVE-2024-XXXXX +``` + +### Debug Commands + +```bash +# Get full snapshot details +oc get snapshot -o yaml + +# View release status +oc get release -o jsonpath='{.status}' | jq + +# Check pipeline status +oc get pipelinerun -l appstudio.openshift.io/release= + +# View task logs +tkn pipelinerun logs --task + +# Manual snapshot validation +./hack/check-snapshots.sh -v v1-0 -s + +# Extract bundle CSV manually +podman pull +podman save -o /tmp/bundle.tar +tar -xf /tmp/bundle.tar -C /tmp/bundle +find /tmp/bundle -name "*.clusterserviceversion.yaml" +``` + +--- + +## Emergency Procedures + +### Stopping a Release in Progress + +If you need to stop a release: + +```bash +# 1. Find the Release CR +oc get releases -n multiarch-tuning-ope-tenant + +# 2. Delete the Release CR (stops pipeline execution) +oc delete release + +# 3. Clean up any partial artifacts +# - Check if images were pushed +# - Delete git tag if created: git push --delete origin v1.0.1 +# - Close FBC PR if created +``` + +### Rolling Back a Release + +If a release is published but needs rollback: + +1. **Operator Image:** Previous versions remain available, no action needed +2. **FBC Catalog:** Revert the FBC PR or create new PR removing the version +3. **Git Tags:** Can delete tag, but better to document in release notes +4. **Advisory:** Contact release engineering to cancel RHEA + +### Manual Release (Fallback) + +If automation is broken, fall back to manual process: + +1. Follow [docs/ocp-release.md](./ocp-release.md) steps 1-8 +2. Document automation failure in JIRA +3. After manual release, create issue to fix automation + +--- + +## References + +### Documentation + +- [Manual Release Process](./ocp-release.md) - Original manual steps, useful for understanding mechanics +- [Snapshot Validation](./snapshot-validation.md) - Detailed validation pipeline documentation +- [FBC Release Automation](./fbc-release-automation.md) - FBC automation implementation details + +### External Resources + +- [Konflux Documentation](https://konflux-ci.dev/docs/) +- [Konflux Release Guide](https://konflux-ci.dev/docs/advanced-how-tos/releasing/) +- [OLM File-Based Catalogs](https://olm.operatorframework.io/docs/reference/file-based-catalogs/) +- [Operator Package Manager](https://github.com/operator-framework/operator-registry) + +### Internal Resources + +- [Konflux Release Data GitLab](https://gitlab.cee.redhat.com/releng/konflux-release-data) +- [Release Epic Template](https://spaces.redhat.com/spaces/ENTMQMAAS/pages/151038604/Creating+a+new+micro+Z-stream+release) +- [Comet Container Registry](https://comet.engineering.redhat.com/containers/repositories/6616582895a35187a06ba2ce) + +### Related Scripts + +- `hack/bump-version.sh` - Version bumping automation +- `hack/check-snapshots.sh` - Manual snapshot validation +- `hack/build-indexs.sh` - FBC catalog bundle appending +- `hack/update-graph.sh` - FBC channel graph updates + +--- + +## Appendix: Complete Release Checklist + +### Planning Phase (1-2 weeks before release) + +- [ ] Create release epic in JIRA +- [ ] Coordinate with docs team for release notes content +- [ ] Notify QE of upcoming release +- [ ] Verify ProdSec clearance (SAST/vulnerability scan) +- [ ] Cherry-pick required fixes to release branch + +### Pre-Merge Configuration (Required BEFORE merging release PR) + +**Step 1: Update Konflux Configuration in konflux-release-data GitLab** + +- [ ] Clone konflux-release-data repository +- [ ] Update ReleasePlanAdmission YAML (`multiarch-tuning-operator-v1-x.yaml`) + - [ ] Add new version to `defaults.tags` array (e.g., `"1.0.1"`, `"1.0.1-{{ timestamp }}"`) + - [ ] Validate OCP version targets in FBC ReleasePlanAdmission (if applicable) + - [ ] Update release notes template (if needed) +- [ ] Update ProjectDevelopmentStream YAML + - [ ] Verify `version` value matches release branch (e.g., `"1.0"`) + - [ ] Update collectors JIRA `fixVersion` in query (e.g., change `"1.3"` to `"1.4"`) +- [ ] Run `./build-manifests.sh` in konflux-release-data +- [ ] Create GitLab MR with configuration changes +- [ ] Get MR reviewed and merged +- [ ] Verify configuration deployed: `oc describe releasePlanAdmission multiarch-tuning-operator-v1-x -n rhtap-releng-tenant` + +**Step 2: Create and Merge Operator Release PR** + +- [ ] Version bump: `make version VERSION=x.y.z && ./hack/bump-version.sh` +- [ ] Verify all version references updated +- [ ] Create PR against release branch (e.g., `v1.0`) +- [ ] All pre-submit checks pass (tests, linters, security scans) +- [ ] PR reviewed and approved +- [ ] **Merge PR** (triggers automated workflow) + +### Post-Merge: Nudge PR and Snapshot Workflow + +**First Snapshot (from main release PR merge):** + +- [ ] On-push pipeline triggered by merge +- [ ] Operator and bundle images built +- [ ] First snapshot created +- [ ] Enterprise Contract tests pass +- [ ] Snapshot validation **EXPECTED TO FAIL** (bundle has old operator image reference) +- [ ] Document first snapshot name + +**Nudge PR (automated by Renovate):** + +- [ ] Wait for Renovate nudge PR to be created automatically (usually within 5-10 minutes) + - [ ] PR title: `[konflux] Update image references` or similar + - [ ] PR updates bundle operator image digest in CSV +- [ ] Verify nudge PR checks are passing +- [ ] Nudge PR auto-merges (or manually merge if needed) + +**Second Snapshot (from nudge PR merge - THIS IS THE RELEASE SNAPSHOT):** + +- [ ] On-push pipeline triggered by nudge PR merge +- [ ] Bundle rebuilt with correct operator image reference +- [ ] Second snapshot created +- [ ] Enterprise Contract tests pass +- [ ] **Snapshot validation PASSES** (SHA match verified) +- [ ] Validate: `./hack/check-snapshots.sh -v v1-x -s ` +- [ ] **Document this snapshot name - this is the one to release** + +### Automated Release Execution (Using nudge PR snapshot) + +- [ ] Create Release CR (or wait for auto-release) using **second snapshot** (from nudge PR) +- [ ] Collectors pipeline completes +- [ ] Snapshot validation pipeline passes (SHA match confirmed) +- [ ] Tenant config pipeline completes +- [ ] Release pipeline completes + - [ ] Images pushed to production registry + - [ ] Enterprise Contract release check passes +- [ ] Post-release final pipeline completes + - [ ] Git tag created: `vx.y.z` + - [ ] FBC PR created automatically + +### Post-Release + +- [ ] Review FBC PR changes +- [ ] Merge FBC PR +- [ ] Create FBC Release CRs for each OCP version +- [ ] Update Comet content streams (manual) +- [ ] Verify release notes in advisory +- [ ] Announce release in Slack +- [ ] Update release epic status + +### Validation + +- [ ] Test operator in OCP cluster from OperatorHub +- [ ] Verify version shows correctly +- [ ] QE validation sign-off +- [ ] Close JIRA epic \ No newline at end of file