Skip to content

CNTRLPLANE-3364: added vault key rotation func#2229

Open
sandeepknd wants to merge 1 commit into
openshift:masterfrom
sandeepknd:vault-key-rotate
Open

CNTRLPLANE-3364: added vault key rotation func#2229
sandeepknd wants to merge 1 commit into
openshift:masterfrom
sandeepknd:vault-key-rotate

Conversation

@sandeepknd
Copy link
Copy Markdown

@sandeepknd sandeepknd commented May 15, 2026

Added the vault kms key rotation function.
Currently two functions are defined :
first one only rotates the key - vault creates multiple versions after rotation; keeping the min_decryption_version same.
second one rotates the key and also updates the min_decryption_version field to the latest version resulting in only 1 key available at at time for decryption.

Let me know your thoughts if both functions are needed.

Summary by CodeRabbit

  • New Features
    • Added test utilities to trigger Vault key rotations and verify the key version increases.
    • Added ability to read Vault key metadata for validation during tests.
    • Support for default and customizable Vault target settings to simplify testing across environments; improved error reporting with captured command output on failures.

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented May 15, 2026

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 15, 2026
@sandeepknd sandeepknd marked this pull request as ready for review May 15, 2026 13:07
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 15, 2026
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 15, 2026

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Adds a VaultKeyRotator test utility with defaults, constructors, methods to read transit key metadata via oc exec, force a rotation, and verify the key version increased by parsing Vault's JSON output with timeouts.

Changes

Vault Key Rotation Utility

Layer / File(s) Summary
Type definition and configuration
test/library/encryption/kms/vault_key_rotation.go
Adds exported defaults (DefaultVaultNamespace, DefaultVaultPodName, DefaultTransitKeyName), KeyRotator interface, VaultKeyRotator struct, and two constructors for default and custom configuration.
Force rotation flow
test/library/encryption/kms/vault_key_rotation.go
Implements ForceKeyRotation: reads current latest_version, runs vault write -f transit/keys/<key>/rotate via oc exec, re-reads version, and errors if version did not increase; includes rotateKey and getCurrentKeyVersion helpers with a command timeout.
GetKeyInfo and JSON parsing
test/library/encryption/kms/vault_key_rotation.go
GetKeyInfo runs oc exec ... vault read -format=json transit/keys/<key>, returns parsed map[string]interface{}, and parseJSON rejects empty output and unmarshals JSON.

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 10 | ❌ 2

❌ Failed checks (2 inconclusive)

Check name Status Explanation Resolution
Topology-Aware Scheduling Compatibility ❓ Inconclusive No result was produced after verification. Marking as INCONCLUSIVE. Re-run the check or adjust instructions to produce a final result.
Title check ❓ Inconclusive The title is partially related to the changeset. It mentions 'vault key rotation func' which refers to a real aspect of the change, but uses vague phrasing ('func') and omits important context about the test helper and Kubernetes integration that are central to the implementation. Consider a more descriptive title that specifies this is a test helper, such as 'Add VaultKeyRotator test helper for Vault Transit key rotation' or similar to better convey the nature and scope of the addition.
✅ Passed checks (10 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The file is a test helper library with no Ginkgo test declarations (It, Describe, etc.), so the test name check does not apply.
Test Structure And Quality ✅ Passed PR adds test helper library, not Ginkgo tests. Custom check applies only to Ginkgo test code (Describe/It blocks, BeforeEach/AfterEach), which is not present in this file.
Microshift Test Compatibility ✅ Passed Adds test helper library, not Ginkgo e2e tests. No It/Describe/Context/When functions. MicroShift check not applicable.
Single Node Openshift (Sno) Test Compatibility ✅ Passed No Ginkgo e2e tests added. The PR adds only a test helper library with interfaces and utilities for Vault key rotation. No test patterns (It, Describe, Context, etc.) detected.
Ote Binary Stdout Contract ✅ Passed No OTE stdout violations. The helper library has no main(), init(), or suite-level functions, no top-level stdout writes, and all fmt usage is within function bodies for error handling only.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed This is a helper/library file with no Ginkgo tests. No IPv4 assumptions or external connectivity found. All operations are cluster-internal.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Tip

💬 Introducing Slack Agent: The best way for teams to turn conversations into code.

Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.

  • Generate code and open pull requests
  • Plan features and break down work
  • Investigate incidents and troubleshoot customer tickets together
  • Automate recurring tasks and respond to alerts with triggers
  • Summarize progress and report instantly

Built for teams:

  • Shared memory across your entire org—no repeating context
  • Per-thread sandboxes to safely plan and execute work
  • Governance built-in—scoped access, auditability, and budget controls

One agent for your entire SDLC. Right inside Slack.

👉 Get started


Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented May 15, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sandeepknd
Once this PR has been reviewed and has the lgtm label, please assign dgrisonnet for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot requested review from dgrisonnet and p0lyn0mial May 15, 2026 13:07
Comment thread test/library/encryption/kms/vault_key_rotation.go Outdated
@sandeepknd
Copy link
Copy Markdown
Author

/assign @tjungblu

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
test/library/encryption/kms/vault_key_rotation.go (1)

63-132: ⚡ Quick win

Extract shared rotation flow to reduce drift between ForceKeyRotation and RotateKeyOnly.

Both methods duplicate the same 5-step sequence (pre-check, rotate, post-check, increment validation, final validation). A single helper with a forceMinDecryption flag will reduce maintenance risk.

♻️ Refactor sketch
 func (v *VaultKeyRotator) ForceKeyRotation(ctx context.Context) (int, error) {
-	// duplicated flow...
-	...
+	return v.rotateAndValidate(ctx, true)
 }
 
 func (v *VaultKeyRotator) RotateKeyOnly(ctx context.Context) (int, error) {
-	// duplicated flow...
-	...
+	return v.rotateAndValidate(ctx, false)
 }
+
+func (v *VaultKeyRotator) rotateAndValidate(ctx context.Context, forceMinDecryption bool) (int, error) {
+	initialVersion, err := v.getCurrentKeyVersion(ctx)
+	if err != nil {
+		return 0, fmt.Errorf("failed to get initial key version: %w", err)
+	}
+	if err := v.rotateKey(ctx); err != nil {
+		return 0, fmt.Errorf("failed to rotate key: %w", err)
+	}
+	newVersion, err := v.getCurrentKeyVersion(ctx)
+	if err != nil {
+		return 0, fmt.Errorf("failed to get new key version: %w", err)
+	}
+	if newVersion <= initialVersion {
+		return 0, fmt.Errorf("rotation failed: version did not increase (before=%d, after=%d)", initialVersion, newVersion)
+	}
+	if forceMinDecryption {
+		if err := v.updateMinDecryptionVersion(ctx, newVersion); err != nil {
+			return 0, fmt.Errorf("failed to update min_decryption_version: %w", err)
+		}
+	}
+	if err := v.ValidateKeyRotation(ctx, newVersion); err != nil {
+		return 0, fmt.Errorf("validation failed: %w", err)
+	}
+	return newVersion, nil
+}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/library/encryption/kms/vault_key_rotation.go` around lines 63 - 132,
Both ForceKeyRotation and RotateKeyOnly duplicate the same 5-step flow
(getCurrentKeyVersion, rotateKey, getCurrentKeyVersion, check version increase,
optional updateMinDecryptionVersion, ValidateKeyRotation); extract that shared
flow into a single helper (e.g., performRotation(ctx context.Context,
forceMinDecryption bool) (int, error)) that calls getCurrentKeyVersion,
rotateKey, re-checks version, enforces the version-increase check, calls
updateMinDecryptionVersion only when forceMinDecryption is true, and finally
calls ValidateKeyRotation; then implement ForceKeyRotation to call
performRotation(ctx, true) and RotateKeyOnly to call performRotation(ctx, false)
to remove duplication while keeping existing helper method names
(getCurrentKeyVersion, rotateKey, updateMinDecryptionVersion,
ValidateKeyRotation).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/library/encryption/kms/vault_key_rotation.go`:
- Around line 86-94: The code updates min_decryption_version via
v.updateMinDecryptionVersion(ctx, newVersion) but then only calls
v.ValidateKeyRotation(ctx, newVersion) which checks latest_version; change the
validation to assert that min_decryption_version actually equals newVersion and
enforces decryption gating: either extend ValidateKeyRotation to also fetch and
compare the stored min_decryption_version to newVersion or add a new check after
updateMinDecryptionVersion that calls a method (e.g., v.getKeyMetadata or
v.fetchMinDecryptionVersion) and verifies the persisted min_decryption_version
== newVersion and that decryption of data encrypted with older versions fails;
update ForceKeyRotation to use this stronger validation path (referencing
updateMinDecryptionVersion, ValidateKeyRotation, and ForceKeyRotation).

---

Nitpick comments:
In `@test/library/encryption/kms/vault_key_rotation.go`:
- Around line 63-132: Both ForceKeyRotation and RotateKeyOnly duplicate the same
5-step flow (getCurrentKeyVersion, rotateKey, getCurrentKeyVersion, check
version increase, optional updateMinDecryptionVersion, ValidateKeyRotation);
extract that shared flow into a single helper (e.g., performRotation(ctx
context.Context, forceMinDecryption bool) (int, error)) that calls
getCurrentKeyVersion, rotateKey, re-checks version, enforces the
version-increase check, calls updateMinDecryptionVersion only when
forceMinDecryption is true, and finally calls ValidateKeyRotation; then
implement ForceKeyRotation to call performRotation(ctx, true) and RotateKeyOnly
to call performRotation(ctx, false) to remove duplication while keeping existing
helper method names (getCurrentKeyVersion, rotateKey,
updateMinDecryptionVersion, ValidateKeyRotation).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 257059ea-a5b6-4d2e-a6c0-dac867470c7a

📥 Commits

Reviewing files that changed from the base of the PR and between dff948b and e6d8357.

📒 Files selected for processing (1)
  • test/library/encryption/kms/vault_key_rotation.go

Comment thread test/library/encryption/kms/vault_key_rotation.go Outdated
Comment thread test/library/encryption/kms/vault_key_rotation.go Outdated
@sandeepknd sandeepknd requested a review from tjungblu May 15, 2026 14:26
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/library/encryption/kms/vault_key_rotation.go`:
- Around line 114-116: The oc exec invocations use the caller ctx which may lack
a deadline and hang; wrap ctx with a bounded fallback timeout (e.g.,
defaultTimeout := 30*time.Second) when ctx.Err()==nil and ctx has no deadline by
creating ctxWithTimeout, using context.WithTimeout(ctx, defaultTimeout),
deferring cancel(), and pass ctxWithTimeout into exec.CommandContext for the
calls that build commands using v.podName, v.namespace, v.transitKeyName (the
"vault read -format=json transit/keys/..." call) and the subsequent oc exec
invocation around lines 135-137; ensure you only create the wrapper when
context.Deadline() returns false so existing deadlines are preserved.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 77ec3172-f5ef-43ae-9d3c-ceb6ef36325c

📥 Commits

Reviewing files that changed from the base of the PR and between e6d8357 and 0532af7.

📒 Files selected for processing (1)
  • test/library/encryption/kms/vault_key_rotation.go

Comment thread test/library/encryption/kms/vault_key_rotation.go Outdated
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
test/library/encryption/kms/vault_key_rotation.go (1)

107-115: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Add a fallback timeout for oc exec to prevent hanging e2e flows.

At Line 109 and Line 130, exec.CommandContext uses the incoming context directly. If upstream passes a context without a deadline, these calls can hang indefinitely and stall tests.

Proposed patch
 import (
 	"context"
 	"encoding/json"
 	"fmt"
 	"os/exec"
+	"time"

 	"k8s.io/client-go/kubernetes"
 )
 
+const defaultOCExecTimeout = 2 * time.Minute
+
+func withDefaultTimeout(ctx context.Context) (context.Context, context.CancelFunc) {
+	if _, hasDeadline := ctx.Deadline(); hasDeadline {
+		return ctx, func() {}
+	}
+	return context.WithTimeout(ctx, defaultOCExecTimeout)
+}
+
 // GetKeyInfo returns information about the transit key including current version and configuration
 func (v *VaultKeyRotator) GetKeyInfo(ctx context.Context) (map[string]interface{}, error) {
 	// Read key information: vault read transit/keys/<key-name>
-	cmd := exec.CommandContext(ctx, "oc", "exec", v.podName, "-n", v.namespace, "--",
+	cmdCtx, cancel := withDefaultTimeout(ctx)
+	defer cancel()
+	cmd := exec.CommandContext(cmdCtx, "oc", "exec", v.podName, "-n", v.namespace, "--",
 		"vault", "read", "-format=json", fmt.Sprintf("transit/keys/%s", v.transitKeyName))
@@
 func (v *VaultKeyRotator) rotateKey(ctx context.Context) error {
@@
-	cmd := exec.CommandContext(ctx, "oc", "exec", v.podName, "-n", v.namespace, "--",
+	cmdCtx, cancel := withDefaultTimeout(ctx)
+	defer cancel()
+	cmd := exec.CommandContext(cmdCtx, "oc", "exec", v.podName, "-n", v.namespace, "--",
 		"vault", "write", "-f", fmt.Sprintf("transit/keys/%s/rotate", v.transitKeyName))

Also applies to: 127-136

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/library/encryption/kms/vault_key_rotation.go` around lines 107 - 115,
The oc exec calls in VaultKeyRotator.GetKeyInfo (and the similar
exec.CommandContext usage around lines 127-136) can hang if the incoming ctx has
no deadline; wrap the provided ctx with a child context that enforces a short
timeout (e.g., 30s) only when ctx.Deadline() returns false, use ctxWithTimeout,
defer cancel(), and pass that context into exec.CommandContext so the command is
bounded; update both GetKeyInfo and the other exec call sites (referencing
VaultKeyRotator, v.podName, v.namespace, v.transitKeyName and the other method
where exec.CommandContext is used) accordingly.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Duplicate comments:
In `@test/library/encryption/kms/vault_key_rotation.go`:
- Around line 107-115: The oc exec calls in VaultKeyRotator.GetKeyInfo (and the
similar exec.CommandContext usage around lines 127-136) can hang if the incoming
ctx has no deadline; wrap the provided ctx with a child context that enforces a
short timeout (e.g., 30s) only when ctx.Deadline() returns false, use
ctxWithTimeout, defer cancel(), and pass that context into exec.CommandContext
so the command is bounded; update both GetKeyInfo and the other exec call sites
(referencing VaultKeyRotator, v.podName, v.namespace, v.transitKeyName and the
other method where exec.CommandContext is used) accordingly.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a0de628d-6a3b-4b93-870f-82ebf25a377f

📥 Commits

Reviewing files that changed from the base of the PR and between 0532af7 and d586f5c.

📒 Files selected for processing (1)
  • test/library/encryption/kms/vault_key_rotation.go

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented May 15, 2026

@sandeepknd: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@sandeepknd sandeepknd changed the title added vault key rotation func CNTRLPLANE-3364: added vault key rotation func May 15, 2026
@openshift-ci-robot
Copy link
Copy Markdown

openshift-ci-robot commented May 15, 2026

@sandeepknd: This pull request references CNTRLPLANE-3364 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Added the vault kms key rotation function.
Currently two functions are defined :
first one only rotates the key - vault creates multiple versions after rotation; keeping the min_decryption_version same.
second one rotates the key and also updates the min_decryption_version field to the latest version resulting in only 1 key available at at time for decryption.

Let me know your thoughts if both functions are needed.

Summary by CodeRabbit

  • New Features
  • Added test utilities to trigger Vault key rotations and verify the key version increases.
  • Added ability to read Vault key metadata for validation during tests.
  • Support for default and customizable Vault target settings to simplify testing across environments; improved error reporting with captured command output on failures.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label May 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants