Skip to content

Enhancement: etcd data re-encryption for key rotation in HyperShift#1969

Open
muraee wants to merge 1 commit intoopenshift:masterfrom
muraee:hypershift-etcd-reencryption
Open

Enhancement: etcd data re-encryption for key rotation in HyperShift#1969
muraee wants to merge 1 commit intoopenshift:masterfrom
muraee:hypershift-etcd-reencryption

Conversation

@muraee
Copy link
Copy Markdown

@muraee muraee commented Apr 9, 2026

Summary

  • Add enhancement proposal for etcd data re-encryption after encryption key rotation in HyperShift
  • Introduces a new HCCO controller that leverages KubeStorageVersionMigrator from library-go to create StorageVersionMigration CRs in the guest cluster, transparently re-encrypting all encrypted resources with the active key
  • Adds EtcdDataEncryptionUpToDate condition on HCP/HostedCluster for progress tracking
  • Guards against premature backup key removal
  • Supports all encryption types (Azure KMS, AWS KMS, IBM Cloud KMS, AESCBC)

Tracks: OCPSTRAT-2527, OCPSTRAT-2540
Related: ARO-21568, ARO-21456

Test plan

  • Unit tests for key fingerprint computation and controller reconciliation logic
  • Integration tests for StorageVersionMigration CR lifecycle
  • E2E tests for Azure KMS and AESCBC key rotation with re-encryption

🤖 Generated with Claude Code

@openshift-ci openshift-ci bot requested review from csrwng and sjenning April 9, 2026 16:17
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Apr 9, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign sjenning for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@muraee muraee force-pushed the hypershift-etcd-reencryption branch from ae1dfec to eabd02a Compare April 10, 2026 09:33
### Non-Goals

1. Management of the creation and renewal of encryption keys --
keys are managed externally (by the ARO RP or user).
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe the ARO HCP specific language should be dropped here since this works on other platforms?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. The motivation for calling out ARO-HCP specifically is that it's the primary driver for this work (the S360 compliance requirement is what makes re-encryption mandatory rather than nice-to-have). That said, the solution itself is fully generic and platform-agnostic.

I can rephrase to lead with the generic value ("any customer relying on key rotation as a security control") and mention ARO-HCP as the motivating use case rather than making it sound ARO-specific. Happy to update this if you'd like.


AI-assisted response via Claude Code

@muraee muraee force-pushed the hypershift-etcd-reencryption branch from eabd02a to 7b4c875 Compare April 10, 2026 14:39
Copy link
Copy Markdown
Member

@ardaguclu ardaguclu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I specifically focused on Why KubeStorageVersionMigrator Instead of MigrationController section. It looks good to me. I dropped a comment more about agreement instead of any objection.


### Implementation Details/Notes/Constraints

#### Why KubeStorageVersionMigrator Instead of MigrationController
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not using MigrationController is better idea because it is tightly coupled with other encryption controllers and I'm not sure that it works properly without them.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the review!

muraee added a commit to muraee/hypershift that referenced this pull request Apr 13, 2026
Add a re-encryption controller in the HCCO that triggers
StorageVersionMigration after an encryption key rotation, ensuring
all existing etcd data is re-encrypted with the new active key.

Components:
- API: EtcdDataEncryptionUpToDate condition type and reasons
- CPO: key fingerprint computation and rekey-needed annotation
  on kas-secret-encryption-config secret
- HCCO: new reencryption controller using library-go's
  KubeStorageVersionMigrator to drive StorageVersionMigration CRs
- HyperShift Operator: condition bubble-up from HCP to HostedCluster

Ref: OCPSTRAT-2527, OCPSTRAT-2540
Enhancement: openshift/enhancements#1969

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
muraee added a commit to muraee/hypershift that referenced this pull request Apr 13, 2026
Add a re-encryption controller in the HCCO that triggers
StorageVersionMigration after an encryption key rotation, ensuring
all existing etcd data is re-encrypted with the new active key.

Components:
- API: EtcdDataEncryptionUpToDate condition type and reasons
- CPO: key fingerprint computation and rekey-needed annotation
  on kas-secret-encryption-config secret
- HCCO: new reencryption controller using library-go's
  KubeStorageVersionMigrator to drive StorageVersionMigration CRs
- HyperShift Operator: condition bubble-up from HCP to HostedCluster

Ref: OCPSTRAT-2527, OCPSTRAT-2540
Enhancement: openshift/enhancements#1969

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
muraee added a commit to muraee/hypershift that referenced this pull request Apr 13, 2026
Add a re-encryption controller in the HCCO that triggers
StorageVersionMigration after an encryption key rotation, ensuring
all existing etcd data is re-encrypted with the new active key.

Components:
- API: EtcdDataEncryptionUpToDate condition type and reasons
- CPO: key fingerprint computation and rekey-needed annotation
  on kas-secret-encryption-config secret
- HCCO: new reencryption controller using library-go's
  KubeStorageVersionMigrator to drive StorageVersionMigration CRs
- HyperShift Operator: condition bubble-up from HCP to HostedCluster

Ref: OCPSTRAT-2527, OCPSTRAT-2540
Enhancement: openshift/enhancements#1969

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
muraee added a commit to muraee/hypershift that referenced this pull request Apr 13, 2026
Add a re-encryption controller in the HCCO that triggers
StorageVersionMigration after an encryption key rotation, ensuring
all existing etcd data is re-encrypted with the new active key.

Components:
- API: EtcdDataEncryptionUpToDate condition type and reasons
- CPO: key fingerprint computation and rekey-needed annotation
  on kas-secret-encryption-config secret
- HCCO: new reencryption controller using library-go's
  KubeStorageVersionMigrator to drive StorageVersionMigration CRs
- HyperShift Operator: condition bubble-up from HCP to HostedCluster

Ref: OCPSTRAT-2527, OCPSTRAT-2540
Enhancement: openshift/enhancements#1969

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
muraee added a commit to muraee/hypershift that referenced this pull request Apr 14, 2026
Add a re-encryption controller in the HCCO that triggers
StorageVersionMigration after an encryption key rotation, ensuring
all existing etcd data is re-encrypted with the new active key.

Components:
- API: EtcdDataEncryptionUpToDate condition type and reasons
- CPO: key fingerprint computation and rekey-needed annotation
  on kas-secret-encryption-config secret
- HCCO: new reencryption controller using library-go's
  KubeStorageVersionMigrator to drive StorageVersionMigration CRs
- HyperShift Operator: condition bubble-up from HCP to HostedCluster

Ref: OCPSTRAT-2527, OCPSTRAT-2540
Enhancement: openshift/enhancements#1969

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@muraee muraee force-pushed the hypershift-etcd-reencryption branch from 7b4c875 to 82ddb23 Compare April 14, 2026 10:12
muraee added a commit to muraee/hypershift that referenced this pull request Apr 14, 2026
Add a re-encryption controller in the HCCO that triggers
StorageVersionMigration after an encryption key rotation, ensuring
all existing etcd data is re-encrypted with the new active key.

Components:
- API: EtcdDataEncryptionUpToDate condition type and reasons
- CPO: key fingerprint computation and rekey-needed annotation
  on kas-secret-encryption-config secret
- HCCO: new reencryption controller using library-go's
  KubeStorageVersionMigrator to drive StorageVersionMigration CRs
- HyperShift Operator: condition bubble-up from HCP to HostedCluster

Ref: OCPSTRAT-2527, OCPSTRAT-2540
Enhancement: openshift/enhancements#1969

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Apr 14, 2026

@muraee: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

muraee added a commit to muraee/hypershift that referenced this pull request Apr 14, 2026
Add a re-encryption controller in the HCCO that triggers
StorageVersionMigration after an encryption key rotation, ensuring
all existing etcd data is re-encrypted with the new active key.

Components:
- API: EtcdDataEncryptionUpToDate condition type and reasons
- CPO: key fingerprint computation and rekey-needed annotation
  on kas-secret-encryption-config secret
- HCCO: new reencryption controller using library-go's
  KubeStorageVersionMigrator to drive StorageVersionMigration CRs
- HyperShift Operator: condition bubble-up from HCP to HostedCluster

Ref: OCPSTRAT-2527, OCPSTRAT-2540
Enhancement: openshift/enhancements#1969

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants