[sig-cluster-lifecycle] CSR approver test fails with 401 Unauthorized on ROSA STS clusters in 4.22+

[dustman9000]

Bug Report

The test [sig-cluster-lifecycle] CSRs from machines that are not recognized by the cloud provider are not approved fails with 401 Unauthorized on ROSA Classic STS clusters starting in OCP 4.22. The test passes on 4.21.

Affected Jobs

• periodic-ci-openshift-release-main-nightly-4.22-e2e-rosa-sts-ovn
• periodic-ci-openshift-release-main-nightly-4.23-e2e-rosa-sts-ovn
• periodic-ci-openshift-release-main-nightly-5.0-e2e-rosa-sts-ovn

What the Test Does

The test (in test/extended/csrapprover/csrapprover.go, lines 93-155):

1. Creates a bogus CSR for system:node:hacking-node.ec2.internal
2. Gets a bearer token for the node-bootstrapper SA in openshift-machine-config-operator via TokenRequest API
3. Uses that token to create a CertificateSigningRequest
4. Expects either success (then verifies it was not approved) or timeout

What Fails

Step 3 returns 401 Unauthorized at csrapprover.go:133. The node-bootstrapper SA token is rejected by the kube-apiserver on ROSA STS clusters in 4.22+.

Expected Behavior

The test should handle 401 Unauthorized as a valid "not approved" outcome, since a rejected token means the CSR can never be approved. The current code only expects success or timeout.

Suggested Fix

In csrapprover.go:133, accept 401 Unauthorized as a valid outcome alongside the existing success and timeout paths. The 401 indicates stricter SA token validation on STS clusters in 4.22, which is arguably correct behavior (the token is rejected before the CSR is even created).

Environment

• Platform: ROSA Classic STS (AWS)
• OCP versions affected: 4.22+
• OCP versions passing: 4.21
• The test does not exist in the 4.19 openshift-tests binary

/area test
/sig cluster-lifecycle

[openshift-ci]

@dustman9000: The label(s) area/test cannot be applied, because the repository doesn't have them.

Details

In response to this:

Bug Report

The test [sig-cluster-lifecycle] CSRs from machines that are not recognized by the cloud provider are not approved fails with 401 Unauthorized on ROSA Classic STS clusters starting in OCP 4.22. The test passes on 4.21.

Affected Jobs

• periodic-ci-openshift-release-main-nightly-4.22-e2e-rosa-sts-ovn
• periodic-ci-openshift-release-main-nightly-4.23-e2e-rosa-sts-ovn
• periodic-ci-openshift-release-main-nightly-5.0-e2e-rosa-sts-ovn

What the Test Does

The test (in test/extended/csrapprover/csrapprover.go, lines 93-155):

1. Creates a bogus CSR for system:node:hacking-node.ec2.internal
2. Gets a bearer token for the node-bootstrapper SA in openshift-machine-config-operator via TokenRequest API
3. Uses that token to create a CertificateSigningRequest
4. Expects either success (then verifies it was not approved) or timeout

What Fails

Step 3 returns 401 Unauthorized at csrapprover.go:133. The node-bootstrapper SA token is rejected by the kube-apiserver on ROSA STS clusters in 4.22+.

Expected Behavior

The test should handle 401 Unauthorized as a valid "not approved" outcome, since a rejected token means the CSR can never be approved. The current code only expects success or timeout.

Suggested Fix

In csrapprover.go:133, accept 401 Unauthorized as a valid outcome alongside the existing success and timeout paths. The 401 indicates stricter SA token validation on STS clusters in 4.22, which is arguably correct behavior (the token is rejected before the CSR is even created).

Environment

• Platform: ROSA Classic STS (AWS)
• OCP versions affected: 4.22+
• OCP versions passing: 4.21
• The test does not exist in the 4.19 openshift-tests binary

/area test
/sig cluster-lifecycle

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[dustman9000]

Update: Root cause narrowed to library-go certificate change

Further investigation shows the CSR test failure started between two specific 4.22 nightlies on May 11:

• Passing: 4.22.0-0.nightly-2026-05-11-031617
• Failing: 4.22.0-0.nightly-2026-05-11-145855

The CSR test also passes on 4.21 in the same timeframe, so this is not a staging environment issue.

The payload diff between these two nightlies includes a library-go bump in cluster-kube-apiserver-operator:

• PR: OCPBUGS-84536: Bump library-go cluster-kube-apiserver-operator#2131 (OCPBUGS-84536)
• Which bumps: OCPBUGS-84536: Update MakeServerCert to use time.Duration instead of days library-go#2183
• Change: MakeServerCert() updated from days to time.Duration, affecting certificate creation in AddDefaultRotationToConfig()

The 401 Unauthorized occurs at the authentication layer (no audit log entries for the node-bootstrapper SA token request), which is consistent with a certificate/TLS-level rejection. The TokenRequest API creates a token that the kube-apiserver then fails to validate, likely due to the serving cert change affecting the token signing/validation chain.

The kube-apiserver does accept https://kubernetes.default.svc as a valid audience (confirmed via api-audiences config), so the audience is not the issue.