OCPBUGS-78557: Re-Apply Fix admin context race condition in parallel tests

[kuiwang02]

/cc @jianzhangbjz @bandrade @Xia-Zhao-rh

Fix admin context race condition in parallel tests

This PR fixes an intermittent test failure where AsAdmin() occasionally uses the wrong service account instead of cluster-admin credentials, caused by concurrent tests sharing the same kubeconfig file.

Background

PR #1256 introduced a fix using double insurance (per-test temp kubeconfig files + explicit --context=admin flag), but was reverted by PR #1258 due to log pollution in images command.

Root cause of revert: The fix initialized admin context during test tree construction (in NewCLI()), which caused logs to appear when running images commands that only build the test tree without executing tests.

Changes in This PR

This PR reintroduces the race condition fix with lazy-loading pattern to prevent log pollution:

1. Lazy initialization: admContextName is initialized on first AsAdmin() call (during test execution), not in NewCLI() (during tree construction)
2. Verbose parameter: Added verbose bool parameter to duplicateFileToTempOrEmpty() to control logging
   • Constructor calls: verbose=false (silent during tree construction)
   • Execution calls: verbose=true (logs during actual test runs)

Assisted-By: Claude Code

[coderabbitai]

Walkthrough

Extends CLI state with admin context caching and temporary kubeconfig path handling for concurrent test execution. Adds safe, error-returning file-operation helpers for initialization phases, enabling error propagation instead of Gomega assertions. Refactors existing file utilities to delegate to new safe variants.

Changes

Cohort / File(s)	Summary
Admin Context Caching tests-extension/test/qe/util/client.go	Adds global admin context cache and per-CLI lazy-loading. Extends CLI with tempAdmConfPath, admContextName, and isUseAdminCxt fields. Updates AsAdmin(), SetAdminKubeconf(), and AdminConfig() to use temporary per-test admin kubeconfig and cached context names. Adds getAdminContextName() and getAdminContextNameCached() helpers to locate and cache admin context in kubeconfig.
Safe File Operation Helpers tests-extension/test/qe/util/tools.go	Introduces duplicateFileToPathSafe(), duplicateFileToTempSafe(), and duplicateFileToTempOrEmpty() for error-based file operations. Refactors existing DuplicateFileToPath() and DuplicateFileToTemp() to delegate to safe variants, enabling initialization-phase error propagation while maintaining test-phase Gomega assertions.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~22 minutes

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

📝 Coding Plan

• Generate coding plan for human review comments

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Tip

You can customize the high-level summary generated by CodeRabbit.

Configure the reviews.high_level_summary_instructions setting to provide custom instructions for generating the high-level summary.

[kuiwang02]

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ipi-ovn-ipv4

[openshift-ci]

@kuiwang02: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ipi-ovn-ipv4

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/01e7d5f0-2268-11f1-8c88-608126d8fa68-0

[kuiwang02]

/retitle OCPBUGS-78557: Re-Apply Fix admin context race condition in parallel tests

[openshift-ci-robot]

@kuiwang02: This pull request references Jira Issue OCPBUGS-78557, which is invalid:

• expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

/cc @jianzhangbjz @bandrade @Xia-Zhao-rh

Fix admin context race condition in parallel tests

This PR fixes an intermittent test failure where AsAdmin() occasionally uses the wrong service account instead of cluster-admin credentials, caused by concurrent tests sharing the same kubeconfig file.

Background

PR #1256 introduced a fix using double insurance (per-test temp kubeconfig files + explicit --context=admin flag), but was reverted by PR #1258 due to log pollution in openshift-tests images command.

Root cause of revert: The fix initialized admin context during test tree construction (in NewCLI()), which caused logs to appear when running images commands that only build the test tree without executing tests.

Changes in This PR

This PR reintroduces the race condition fix with lazy-loading pattern to prevent log pollution:

1. Lazy initialization: admContextName is initialized on first AsAdmin() call (during test execution), not in NewCLI() (during tree construction)
2. Verbose parameter: Added verbose bool parameter to duplicateFileToTempOrEmpty() to control logging

• Constructor calls: verbose=false (silent during tree construction)
• Execution calls: verbose=true (logs during actual test runs)

Assisted-By: Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[kuiwang02]

/jira refresh

[openshift-ci-robot]

@kuiwang02: This pull request references Jira Issue OCPBUGS-78557, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @kuiwang02

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

@openshift-ci-robot: GitHub didn't allow me to request PR reviews from the following users: kuiwang02.

Note that only openshift members and repo collaborators can review this PR, and authors cannot review their own PRs.

Details

In response to this:

@kuiwang02: This pull request references Jira Issue OCPBUGS-78557, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @kuiwang02

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[coderabbitai]

🧹 Nitpick comments (1)

tests-extension/test/qe/util/client.go (1)

1230-1240: Global cache assumes uniform kubeconfig across tests.

The sync.Once caches the admin context name from the first kubeconfig file encountered. If tests use different kubeconfig files with different admin context names, only the first will be cached. This is likely acceptable for this test framework's usage patterns, but worth noting.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests-extension/test/qe/util/client.go` around lines 1230 - 1240, The current
getAdminContextNameCached uses a single
globalAdmContextOnce/globalAdmContextName so the first kubeConfigFile wins;
change it to cache per-file instead: replace the single
globalAdmContextOnce/globalAdmContextName with a map (or sync.Map) keyed by
kubeConfigFile and protected by a mutex, then update getAdminContextNameCached
to check the map for kubeConfigFile, call getAdminContextName(kubeConfigFile)
only if missing, store the result in the map, and log using e2e.Logf; keep
function name getAdminContextNameCached and use the new cache map (or sync.Map)
and mutex symbols to locate and modify related code.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@tests-extension/test/qe/util/client.go`:
- Around line 1230-1240: The current getAdminContextNameCached uses a single
globalAdmContextOnce/globalAdmContextName so the first kubeConfigFile wins;
change it to cache per-file instead: replace the single
globalAdmContextOnce/globalAdmContextName with a map (or sync.Map) keyed by
kubeConfigFile and protected by a mutex, then update getAdminContextNameCached
to check the map for kubeConfigFile, call getAdminContextName(kubeConfigFile)
only if missing, store the result in the map, and log using e2e.Logf; keep
function name getAdminContextNameCached and use the new cache map (or sync.Map)
and mutex symbols to locate and modify related code.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: dd631288-c980-4abe-bdf5-acc601bd7ba5

📥 Commits

Reviewing files that changed from the base of the PR and between 64e80c7 and a3a9389.

📒 Files selected for processing (2)

• tests-extension/test/qe/util/client.go
• tests-extension/test/qe/util/tools.go

[kuiwang02]

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ipi-ovn-ipv4

[openshift-ci]

@kuiwang02: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ipi-ovn-ipv4

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/7d274ce0-227d-11f1-8df9-129e606b316c-0

[kuiwang02]

/payload-job periodic-ci-openshift-operator-framework-olm-release-4.22-periodics-e2e-aws-ovn-proxy-extended-f2

[openshift-ci]

@kuiwang02: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-operator-framework-olm-release-4.22-periodics-e2e-aws-ovn-proxy-extended-f2

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/97721ad0-227d-11f1-870b-8fd11430f706-0

[kuiwang02]

/payload-job periodic-ci-openshift-operator-framework-olm-release-4.22-periodics-e2e-aws-ovn-fips-techpreview-extended-f1

[openshift-ci]

@kuiwang02: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-operator-framework-olm-release-4.22-periodics-e2e-aws-ovn-fips-techpreview-extended-f1

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/9dbd95e0-227d-11f1-8930-28b77953248d-0

[kuiwang02]

/test e2e-gcp-ovn

[tmshort]

/approve

[tmshort]

Does this mean that #1257 should be updated as well?

[kuiwang02]

#1257

@tmshort yes, thanks for remindering!. I will handle it.

[kuiwang02]

/verified by @kuiwang02

[openshift-ci-robot]

@kuiwang02: This PR has been marked as verified by @kuiwang02.

Details

In response to this:

/verified by @kuiwang02

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[jianzhangbjz]

/approve
/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jianzhangbjz, kuiwang02, tmshort

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• DOWNSTREAM_OWNERS [jianzhangbjz,tmshort]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@kuiwang02: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci-robot]

@kuiwang02: Jira Issue Verification Checks: Jira Issue OCPBUGS-78557
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-78557 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

/cc @jianzhangbjz @bandrade @Xia-Zhao-rh

Fix admin context race condition in parallel tests

This PR fixes an intermittent test failure where AsAdmin() occasionally uses the wrong service account instead of cluster-admin credentials, caused by concurrent tests sharing the same kubeconfig file.

Background

PR #1256 introduced a fix using double insurance (per-test temp kubeconfig files + explicit --context=admin flag), but was reverted by PR #1258 due to log pollution in images command.

Root cause of revert: The fix initialized admin context during test tree construction (in NewCLI()), which caused logs to appear when running images commands that only build the test tree without executing tests.

Changes in This PR

This PR reintroduces the race condition fix with lazy-loading pattern to prevent log pollution:

1. Lazy initialization: admContextName is initialized on first AsAdmin() call (during test execution), not in NewCLI() (during tree construction)
2. Verbose parameter: Added verbose bool parameter to duplicateFileToTempOrEmpty() to control logging

• Constructor calls: verbose=false (silent during tree construction)
• Execution calls: verbose=true (logs during actual test runs)

Assisted-By: Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[kuiwang02]

/cherry-pick release-4.21

[openshift-cherrypick-robot]

@kuiwang02: new pull request created: #1261

Details

In response to this:

/cherry-pick release-4.21

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.