Skip to content

fix/opv2-jwt - #318

Open
collinol wants to merge 1 commit into
mainfrom
fix/opv2-jwt
Open

fix/opv2-jwt#318
collinol wants to merge 1 commit into
mainfrom
fix/opv2-jwt

Conversation

@collinol

@collinol collinol commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

The defaulting webhook populated oidcIssuer with https://kubernetes.default.svc.cluster.local so on EKS the iss of the projected tokens did not correspond to the jwt-issuer-map, got a 401 err

Now operator discovers real issuer from /.well-known/openid-configuration at startup, with precedence over the explicit value in the CR.

If the issuer remains unknown the CR goes not-ready with reason ServiceAccountIssuerUnknown instead of emitting an incorrect value.

The RBAC rule is added to both the generated role and the chart's ClusterRole, I believe this requires a re-release

Summary by CodeRabbit

  • New Features

    • Added automatic discovery and validation of the Kubernetes service-account token issuer.
    • Added permissions required to read the cluster’s OpenID Connect configuration.
  • Bug Fixes

    • Prevented application reconciliation when internal service authentication has no resolvable issuer, while reporting a readiness condition and retrying.
    • Avoided assigning an incorrect default issuer; issuer configuration now remains explicit.

@collinol
collinol marked this pull request as ready for review August 5, 2026 21:38
@collinol
collinol requested a review from a team as a code owner August 5, 2026 21:38
@coderabbitai

coderabbitai Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The manager discovers and stores the Kubernetes service-account issuer. RBAC permits access to the discovery endpoint. The webhook stops defaulting the issuer. Reconciliation resolves the issuer centrally and pauses when it is unavailable.

Changes

Service-account issuer discovery

Layer / File(s) Summary
Issuer discovery and API access
cmd/manager/main.go, config/rbac/role.yaml, deploy/operator/templates/wandb-operator-wandb-role.yaml, internal/controller/weightsandbiases_controller.go
Startup discovers and validates the Kubernetes service-account issuer, stores it through utils.SetServiceAccountIssuer, and logs discovery errors without stopping the manager. RBAC permits access to the OIDC discovery endpoint.
Issuer configuration and reconciliation
internal/webhook/v2/weightsandbiases_webhook.go, internal/webhook/v2/weightsandbiases_webhook_test.go, internal/controller/reconciler/reconcile_v2.go, internal/controller/reconciler/pods.go
The webhook leaves OIDCIssuer empty. Reconciliation blocks when the issuer cannot be resolved and skips JWT mappings without a resolved issuer. The defaulting test reflects the new behavior.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested reviewers: casey-coreweave, danielpanzella

Sequence Diagram(s)

sequenceDiagram
  participant ManagerStartup
  participant KubernetesAPIServer
  participant ServiceIssuerStore
  participant ControllerReconciler
  ManagerStartup->>KubernetesAPIServer: GET /.well-known/openid-configuration
  KubernetesAPIServer-->>ManagerStartup: issuer discovery response
  ManagerStartup->>ServiceIssuerStore: SetServiceAccountIssuer(validated issuer)
  ControllerReconciler->>ServiceIssuerStore: resolveInternalServiceAuthIssuer()
  alt issuer resolved
    ControllerReconciler->>ControllerReconciler: reconcile applications and JWT issuer mappings
  else issuer unavailable
    ControllerReconciler->>ControllerReconciler: update readiness and requeue after one minute
  end
Loading
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title identifies a JWT-related fix that matches the pull request’s main change, although it uses abbreviated wording.
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/opv2-jwt

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cmd/manager/main.go`:
- Around line 291-297: Update the startup issuer-discovery call around
RegisterServiceAccountIssuer to use a context.WithTimeout with an appropriate
bounded duration instead of context.Background(), and ensure the derived context
is canceled. Preserve the existing non-fatal error logging and manager startup
behavior when discovery times out or fails.

In `@internal/controller/reconciler/reconcile_v2.go`:
- Around line 527-530: Restore shared declarations for
internalServiceAuthEnabled, resolveInternalServiceAuthIssuer,
serviceAccountIssuerUnknownReason, and serviceAccountIssuerUnknownMessage, then
reference those declarations in internal/controller/reconciler/reconcile_v2.go
lines 527-530 and internal/controller/reconciler/pods.go lines 413-416. Ensure
both call sites use the same helpers and readiness constants, then run make lint
and make test.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: ff361e20-ef45-4246-bd5d-00192898ad2e

📥 Commits

Reviewing files that changed from the base of the PR and between eb3abfe and 5cd72b2.

📒 Files selected for processing (8)
  • cmd/manager/main.go
  • config/rbac/role.yaml
  • deploy/operator/templates/wandb-operator-wandb-role.yaml
  • internal/controller/reconciler/pods.go
  • internal/controller/reconciler/reconcile_v2.go
  • internal/controller/weightsandbiases_controller.go
  • internal/webhook/v2/weightsandbiases_webhook.go
  • internal/webhook/v2/weightsandbiases_webhook_test.go

Comment thread cmd/manager/main.go
Comment on lines +291 to +297
// Non-fatal: a cluster that blocks the discovery endpoint should still start.
// The reconciler surfaces a not-ready condition rather than asserting an
// issuer it never verified.
if err := RegisterServiceAccountIssuer(context.Background()); err != nil {
setupLog.Error(err, "failed to discover the cluster service-account issuer; "+
"internal service auth will not reconcile until spec.wandb.internalServiceAuth.oidcIssuer is set")
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Add a deadline to issuer discovery.

context.Background() at Line 294 has no deadline. If the API server stalls, DoRaw blocks before the manager starts. The non-fatal error path does not run.

Use context.WithTimeout for this startup lookup.

Proposed fix
+	"time"
+
-	if err := RegisterServiceAccountIssuer(context.Background()); err != nil {
+	issuerCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	if err := RegisterServiceAccountIssuer(issuerCtx); err != nil {
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
// Non-fatal: a cluster that blocks the discovery endpoint should still start.
// The reconciler surfaces a not-ready condition rather than asserting an
// issuer it never verified.
if err := RegisterServiceAccountIssuer(context.Background()); err != nil {
setupLog.Error(err, "failed to discover the cluster service-account issuer; "+
"internal service auth will not reconcile until spec.wandb.internalServiceAuth.oidcIssuer is set")
}
// Non-fatal: a cluster that blocks the discovery endpoint should still start.
// The reconciler surfaces a not-ready condition rather than asserting an
// issuer it never verified.
issuerCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if err := RegisterServiceAccountIssuer(issuerCtx); err != nil {
setupLog.Error(err, "failed to discover the cluster service-account issuer; "+
"internal service auth will not reconcile until spec.wandb.internalServiceAuth.oidcIssuer is set")
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmd/manager/main.go` around lines 291 - 297, Update the startup
issuer-discovery call around RegisterServiceAccountIssuer to use a
context.WithTimeout with an appropriate bounded duration instead of
context.Background(), and ensure the derived context is canceled. Preserve the
existing non-fatal error logging and manager startup behavior when discovery
times out or fails.

Source: Coding guidelines

Comment on lines +527 to +530
if internalServiceAuthEnabled(wandb) && resolveInternalServiceAuthIssuer(wandb) == "" {
logger.Info("Cluster service-account issuer unknown; not reconciling applications")
if err := updateReadyStatus(ctx, client, wandb, statusBefore, false,
serviceAccountIssuerUnknownReason, serviceAccountIssuerUnknownMessage); err != nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win

Restore the shared issuer-resolution declarations.

internalServiceAuthEnabled, resolveInternalServiceAuthIssuer, serviceAccountIssuerUnknownReason, and serviceAccountIssuerUnknownMessage are undefined in package reconciler. The golangci-lint typecheck phase fails before the operator can build.

  • internal/controller/reconciler/reconcile_v2.go#L527-L530: add or reference the shared helper functions and readiness constants.
  • internal/controller/reconciler/pods.go#L413-L416: use the same declared helper functions.

After the fix, run make lint and make test. As per coding guidelines, “Run both make lint and make test before considering a task complete.”

📍 Affects 2 files
  • internal/controller/reconciler/reconcile_v2.go#L527-L530 (this comment)
  • internal/controller/reconciler/pods.go#L413-L416
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/controller/reconciler/reconcile_v2.go` around lines 527 - 530,
Restore shared declarations for internalServiceAuthEnabled,
resolveInternalServiceAuthIssuer, serviceAccountIssuerUnknownReason, and
serviceAccountIssuerUnknownMessage, then reference those declarations in
internal/controller/reconciler/reconcile_v2.go lines 527-530 and
internal/controller/reconciler/pods.go lines 413-416. Ensure both call sites use
the same helpers and readiness constants, then run make lint and make test.

Sources: Coding guidelines, Linters/SAST tools

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant