Skip to content

fix(agents-login): drop the broken K8s lease that failed credential finalize#736

Merged
ExtraToast merged 1 commit into
mainfrom
fix/agents-login-drop-broken-lease
Jun 26, 2026
Merged

fix(agents-login): drop the broken K8s lease that failed credential finalize#736
ExtraToast merged 1 commit into
mainfrom
fix/agents-login-drop-broken-lease

Conversation

@ExtraToast

Copy link
Copy Markdown
Owner

What was broken

Submitting the authorization code on the credentials page minted the Claude OAuth token but the login then failed with fetch failed, and the page stayed "Not connected — No credentials stored yet".

Worker logs showed the token was captured, then finalize died immediately:

login session credential capture completed  dataKeys:[…,oauth_token,claude_json]
login session finalize failed               error:"fetch failed"
finalizing → failed                          reason:"fetch failed"

Note the absence of any posting credentials to agents-api line — the failure happened before the persist call.

Root cause

finalize() wrapped the agents-api persist in lease.withLock(…), where lease was a K8sLeaseLock. Acquiring the lock issues a GET to the kube-apiserver over HTTPS (https://$KUBERNETES_SERVICE_HOST/apis/coordination.k8s.io/v1/…) using the global fetch. Nothing wired the in-cluster CA into that request — DEFAULT_CA/caCertPath exist in lease.ts but are never applied, and NODE_EXTRA_CA_CERTS is not set on the deployment. undici therefore could not verify the k3s API-server certificate and threw TypeError: fetch failed in tryAcquire(), aborting finalize before the token reached agents-api.

Fix

The lease only ever guarded the old Vault CAS write. Credentials now persist as a per-(user, provider) Postgres upsert through agents-api, and the worker runs as a single replica with at most one session per provider, so concurrent writers cannot collide. The distributed lock guards nothing and was the sole broken apiserver call, so the worker now wires NoopLeaseLock and posts the captured credential directly.

K8sLeaseLock and its now-unused coordination.k8s.io RBAC/NetworkPolicy egress are left in place for a separate follow-up rather than expanding this fix.

Validation

tsc --noEmit clean; vitest run green (91/91). The session tests already drive finalize through NoopLeaseLock and assert it posts to agents-api, so they now match production wiring.

Credential finalize captured the Claude OAuth token, then wrapped the
agents-api persist call in a coordination.k8s.io lease acquired via the
worker's K8sLeaseLock. That lock reached the kube-apiserver over HTTPS
using the global fetch with no cluster-CA trust, so the lease GET threw
`fetch failed`; finalize aborted before the captured token was posted to
agents-api. The login surfaced "fetch failed" and the credentials page
stayed "Not connected" even though the token had been minted.

The lease only ever guarded the old Vault CAS write. Credentials now
persist as a per-(user, provider) Postgres upsert through agents-api, and
the worker runs as a single replica with at most one session per
provider, so concurrent writers cannot collide. The worker now wires
NoopLeaseLock and posts the captured credential directly.
@ExtraToast ExtraToast added the bug Something isn't working label Jun 26, 2026
@ExtraToast ExtraToast self-assigned this Jun 26, 2026
@ExtraToast ExtraToast merged commit 2153a1f into main Jun 26, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant