fix(memfs): dynamic git credential helper for memory repos - #3668
fix(memfs): dynamic git credential helper for memory repos#3668cpacker wants to merge 8 commits into
Conversation
|
Pushed fixes for the external design review's blockers (commit 1527e4e): Finding 1 (refresh race) — fixed. Finding 2 (durable persistence / exit truncation) — fixed within limits. Tokens are flushed before the lock releases and the stored key is read back to surface silent persistence failures; the helper awaits its stdout write before Finding 7 (host matching) — hardened. Exact protocol + host[:port] match; an http remote at the Letta hostname no longer receives the token. Finding 4 (PATH coverage) — confirmed, partially open. Finding 5 (measure built artifact) — open. Numbers so far are dev-source via Finding 3 needed no change (it argues for this design). Finding 6's wording fix: this PR claims no repo-local credential copies — the keychain-less file fallback in |
|
Re-review blockers addressed in 8ebff23: 1 (waiter doesn't observe winner) — fixed. The under-lock re-read no longer goes through 2 (verification can lie) — fixed within stated limits. Read-back now uses the same strict cache-bypassing snapshot and verifies both tokens, so the partial-write scenario (new access, old refresh) throws instead of passing. Non-strict read-backs (keychain unavailable, runtime scope, read timeout) are accepted and explicitly marked unverifiable — that's the honest floor without refactoring settings-manager's swallow-errors persistence, which I'd still rather do as its own change. 3 (deadline orphans lock 90s) — fixed by bounding holds rather than plumbing cancellation. Every operation under the lock is now individually bounded: refresh fetch aborts at 15s ( On CI: the Linux x64 failure is Still open, unchanged: PATH launch-surface matrix (finding 4) and built-artifact benchmarks (finding 5). |
|
Final scoping pass per maintainer direction (8e68abe) — the unified-credential-storage branch is parked; this PR is the complete auth story for the git helper. One refresh lane. Established lock. proper-lockfile replaces the hand-rolled stale-lock algorithm for the refresh lane. Live holders keep the lock fresh via mtime touch; staleness only reaps dead holders. Fail-closed durable reads. Multi-process proof (ported from the parked branch's c23c31a3): four real bun subprocesses, one shared store — exactly 1 refresh locked, >1 in the barrier-controlled unlocked control, dead-holder lock reaped. Real git integration: Built-artifact latency (macOS arm64, Launch surfaces, stated precisely: production install → Desktop proxy mode: no changes anywhere in this pass. |
Carved out of #3653 so the auth approach can be reviewed and reverted independently of the shared-memory subcommand. Git accumulates credential helpers across config scopes and asks each in order; on macOS, Xcode's system gitconfig registers osxkeychain, which can hold a stale Letta token and answer before the repo-local helper (observed as HTTP 500s on agent-run `git pull` in shared memory mounts, LET-10545). Write an empty helper entry before ours to reset the inherited list, scoped to the Letta remote URL only. This matches what the harness's own git invocations have always done via `-c credential.helper=` in buildGitAuthArgs. Also makes the stale mount-path collision error actionable and exports cloneRepositoryMount for tests. Known issue (do not merge yet): the two new credential-reset tests assume the unix inline helper and fail on Windows CI; they need to be skipped or ported to the .cmd helper path. Open design question from review: whether to keep the persisted-token approach at all, vs injecting auth into agent shells (getShellEnv) like the harness does per-invocation, which would eliminate the on-disk token. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replace the static token persisted in memory repos' .git/config (and the Windows .cmd token file) with `letta git-credential` — a dynamic helper following the `gh auth git-credential` pattern. The repo config now holds only a command reference: [credential "https://api.letta.com"] helper = ; resets inherited helpers (osxkeychain) helper = !letta git-credential ; resolves harness auth per operation Fixes both defects from LET-10545's plain-git failure mode: the keychain can no longer answer first with a stale token (reset entry, kept from the previous commit), and rotation can no longer strand a stale token on disk (token resolved fresh via getClient — env key, keychain OAuth, single- flight refresh — on every git network operation). store/erase are no-ops, so nothing writes our token back into the keychain either. Latency: the helper runs on every push/pull, so standalone-entry.ts dispatches `git-credential` before importing the main CLI graph and the subcommand module has no static imports. Measured ~50-70ms end to end vs ~1.1s through the full graph. Also drops the platform split that broke Windows CI: `!` helpers run under git's bundled sh everywhere, and the rewritten tests use argv-array git invocations with no chmod'd scripts. Legacy .cmd helper files are removed on the next configure pass; existing repos self-heal on pull/sync. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…lper Addresses the two blockers from external design review of the dynamic credential helper, plus a host-matching hardening: 1. Cross-process refresh race: the server rotates the refresh token on every refresh (refresh_token_mode: "new"), and the existing single-flight guard is per-process — concurrent letta processes (CLI sessions, listeners, git-spawned helper invocations) could both burn the same refresh token and race their keychain writes, with the loser durably persisting an invalidated token (= logout). getClient()'s refresh block now runs under a file lock (~/.letta/oauth-refresh.lock, via the existing withFileLock util); waiters re-read the stored tokens after acquiring the lock and reuse the winner's result instead of refreshing again. Since both git delivery paths (harness extraHeader and the credential helper) funnel through getClient(), the lock sits below both — one resolution authority, two thin delivery adapters. 2. Durable persistence before exit: the rotated tokens are flushed before the lock releases, and the stored key is read back to surface silent persistence failures (best-effort when no keychain is available). The helper also awaits its stdout write, so the process.exit() in standalone-entry cannot truncate the credential handed to git. 3. Host matching now requires exact protocol + host[:port]: a plaintext-http remote pointed at the Letta hostname no longer receives the token. Fast path unaffected: still ~50ms end to end via standalone-entry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nded Closes the three blockers from re-review of 1527e4e: 1. Waiter-reuses-winner now actually observes the winner. The under-lock re-read previously used getSettingsWithSecureTokens(), whose expiry is this process's in-memory copy and whose keychain read is skipped inside runtime scopes — so a fresh helper re-rotated unnecessarily and a long-running harness could burn an already-invalidated refresh token. New readPersistedAuthTokens() (src/auth/persisted-tokens.ts) reads the settings file from disk and the keychain directly, bypassing every in-process cache, and reports whether the read was strict. The refresh also now prefers the persisted refresh token over the caller's copy. 2. Persistence verification is no longer cache-backed. The post-flush read-back uses the same strict snapshot and verifies BOTH tokens — a partial keychain write (new access token, old refresh token) previously passed the access-only check and stranded auth on the next refresh. Non-strict read-backs (no keychain / runtime scope) are accepted and documented as unverifiable. 3. Lock holds are bounded so a deadline can no longer orphan the lock for 90s: the refresh fetch aborts at 15s (AbortSignal.timeout), keychain snapshot reads soft-cap at 5s, lock acquisition times out at 20s, and the lock's stale reap is 30s. The helper deadline rises to 60s — above the sum of all internal bounds — making it a true backstop instead of something that fires mid-lock. refreshTokensUnderCrossProcessLock is now exported with injectable deps; tests cover waiter-reuse (zero refreshes), persisted-refresh-token preference, partial-persist detection, non-strict acceptance, lock release on failure, and two contenders yielding exactly one refresh. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…lane Final scoping pass on the credential-helper PR: - refreshTokensCoordinated moves to @/auth/oauth-refresh (the listener layer cannot import backend/api/client) and now backs every path that spends a refresh token: getClient(), the WebSocket listener, the startup refresh in index.ts, and the ChatGPT usage service — the review asked for three; the fourth (usage service) was doing the same uncoordinated rotation and is routed too. One resolution authority, thin delivery adapters above it. - The lock is proper-lockfile instead of the hand-rolled stale-lock algorithm: a live holder keeps the lock fresh via mtime touch, so staleness only ever reaps dead holders. Custom file-lock.ts remains for its other consumer (reflection transcripts). - Durable reads fail closed. readPersistedAuthTokens now distinguishes its sources: "keychain" (authoritative), "file" (keychain genuinely unavailable — the settings file IS durable storage, used and verified), and "runtime-scope" (reads skipped; unverifiable). A keychain that is available but errors or times out mid-read throws KeychainReadError instead of degrading — no rotation on possibly-stale data. - Waiter-reuses-winner gains the rotation signal: a persisted refresh token that differs from the caller's means a peer rotated, so the winner's access token is adopted even when expiry alone is inconclusive. - Multi-process proof (ported from the parked coordination branch): four real bun subprocesses against one shared store — exactly one refresh with the shared lock, >1 in the barrier-controlled unlocked control, and an abandoned lock from a dead holder is reaped. - Real-git integration test: `git credential fill` in a mount configured by cloneRepositoryMount, with a hostile inherited helper and a fake `letta` on PATH — proves the reset entry silences the inherited helper and the dynamic helper's answer wins, on every platform (extensionless sh script; git runs helpers under its bundled sh). Built-artifact latency (bun letta.js, macOS arm64): cold 0.9s, warm ~310ms per credential fill — bundle parse dominates; dev-source fast path is ~50ms. A separately-compiled tiny helper entrypoint can recover the difference if 310ms is deemed too slow. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
8e68abe to
2e3c238
Compare
client-soft-fail and the two listener auth suites neutered persistence (no-op updateSettings) or omitted a snapshot source, so the coordinated refresh's durable read-back correctly rejected them as failed persists — which is the exact behavior the read-back exists to catch, but here it was the test harness, not the product, failing to persist. - client-soft-fail scenarios run under LETTA_SKIP_KEYCHAIN_CHECK so the durable snapshot is file-backed and deterministic on every platform (Linux CI has no keychain; macOS runners do), and the keychain-recovery scenario persists refreshed tokens into the temp-HOME settings file so the read-back can verify them. - Both listener suites inject a readPersistedTokens fake that reflects what their stubbed updateSettings captured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ptors
Root cause of the Linux CI mass failure (~190 tests, present on every
run of this branch): standalone-entry.test.ts bundles standalone-entry
with Bun.build IN-PROCESS, and the git-credential fast path made that
traversal pull the subcommand's entire lazy graph (settings, auth,
telemetry — probe bundle 94KB → 349KB). On low-ulimit runners the file
descriptor pressure broke bun's module resolution for every test file
loaded afterwards ("Cannot find module '@/backend'"), failing ~190
unrelated tests. Reproduced locally with ulimit -n 512 and bisected by
A/B-swapping standalone-entry between 7d20005 and HEAD.
The probe now stubs ./cli/subcommands/git-credential exactly like it
already stubs ./index — the test verifies pi-ai OAuth flows are
statically embedded, nothing else. The real build is unaffected and
still bundles the subcommand.
Also hardens the multi-process suite for full-suite runs: hermetic
worker env (no inherited suite env mutations), 30s barrier deadline for
loaded runners, and the unsynchronized control now accepts the two
timing-dependent worker outcomes it legitimately produces — every
worker persists to one store, so read-back verification correctly
rejects whoever was overwritten ("failed to persist"); the control's
oracle is the refresh count, which must exceed one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Summary
Companion to #3653 (LET-10545). That PR made shared memory's data plane plain git; this one makes plain git actually authenticate reliably — for shared memory mounts and agent MemFS, which share the same plumbing.
Two defects in the status quo (both observed in the wild, see the LET-10545 Slack thread):
osxkeychainon every Mac; once a stale Letta token lands in the keychain, it answers before our repo-local helper and cloud returns HTTP 500 (wrong actor). The keychain gets poisoned in the first place because gitstores a credential to every configured helper after a successful plain-git auth..git/config(plus a.cmdfile on Windows) that OAuth rotation or a project switch silently invalidates.Fix
letta git-credential, a dynamic credential helper following thegh auth git-credential/gcloud auth git-helperpattern. Repo config now holds only:getanswers only for the configured Letta host (foreign hosts get silence, exit 0); token resolution delegates togetClient()— env key → keychain OAuth → single-flight refresh — so auth behavior cannot fork from the rest of the CLI.store/eraseare deliberate no-ops: nothing ever writes our token to the keychain again. Existing poisoned keychains are simply never consulted..cmdtoken files are deleted on the next configure pass. Existing repos self-heal on pull/sync (the config is rewritten by the harness routinely).buildGitAuthArgs) and overrides helpers entirely. Desktop proxy mode is untouched — it still clears credential config and routes through the localhost proxy.Latency
The helper runs on every
git push/pull, so it must not pay full CLI startup.standalone-entry.tsdispatchesgit-credentialbefore importing the main graph, and the subcommand module has zero static imports. Measured: ~50–70ms end to end (including token resolution) vs ~1.1s through the full CLI graph. A 10s deadline fails fast instead of hanging a push on a wedged keychain read.Windows
The previous branch state failed Windows CI in the two credential tests (they assumed the unix inline helper). The platform split is now gone by construction —
!helpers run under git's bundled sh on every platform — and the rewritten tests use argv-array git invocations with no chmod'd scripts or shell quoting.Known limitations / review focus
lettafrom PATH at git-invocation time (same trade-off asgh). Agent shells get it via the shell shim; a human running git by hand needslettainstalled. An older installedlettawithout the subcommand fails auth until the binary updates.Testing
password=is persisted; a regression-control test documents that without the reset entry the inherited helper answers first.git credential fillwith the helper wired into a repo, and timed viastandalone-entry(numbers above).bun run checkgreen; all memory-git suites (97 tests) pass.👾 Generated with Letta Code