Skip to content

fix(memfs): dynamic git credential helper for memory repos - #3668

Draft
cpacker wants to merge 8 commits into
mainfrom
fix/memfs-credential-helper-reset
Draft

fix(memfs): dynamic git credential helper for memory repos#3668
cpacker wants to merge 8 commits into
mainfrom
fix/memfs-credential-helper-reset

Conversation

@cpacker

@cpacker cpacker commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

Companion to #3653 (LET-10545). That PR made shared memory's data plane plain git; this one makes plain git actually authenticate reliably — for shared memory mounts and agent MemFS, which share the same plumbing.

Two defects in the status quo (both observed in the wild, see the LET-10545 Slack thread):

  1. Keychain preemption. Git stacks credential helpers across config scopes and takes the first answer. Xcode's system gitconfig registers osxkeychain on every Mac; once a stale Letta token lands in the keychain, it answers before our repo-local helper and cloud returns HTTP 500 (wrong actor). The keychain gets poisoned in the first place because git stores a credential to every configured helper after a successful plain-git auth.
  2. Staleness. The repo-local helper embedded a static token written at clone/sync time — a plaintext secret in .git/config (plus a .cmd file on Windows) that OAuth rotation or a project switch silently invalidates.

Fix

letta git-credential, a dynamic credential helper following the gh auth git-credential / gcloud auth git-helper pattern. Repo config now holds only:

[credential "https://api.letta.com"]
    helper =                        ← empty entry resets inherited helpers (kills #1)
    helper = !letta git-credential  ← resolves harness auth per operation (kills #2)
  • get answers only for the configured Letta host (foreign hosts get silence, exit 0); token resolution delegates to getClient() — env key → keychain OAuth → single-flight refresh — so auth behavior cannot fork from the rest of the CLI.
  • store/erase are deliberate no-ops: nothing ever writes our token to the keychain again. Existing poisoned keychains are simply never consulted.
  • No secret on disk anywhere; legacy Windows .cmd token files are deleted on the next configure pass. Existing repos self-heal on pull/sync (the config is rewritten by the harness routinely).
  • Harness-run git is untouched — it keeps passing auth per-invocation (buildGitAuthArgs) and overrides helpers entirely. Desktop proxy mode is untouched — it still clears credential config and routes through the localhost proxy.

Latency

The helper runs on every git push/pull, so it must not pay full CLI startup. standalone-entry.ts dispatches git-credential before importing the main graph, and the subcommand module has zero static imports. Measured: ~50–70ms end to end (including token resolution) vs ~1.1s through the full CLI graph. A 10s deadline fails fast instead of hanging a push on a wedged keychain read.

Windows

The previous branch state failed Windows CI in the two credential tests (they assumed the unix inline helper). The platform split is now gone by construction — ! helpers run under git's bundled sh on every platform — and the rewritten tests use argv-array git invocations with no chmod'd scripts or shell quoting.

Known limitations / review focus

  • PATH reliance: the config value resolves letta from PATH at git-invocation time (same trade-off as gh). Agent shells get it via the shell shim; a human running git by hand needs letta installed. An older installed letta without the subcommand fails auth until the binary updates.
  • Cross-process refresh race: two concurrent git ops near token expiry spawn two helper processes that can race the single-flight OAuth refresh (it is per-process). Window is narrow (refresh fires only in the last 5 min of a token's life) and the failure is one retryable op, but flagging for review.
  • Cloud returning 500 instead of 403/404 for a wrong-actor git request (what made this so hard to debug) is a server-side issue, tracked separately.

Testing

  • 12 unit tests on the subcommand (protocol parsing, host gating, silence when unauthenticated, no-op store/erase, deadline, no secret echo on error).
  • Updated mount-clone tests assert the reset entry + dynamic helper and that no password= is persisted; a regression-control test documents that without the reset entry the inherited helper answers first.
  • E2E verified through real git credential fill with the helper wired into a repo, and timed via standalone-entry (numbers above). bun run check green; all memory-git suites (97 tests) pass.

👾 Generated with Letta Code

@cpacker

cpacker commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator Author

Pushed fixes for the external design review's blockers (commit 1527e4e):

Finding 1 (refresh race) — fixed. getClient()'s refresh block now runs under ~/.letta/oauth-refresh.lock (existing withFileLock util); waiters re-read stored tokens after acquiring and reuse the winner's result instead of burning the rotated refresh token again. Note this race predates this PR — any two letta processes (CLI + listener, two sessions) could already hit it; the helper just made it more likely. Since harness git (getAuthToken) and the helper both resolve through getClient(), the lock lands below both delivery paths — which is the "one resolution authority, two thin adapters" shape from the review follow-up. The shared bottleneck already existed; it just wasn't visible.

Finding 2 (durable persistence / exit truncation) — fixed within limits. Tokens are flushed before the lock releases and the stored key is read back to surface silent persistence failures; the helper awaits its stdout write before standalone-entry exits. Residual gap: on keychain-less installs the file-fallback write cannot be verified by read-back (settingsManager.updateSettings swallows persist errors internally); propagating those would mean refactoring settings-manager, deferred.

Finding 7 (host matching) — hardened. Exact protocol + host[:port] match; an http remote at the Letta hostname no longer receives the token.

Finding 4 (PATH coverage) — confirmed, partially open. resolveLettaInvocation indeed writes no shim for the production install — it relies on letta being on the inherited PATH (true for terminal launches, which is also how the repo got configured in the first place). Dev runs get the shim; Desktop is proxy-mode (helper not written, moot). The uncovered case is a non-shell launch surface whose PATH lacks letta — needs the launch-surface matrix from the review before this is called closed.

Finding 5 (measure built artifact) — open. Numbers so far are dev-source via bun src/standalone-entry.ts (~50ms incl. token resolution, vs ~1.1s full graph). Built-artifact cold/warm benchmarks on all three platforms still to do.

Finding 3 needed no change (it argues for this design). Finding 6's wording fix: this PR claims no repo-local credential copies — the keychain-less file fallback in ~/.letta/settings.json is pre-existing and unchanged.

@cpacker

cpacker commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator Author

Re-review blockers addressed in 8ebff23:

1 (waiter doesn't observe winner) — fixed. The under-lock re-read no longer goes through getSettingsWithSecureTokens(). New readPersistedAuthTokens() (src/auth/persisted-tokens.ts) reads the settings file from disk and the keychain directly — bypassing the in-memory expiry, the secure-token cache, and the runtime-scope skip — and reports whether the read was strict. The refresh also prefers the persisted refresh token over the caller's in-memory copy, closing the long-running-harness-burns-invalid-token case. Tests: waiter reuses winner with zero refreshes; two contenders → exactly one refresh; refresh called with the disk token, not the caller's stale one.

2 (verification can lie) — fixed within stated limits. Read-back now uses the same strict cache-bypassing snapshot and verifies both tokens, so the partial-write scenario (new access, old refresh) throws instead of passing. Non-strict read-backs (keychain unavailable, runtime scope, read timeout) are accepted and explicitly marked unverifiable — that's the honest floor without refactoring settings-manager's swallow-errors persistence, which I'd still rather do as its own change.

3 (deadline orphans lock 90s) — fixed by bounding holds rather than plumbing cancellation. Every operation under the lock is now individually bounded: refresh fetch aborts at 15s (AbortSignal.timeout), keychain snapshot reads soft-cap at 5s, lock acquisition times out at 20s, stale reap tightened 90s → 30s. The helper deadline rises 10s → 60s, above the sum of all internal bounds, so it's a backstop that fires only on a primitive the bounds themselves failed to contain — and if that ever happens the orphan reaps in 30s, not 90s. Test: lock file absent after a failed refresh.

On CI: the Linux x64 failure is src/channels/credential-store.test.ts — untouched by this PR (last modified in #3371), passes locally and standalone; looks like a Linux keyring/env flake. Watching whether it recurs on this push.

Still open, unchanged: PATH launch-surface matrix (finding 4) and built-artifact benchmarks (finding 5).

@cpacker

cpacker commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator Author

Final scoping pass per maintainer direction (8e68abe) — the unified-credential-storage branch is parked; this PR is the complete auth story for the git helper.

One refresh lane. refreshTokensCoordinated moved to @/auth/oauth-refresh and now backs every path that spends a refresh token: getClient(), the WebSocket listener, the startup refresh in index.ts — plus a fourth the review didn't list: the ChatGPT usage service was doing the same uncoordinated rotation and is routed too.

Established lock. proper-lockfile replaces the hand-rolled stale-lock algorithm for the refresh lane. Live holders keep the lock fresh via mtime touch; staleness only reaps dead holders.

Fail-closed durable reads. readPersistedAuthTokens distinguishes keychain (authoritative, verified), file (keychain genuinely unavailable → the settings file IS durable storage, used and verified), and runtime-scope (reads skipped, unverifiable). Available-but-erroring/timing-out keychain throws KeychainReadError — no rotation on possibly-stale data. Waiter-reuse also gains the rotation signal: persisted refresh token ≠ caller's ⇒ peer rotated ⇒ adopt theirs.

Multi-process proof (ported from the parked branch's c23c31a3): four real bun subprocesses, one shared store — exactly 1 refresh locked, >1 in the barrier-controlled unlocked control, dead-holder lock reaped.

Real git integration: git credential fill in a cloneRepositoryMount-configured mount with a hostile inherited helper and a fake letta on PATH — reset entry silences the inherited helper, dynamic helper's answer wins. Platform-agnostic (extensionless sh script; git runs ! helpers under its bundled sh everywhere including Git-for-Windows).

Built-artifact latency (macOS arm64, bun letta.js): cold 0.9s, warm ~310ms per credential fill vs ~550ms full-graph — bundle parse dominates the built artifact, unlike dev-source where the bypass gets ~50ms. If 310ms/git-op is unacceptable, the known fix is a separately-compiled tiny helper entrypoint; flagging for maintainer acceptance rather than pre-building it.

Launch surfaces, stated precisely: production install → letta on inherited PATH (how the repo got configured in the first place); dev → shell shim via getShellEnv; scheduled/cron → runs inside harness shells, same shim/PATH; Desktop → proxy mode, helper config actively cleared, untouched; human terminal → installed CLI. The uncovered residual is a non-shell launcher whose PATH lacks letta — the helper then fails with git's standard "helper not found," and the harness rewrites config on next sync.

Desktop proxy mode: no changes anywhere in this pass.

cpacker and others added 6 commits August 3, 2026 20:45
Carved out of #3653 so the auth approach can be reviewed and reverted
independently of the shared-memory subcommand.

Git accumulates credential helpers across config scopes and asks each
in order; on macOS, Xcode's system gitconfig registers osxkeychain,
which can hold a stale Letta token and answer before the repo-local
helper (observed as HTTP 500s on agent-run `git pull` in shared memory
mounts, LET-10545). Write an empty helper entry before ours to reset
the inherited list, scoped to the Letta remote URL only. This matches
what the harness's own git invocations have always done via
`-c credential.helper=` in buildGitAuthArgs.

Also makes the stale mount-path collision error actionable and exports
cloneRepositoryMount for tests.

Known issue (do not merge yet): the two new credential-reset tests
assume the unix inline helper and fail on Windows CI; they need to be
skipped or ported to the .cmd helper path. Open design question from
review: whether to keep the persisted-token approach at all, vs
injecting auth into agent shells (getShellEnv) like the harness does
per-invocation, which would eliminate the on-disk token.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replace the static token persisted in memory repos' .git/config (and the
Windows .cmd token file) with `letta git-credential` — a dynamic helper
following the `gh auth git-credential` pattern. The repo config now holds
only a command reference:

  [credential "https://api.letta.com"]
      helper =                        ; resets inherited helpers (osxkeychain)
      helper = !letta git-credential  ; resolves harness auth per operation

Fixes both defects from LET-10545's plain-git failure mode: the keychain
can no longer answer first with a stale token (reset entry, kept from the
previous commit), and rotation can no longer strand a stale token on disk
(token resolved fresh via getClient — env key, keychain OAuth, single-
flight refresh — on every git network operation). store/erase are no-ops,
so nothing writes our token back into the keychain either.

Latency: the helper runs on every push/pull, so standalone-entry.ts
dispatches `git-credential` before importing the main CLI graph and the
subcommand module has no static imports. Measured ~50-70ms end to end vs
~1.1s through the full graph.

Also drops the platform split that broke Windows CI: `!` helpers run under
git's bundled sh everywhere, and the rewritten tests use argv-array git
invocations with no chmod'd scripts. Legacy .cmd helper files are removed
on the next configure pass; existing repos self-heal on pull/sync.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…lper

Addresses the two blockers from external design review of the dynamic
credential helper, plus a host-matching hardening:

1. Cross-process refresh race: the server rotates the refresh token on
   every refresh (refresh_token_mode: "new"), and the existing
   single-flight guard is per-process — concurrent letta processes (CLI
   sessions, listeners, git-spawned helper invocations) could both burn
   the same refresh token and race their keychain writes, with the loser
   durably persisting an invalidated token (= logout). getClient()'s
   refresh block now runs under a file lock (~/.letta/oauth-refresh.lock,
   via the existing withFileLock util); waiters re-read the stored tokens
   after acquiring the lock and reuse the winner's result instead of
   refreshing again. Since both git delivery paths (harness extraHeader
   and the credential helper) funnel through getClient(), the lock sits
   below both — one resolution authority, two thin delivery adapters.

2. Durable persistence before exit: the rotated tokens are flushed before
   the lock releases, and the stored key is read back to surface silent
   persistence failures (best-effort when no keychain is available). The
   helper also awaits its stdout write, so the process.exit() in
   standalone-entry cannot truncate the credential handed to git.

3. Host matching now requires exact protocol + host[:port]: a
   plaintext-http remote pointed at the Letta hostname no longer receives
   the token.

Fast path unaffected: still ~50ms end to end via standalone-entry.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nded

Closes the three blockers from re-review of 1527e4e:

1. Waiter-reuses-winner now actually observes the winner. The under-lock
   re-read previously used getSettingsWithSecureTokens(), whose expiry is
   this process's in-memory copy and whose keychain read is skipped inside
   runtime scopes — so a fresh helper re-rotated unnecessarily and a
   long-running harness could burn an already-invalidated refresh token.
   New readPersistedAuthTokens() (src/auth/persisted-tokens.ts) reads the
   settings file from disk and the keychain directly, bypassing every
   in-process cache, and reports whether the read was strict. The refresh
   also now prefers the persisted refresh token over the caller's copy.

2. Persistence verification is no longer cache-backed. The post-flush
   read-back uses the same strict snapshot and verifies BOTH tokens — a
   partial keychain write (new access token, old refresh token) previously
   passed the access-only check and stranded auth on the next refresh.
   Non-strict read-backs (no keychain / runtime scope) are accepted and
   documented as unverifiable.

3. Lock holds are bounded so a deadline can no longer orphan the lock for
   90s: the refresh fetch aborts at 15s (AbortSignal.timeout), keychain
   snapshot reads soft-cap at 5s, lock acquisition times out at 20s, and
   the lock's stale reap is 30s. The helper deadline rises to 60s — above
   the sum of all internal bounds — making it a true backstop instead of
   something that fires mid-lock.

refreshTokensUnderCrossProcessLock is now exported with injectable deps;
tests cover waiter-reuse (zero refreshes), persisted-refresh-token
preference, partial-persist detection, non-strict acceptance, lock release
on failure, and two contenders yielding exactly one refresh.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…lane

Final scoping pass on the credential-helper PR:

- refreshTokensCoordinated moves to @/auth/oauth-refresh (the listener
  layer cannot import backend/api/client) and now backs every path that
  spends a refresh token: getClient(), the WebSocket listener, the
  startup refresh in index.ts, and the ChatGPT usage service — the
  review asked for three; the fourth (usage service) was doing the same
  uncoordinated rotation and is routed too. One resolution authority,
  thin delivery adapters above it.

- The lock is proper-lockfile instead of the hand-rolled stale-lock
  algorithm: a live holder keeps the lock fresh via mtime touch, so
  staleness only ever reaps dead holders. Custom file-lock.ts remains
  for its other consumer (reflection transcripts).

- Durable reads fail closed. readPersistedAuthTokens now distinguishes
  its sources: "keychain" (authoritative), "file" (keychain genuinely
  unavailable — the settings file IS durable storage, used and
  verified), and "runtime-scope" (reads skipped; unverifiable). A
  keychain that is available but errors or times out mid-read throws
  KeychainReadError instead of degrading — no rotation on
  possibly-stale data.

- Waiter-reuses-winner gains the rotation signal: a persisted refresh
  token that differs from the caller's means a peer rotated, so the
  winner's access token is adopted even when expiry alone is
  inconclusive.

- Multi-process proof (ported from the parked coordination branch):
  four real bun subprocesses against one shared store — exactly one
  refresh with the shared lock, >1 in the barrier-controlled unlocked
  control, and an abandoned lock from a dead holder is reaped.

- Real-git integration test: `git credential fill` in a mount
  configured by cloneRepositoryMount, with a hostile inherited helper
  and a fake `letta` on PATH — proves the reset entry silences the
  inherited helper and the dynamic helper's answer wins, on every
  platform (extensionless sh script; git runs helpers under its
  bundled sh).

Built-artifact latency (bun letta.js, macOS arm64): cold 0.9s, warm
~310ms per credential fill — bundle parse dominates; dev-source fast
path is ~50ms. A separately-compiled tiny helper entrypoint can
recover the difference if 310ms is deemed too slow.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@cpacker
cpacker force-pushed the fix/memfs-credential-helper-reset branch from 8e68abe to 2e3c238 Compare August 4, 2026 03:46
cpacker and others added 2 commits August 3, 2026 20:56
client-soft-fail and the two listener auth suites neutered persistence
(no-op updateSettings) or omitted a snapshot source, so the coordinated
refresh's durable read-back correctly rejected them as failed persists —
which is the exact behavior the read-back exists to catch, but here it
was the test harness, not the product, failing to persist.

- client-soft-fail scenarios run under LETTA_SKIP_KEYCHAIN_CHECK so the
  durable snapshot is file-backed and deterministic on every platform
  (Linux CI has no keychain; macOS runners do), and the keychain-recovery
  scenario persists refreshed tokens into the temp-HOME settings file so
  the read-back can verify them.
- Both listener suites inject a readPersistedTokens fake that reflects
  what their stubbed updateSettings captured.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ptors

Root cause of the Linux CI mass failure (~190 tests, present on every
run of this branch): standalone-entry.test.ts bundles standalone-entry
with Bun.build IN-PROCESS, and the git-credential fast path made that
traversal pull the subcommand's entire lazy graph (settings, auth,
telemetry — probe bundle 94KB → 349KB). On low-ulimit runners the file
descriptor pressure broke bun's module resolution for every test file
loaded afterwards ("Cannot find module '@/backend'"), failing ~190
unrelated tests. Reproduced locally with ulimit -n 512 and bisected by
A/B-swapping standalone-entry between 7d20005 and HEAD.

The probe now stubs ./cli/subcommands/git-credential exactly like it
already stubs ./index — the test verifies pi-ai OAuth flows are
statically embedded, nothing else. The real build is unaffected and
still bundles the subcommand.

Also hardens the multi-process suite for full-suite runs: hermetic
worker env (no inherited suite env mutations), 30s barrier deadline for
loaded runners, and the unsynchronized control now accepts the two
timing-dependent worker outcomes it legitimately produces — every
worker persists to one store, so read-back verification correctly
rejects whoever was overwritten ("failed to persist"); the control's
oracle is the refresh count, which must exceed one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@cpacker
cpacker marked this pull request as draft August 5, 2026 06:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant