fix(auth): recover expired sessions instead of silently decaying - #75
Merged
Conversation
The durable tickler treated any error as fatal and exited, and nothing ever respawned it. A single transient network blip therefore killed session maintenance permanently: production logs show zero /tickle calls for two days straight while the gateway's own /sso/ping kept returning 200, so the SSO layer stayed alive while the iserver brokerage session quietly decayed. The next morning the first tool call failed, and only a manual restart brought it back. Recovery was also far more expensive than it needed to be. ensureAuth() jumped straight from "not authenticated" to a full headless browser + TOTP login, even though the SSO session is usually still valid and a plain reauthenticate would have restored the brokerage session with a single POST - no browser, no 2FA, and no exposure to IBKR's login rate limits. - tickler: classify outcomes as ok/transient/terminal. Transient errors retry with backoff (capped at the tickle cadence so the session never goes untickled) instead of terminating the process. It exits only after sustained failure, and the MCP server respawns it on the next authenticated call. - add a recovery ladder (tickle -> /iserver/reauthenticate -> full login) and try the cheap rung first, in both headless and browser mode. - treat a competing session as unhealthy. `competing` was referenced nowhere, while `established: true` sits right beside it and was trusted unconditionally, so a session IBKR had handed to another client was reported as healthy and then failed on the next call. - add a circuit breaker after 5 consecutive failed logins so a broken login stops firing a fresh browser session on every tool call and risking a lockout. - classify transport errors (ECONNRESET/EPIPE/socket hang up/stream destroyed) as auth-recoverable so they enter the ladder rather than surfacing raw. Deliberately excluded: gateway process control. v2 adopts a gateway it did not start and never kills it, so restarting the gateway is not ours to do; recovery stays entirely at the HTTP session layer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
Users report that once the IBKR session expires, auth "doesn't work properly" and only a restart fixes it. The production logs on a live install show exactly why.
Counting
/ticklecalls against the Gateway's own/sso/pingingw.2026-07-13.log:Zero tickles all night, and zero the previous day. The Gateway's internal
/sso/pingkept returning200 {"result":"true"}every 5 minutes, so the SSO layer stayed alive — but nothing was calling/tickle, so the iserver brokerage session was never maintained. That's the session that was dead the next morning.Why the tickler wasn't running
src/scripts/tickler.tstreated any error as fatal:One transient blip — a Gateway hiccup, a laptop suspend — and the tickler exits permanently. Nothing respawns it:
spawnDurableTickler()is only reachable fromstartTickle(), which only runs after a successful auth check inside the MCP process.Why recovery was more expensive than necessary
ensureAuth()jumped straight from "not authenticated" to a full headless browser + TOTP login. But the logs show SSO was healthy the entire time — only the brokerage session was dead. A singlePOST /iserver/reauthenticatewould have restored it: no browser, no 2FA, no exposure to IBKR's login rate limits (the ones we already hit in #73).reauthenticate()already existed inib-client.ts— it was just never wired into session death.The fix
A recovery ladder, cheapest rung first:
tickle → POST /iserver/reauthenticate → full login.ok/transient/terminal. Transient errors retry with backoff (5s→10s→20s, capped at the 30s cadence so the session never goes untickled). It exits only after sustained failure, and the MCP server respawns it on the next authenticated call. It can also perform the cheap reauth rung itself (it holds cookies, so no credentials needed).competingis no longer ignored.grep -rn "competing" src/previously returned zero hits, whileestablished: truesits right beside it in the real payload and was trusted unconditionally. So a session IBKR had handed to another client (phone, web portal) was reported healthy and then failed on the next call.AUTHENTICATION_CIRCUIT_OPENinstead of firing a fresh browser login on every tool call. Directly targets the IBKR rate-limiting from fix(2fa): handle IBKR multi-device second-factor chooser for TOTP #73. An explicitauthenticatecall resets it.ECONNRESET/EPIPE/socket hang up/stream was destroyed) are now classified as auth-recoverable, so they enter the ladder instead of surfacing raw. PreviouslyisAuthenticationError()only matched 401/403/500, so a dropped socket never triggered recovery.Scope: no gateway process control
Deliberately excluded. v2 (#74) adopts a gateway it did not start (
findExistingGateway()) and explicitly never kills it ("Don't kill gateway - just clean references"). Restarting the Gateway is IBeam's top recovery rung, but IBeam earns it by vendoring the gateway and shipping it in its own container — it owns the process outright. We don't, so recovery stays entirely at the HTTP session layer and terminates in circuit-break-and-report rather than restart.Verification
npm run build✅ ·oxlint✅ (warnings only, matches baseline) · 205 tests pass, 4 new (competing-session, reauth-recovery-without-browser, browser-mode recovery, circuit breaker)reauthenticateis not called in browser mode — that assertion encoded the bug./iserver/auth/statusconfirmscompetingis a real field, returned alongsideestablished: true.Follow-up (not in this PR)
ib-client.ts'sruntimeDirstill points atpath.join(__dirname, "../ib-gateway/.runtime")— insidenode_modules. v2 moves Gateway state to<cache>/run/viaDependencyResolver.runDir()but does not migrate the tickler's PID file. Post-v2 the npx cache dir is content-hashed, so a version bump orphans the PID file andspawnDurableTickler()will spawn duplicates. One-line fix, but it belongs in #74.🤖 Generated with Claude Code