Skip to content

fix(auth): recover expired sessions instead of silently decaying - #75

Merged
NitayRabi merged 1 commit into
masterfrom
t3code/expired-auth-fix
Jul 13, 2026
Merged

fix(auth): recover expired sessions instead of silently decaying#75
NitayRabi merged 1 commit into
masterfrom
t3code/expired-auth-fix

Conversation

@NitayRabi

Copy link
Copy Markdown
Contributor

The bug

Users report that once the IBKR session expires, auth "doesn't work properly" and only a restart fixes it. The production logs on a live install show exactly why.

Counting /tickle calls against the Gateway's own /sso/ping in gw.2026-07-13.log:

  hour 00 -> tickle=0   sso_ping=12
  hour 01 -> tickle=0   sso_ping=12
  ...
  hour 08 -> tickle=0   sso_ping=12
  hour 09 -> tickle=53  sso_ping=8     <- after a manual restart

Zero tickles all night, and zero the previous day. The Gateway's internal /sso/ping kept returning 200 {"result":"true"} every 5 minutes, so the SSO layer stayed alive — but nothing was calling /tickle, so the iserver brokerage session was never maintained. That's the session that was dead the next morning.

Why the tickler wasn't running

src/scripts/tickler.ts treated any error as fatal:

console.log("[TICKLER] Gateway unreachable or network error. Self-terminating.");
return false;   // -> process.exit(0)

One transient blip — a Gateway hiccup, a laptop suspend — and the tickler exits permanently. Nothing respawns it: spawnDurableTickler() is only reachable from startTickle(), which only runs after a successful auth check inside the MCP process.

Why recovery was more expensive than necessary

ensureAuth() jumped straight from "not authenticated" to a full headless browser + TOTP login. But the logs show SSO was healthy the entire time — only the brokerage session was dead. A single POST /iserver/reauthenticate would have restored it: no browser, no 2FA, no exposure to IBKR's login rate limits (the ones we already hit in #73).

reauthenticate() already existed in ib-client.ts — it was just never wired into session death.

The fix

A recovery ladder, cheapest rung first: tickle → POST /iserver/reauthenticate → full login.

  • Tickler no longer self-terminates on transient errors. Outcomes are classified ok / transient / terminal. Transient errors retry with backoff (5s→10s→20s, capped at the 30s cadence so the session never goes untickled). It exits only after sustained failure, and the MCP server respawns it on the next authenticated call. It can also perform the cheap reauth rung itself (it holds cookies, so no credentials needed).
  • Cheap rung tried before a full login, in both headless and browser mode — browser-mode users previously had no recovery at all and had to go click through a login page.
  • competing is no longer ignored. grep -rn "competing" src/ previously returned zero hits, while established: true sits right beside it in the real payload and was trusted unconditionally. So a session IBKR had handed to another client (phone, web portal) was reported healthy and then failed on the next call.
  • Circuit breaker after 5 consecutive failed logins → AUTHENTICATION_CIRCUIT_OPEN instead of firing a fresh browser login on every tool call. Directly targets the IBKR rate-limiting from fix(2fa): handle IBKR multi-device second-factor chooser for TOTP #73. An explicit authenticate call resets it.
  • Transport errors (ECONNRESET/EPIPE/socket hang up/stream was destroyed) are now classified as auth-recoverable, so they enter the ladder instead of surfacing raw. Previously isAuthenticationError() only matched 401/403/500, so a dropped socket never triggered recovery.

Scope: no gateway process control

Deliberately excluded. v2 (#74) adopts a gateway it did not start (findExistingGateway()) and explicitly never kills it ("Don't kill gateway - just clean references"). Restarting the Gateway is IBeam's top recovery rung, but IBeam earns it by vendoring the gateway and shipping it in its own container — it owns the process outright. We don't, so recovery stays entirely at the HTTP session layer and terminates in circuit-break-and-report rather than restart.

Verification

  • npm run build ✅ · oxlint ✅ (warnings only, matches baseline) · 205 tests pass, 4 new (competing-session, reauth-recovery-without-browser, browser-mode recovery, circuit breaker)
  • One existing test updated: it asserted reauthenticate is not called in browser mode — that assertion encoded the bug.
  • New tickler vs. unreachable gateway: retried 3× with backoff over 22s and never self-terminated. The old code exited on the first error.
  • New tickler vs. a live gateway: tickled successfully at 30s cadence, still looping.
  • Live gateway /iserver/auth/status confirms competing is a real field, returned alongside established: true.

Follow-up (not in this PR)

ib-client.ts's runtimeDir still points at path.join(__dirname, "../ib-gateway/.runtime") — inside node_modules. v2 moves Gateway state to <cache>/run/ via DependencyResolver.runDir() but does not migrate the tickler's PID file. Post-v2 the npx cache dir is content-hashed, so a version bump orphans the PID file and spawnDurableTickler() will spawn duplicates. One-line fix, but it belongs in #74.

🤖 Generated with Claude Code

The durable tickler treated any error as fatal and exited, and nothing ever
respawned it. A single transient network blip therefore killed session
maintenance permanently: production logs show zero /tickle calls for two days
straight while the gateway's own /sso/ping kept returning 200, so the SSO layer
stayed alive while the iserver brokerage session quietly decayed. The next
morning the first tool call failed, and only a manual restart brought it back.

Recovery was also far more expensive than it needed to be. ensureAuth() jumped
straight from "not authenticated" to a full headless browser + TOTP login, even
though the SSO session is usually still valid and a plain reauthenticate would
have restored the brokerage session with a single POST - no browser, no 2FA, and
no exposure to IBKR's login rate limits.

- tickler: classify outcomes as ok/transient/terminal. Transient errors retry
  with backoff (capped at the tickle cadence so the session never goes
  untickled) instead of terminating the process. It exits only after sustained
  failure, and the MCP server respawns it on the next authenticated call.
- add a recovery ladder (tickle -> /iserver/reauthenticate -> full login) and
  try the cheap rung first, in both headless and browser mode.
- treat a competing session as unhealthy. `competing` was referenced nowhere,
  while `established: true` sits right beside it and was trusted
  unconditionally, so a session IBKR had handed to another client was reported
  as healthy and then failed on the next call.
- add a circuit breaker after 5 consecutive failed logins so a broken login
  stops firing a fresh browser session on every tool call and risking a lockout.
- classify transport errors (ECONNRESET/EPIPE/socket hang up/stream destroyed)
  as auth-recoverable so they enter the ladder rather than surfacing raw.

Deliberately excluded: gateway process control. v2 adopts a gateway it did not
start and never kills it, so restarting the gateway is not ours to do; recovery
stays entirely at the HTTP session layer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@NitayRabi
NitayRabi merged commit 234897a into master Jul 13, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant