Skip to content
This repository was archived by the owner on Jun 8, 2026. It is now read-only.

FEAT-160: Add gateway server liveness detection and auto-recovery - #42

Merged
shafty023 merged 2 commits into
mainfrom
FEAT-160
Mar 25, 2026
Merged

FEAT-160: Add gateway server liveness detection and auto-recovery#42
shafty023 merged 2 commits into
mainfrom
FEAT-160

Conversation

@shafty023

@shafty023 shafty023 commented Mar 25, 2026

Copy link
Copy Markdown
Contributor
  • Add alive flag, isAlive(), restart() to DesktopGatewayServer
  • Fix stop() to handle stale (ERR_SERVER_NOT_RUNNING) handles gracefully
  • Attach persistent error/close handlers with onUnexpectedClose callback
  • Resolve actual OS-assigned port via server.address() in start()
  • Extract GatewayRecoveryManager with single-flight restart, epoch guard, and gatewayHealthy flag into gateway-recovery.ts
  • Wire recovery into DesktopApplication: liveness probe on cloud reconnect, auto-restart on unexpected close, gatewayHealthy in presence derivation
  • Refactor onCloudSocketStatus to probe gateway before enabling executor
  • Add serverAlive/gatewayHealthy to runtime status endpoint
  • Switch all test files from fixed ports to OS-assigned port 0, eliminating EADDRINUSE failures when test files run concurrently
  • Add 5 server-level liveness tests and 6 recovery state machine tests

Testing:

  • All 323 tests pass with 0 failures
  • Typecheck and lint clean

Risks:

  • CloudSocketService still sends immediate presence on hello.ack before liveness probe completes (documented as known limitation, deferred)

https://app.closedloop.ai/implementation-plans/PLAN-86

- Add alive flag, isAlive(), restart() to DesktopGatewayServer
- Fix stop() to handle stale (ERR_SERVER_NOT_RUNNING) handles gracefully
- Attach persistent error/close handlers with onUnexpectedClose callback
- Resolve actual OS-assigned port via server.address() in start()
- Extract GatewayRecoveryManager with single-flight restart, epoch guard,
  and gatewayHealthy flag into gateway-recovery.ts
- Wire recovery into DesktopApplication: liveness probe on cloud reconnect,
  auto-restart on unexpected close, gatewayHealthy in presence derivation
- Refactor onCloudSocketStatus to probe gateway before enabling executor
- Add serverAlive/gatewayHealthy to runtime status endpoint
- Switch all test files from fixed ports to OS-assigned port 0, eliminating
  EADDRINUSE failures when test files run concurrently
- Add 5 server-level liveness tests and 6 recovery state machine tests

Testing:
- All 323 tests pass with 0 failures
- Typecheck and lint clean

Risks:
- CloudSocketService still sends immediate presence on hello.ack before
  liveness probe completes (documented as known limitation, deferred)
@shafty023
shafty023 requested review from thadeusb and wongk March 25, 2026 19:01
Comment thread apps/desktop/src/main/gateway-recovery.ts
@closedloop-ai-stage

Copy link
Copy Markdown

Code Review Summary

Status: Approved

Reviewers: Bug Hunter A (x3), Bug Hunter B, Unified Auditor, Premise Reviewer, Cloud Command Executor Specialist

Findings

Severity Count
Blocking 0
High 0
Medium 1

MEDIUM Issues (consider)

  1. [P3] [apps/desktop/src/main/gateway-recovery.ts:21] Deduplicated recoverGateway call silently drops second reason -- When a second recovery trigger fires while one is already in-flight, the new reason is silently discarded with no log entry. Add a warn log in the early-return branch.

Validation Stats

  • Agent failures: 0 partitions skipped
  • Cross-file grouped: 0 findings consolidated

Recommendation: Approve -- the gateway liveness detection and auto-recovery feature looks solid. One low-priority observability improvement suggested but not blocking.

Add warn log when recoverGateway deduplicates a concurrent call so the
second trigger reason is not silently dropped.
@shafty023
shafty023 merged commit 03571e7 into main Mar 25, 2026
2 checks passed
@shafty023
shafty023 deleted the FEAT-160 branch March 25, 2026 19:30
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant