Skip to content

fix(channels): roll back timed-out account startups - #3591

Open
GautamSharma99 wants to merge 1 commit into
letta-ai:mainfrom
GautamSharma99:fix/channel-startup-timeout-rollback
Open

fix(channels): roll back timed-out account startups#3591
GautamSharma99 wants to merge 1 commit into
letta-ai:mainfrom
GautamSharma99:fix/channel-startup-timeout-rollback

Conversation

@GautamSharma99

Copy link
Copy Markdown

Summary

Fixes #3581.

startChannelAccountLive() previously raced registry startup against a timer that had no connection to the operation it timed out. When the timer rejected, persisted state was changed back to disabled, but the registry still owned a registered adapter and its start() promise could later resolve into a live integration.

This PR makes timeout a real lifecycle cancellation boundary and orders all account start/stop operations with per-account generations. A startup that times out or is superseded is immediately unpublished, stopped best-effort, and stopped again if its original start() later settles. Cleanup is identity-checked so an old attempt cannot remove or stop a newer retry adapter.

What changed

Timeout now reaches the registry

startChannelAccountLive() now creates an AbortController for the existing 10-second deadline and passes its signal to ChannelRegistry.startChannelAccount().

The registry races the signal at each asynchronous startup boundary:

  • loading account secrets;
  • Signal conflict discovery;
  • stopping an existing account adapter;
  • loading the channel plugin;
  • creating the adapter;
  • starting the adapter.

The original timeout error is carried as the abort reason, preserving the existing actionable user-facing message.

Per-account lifecycle generations

Added a focused ChannelAccountLifecycle owner for generation and retirement state. Every start receives a generation for its (channelId, accountId) key. The registry verifies that generation after each asynchronous step and before reporting startup success.

These operations invalidate older generations:

  • a newer account start;
  • stopChannelAccount();
  • whole-channel stopChannel();
  • registry shutdown via stopAll().

This makes a concurrent stop deterministic even when it arrives before adapter creation or while adapter.start() is pending.

Identity-safe stale cleanup

When startup fails, times out, or becomes stale, cleanup:

  1. unregisters the adapter only if the registry still maps that exact adapter instance;
  2. calls stop() immediately as a best-effort cancellation;
  3. observes the original startup promise;
  4. calls stop() again after late resolution/rejection;
  5. repeats identity-checked unregistration.

The second stop is required for adapters whose first stop() is a no-op while startup has not yet marked itself running. Crucially, cleanup retains the old adapter object and never looks up the key and stops whatever happens to be there later, so a successful retry is unaffected.

Adapters are also removed from the registry before stop is awaited. Callers and snapshots therefore cannot observe a disabled account as registered/live while slow cleanup completes.

Persisted-state race protection

The service layer now tracks its own account lifecycle generation. A stale start failure only rolls persisted enabled state back when it is still the newest service operation.

Without this guard, an older start superseded by a retry could reject later and overwrite the retry state back to disabled even though the new adapter succeeded. Stop and remove operations also advance this generation, and startup rechecks it around display-name refresh before returning success.

Existing failure cleanup

The same retirement path now unregisters adapters after ordinary startup failures, not only timeouts. Stop paths call stop() for registered pending adapters even when isRunning() is still false, allowing adapters with in-progress initialization to cancel early.

Tests

Added real delayed Telegram adapter lifecycle tests covering:

  • service timeout disables persistent state and immediately unregisters the pending adapter;
  • a delayed start() resolving after timeout is stopped and cannot re-register;
  • a retry after timeout remains registered and running after stale cleanup executes;
  • stale failure rollback does not overwrite the retry enabled state;
  • a concurrent stop supersedes pending startup and leaves registry and persistence disabled.

Validation completed:

  • race-specific, registry lifecycle, and Telegram runtime tests: 30 passed;
  • complete src/channels suite: 816 passed, 1 intentional live-network test skipped, 0 failed;
  • bun run check: all 12 repository checks passed;
  • registry.ts remains below the enforced source ceiling at 977 lines after extracting lifecycle ownership.

Concurrency semantics

For one account key, the newest lifecycle operation wins. Cleanup from an older operation may only act on the adapter instance created by that operation. Operations for different account keys remain independent and are not globally serialized.

The adapter interface is unchanged. Existing adapters can continue implementing their own startup timeouts/cancellation, while the registry generation guard provides the cross-adapter safety boundary required by the service-level timeout.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Channel startup timeout can leave a disabled account adapter running

1 participant