Skip to content

GO-7116 Fix deadlock on app close in inboxclient#3057

Merged
requilence merged 4 commits intodevelopfrom
GO-7116-fix-inboxclient-close-deadlock
Apr 15, 2026
Merged

GO-7116 Fix deadlock on app close in inboxclient#3057
requilence merged 4 commits intodevelopfrom
GO-7116-fix-inboxclient-close-deadlock

Conversation

@requilence
Copy link
Copy Markdown
Contributor

Summary

  • Add componentCtx to inboxclient that gets cancelled on Close(), ensuring all operations (both periodic and external callback) are cancellable on shutdown
  • Propagate context through fetchMessages/getOffset/setOffset instead of hardcoded context.Background()

Problem

inboxclient.Close() deadlocks when the coordinator connection is stale. The periodic sync callback (checkMessages) ignores its context parameter and calls fetchMessages() which uses context.Background() for the InboxFetch RPC. When the coordinator doesn't respond, the RPC blocks forever, and periodicCall.Close() waits indefinitely for the loop to finish.

Deadlock chain:

  1. app.Close()inboxclient.Close()periodicCheck.Close() → cancels loop context, waits on <-loopDone
  2. Periodic loop executing checkMessages(ctx) → ignores ctx → fetchMessages()InboxFetch(context.Background()) → blocked forever on stale DRPC stream

Test plan

  • All existing inboxclient tests pass
  • ReceiveNotify (external callback) now uses componentCtx instead of context.Background() — also cancellable on close

Add componentCtx to inboxclient that gets cancelled on Close(),
and propagate context through fetchMessages/getOffset/setOffset
instead of using context.Background().

Previously, periodicCheck.Close() would wait for the periodic loop
to finish, but the loop's callback used context.Background() for
RPC calls (InboxFetch), making them uncancellable. When the
coordinator connection went stale, the RPC would block forever,
causing a deadlock on shutdown.
Comment thread core/inboxclient/inboxclient.go Outdated
@requilence requilence requested a review from deff7 March 31, 2026 15:50
@github-actions
Copy link
Copy Markdown

New Coverage 47.2% of statements
Patch Coverage 47.6% of changed statements (20/42)

Coverage provided by https://github.com/seriousben/go-patch-cover-action

@requilence requilence merged commit b33babd into develop Apr 15, 2026
14 checks passed
@requilence requilence deleted the GO-7116-fix-inboxclient-close-deadlock branch April 15, 2026 12:06
@github-actions github-actions Bot locked and limited conversation to collaborators Apr 15, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants