Skip to content

Multiple clients sharing result queues causes race condition — support client-specific result routing #150

@anhthii

Description

@anhthii

Problem

When two or more MPCClient instances connect to the same NATS cluster, they all subscribe to the same result work queues:

Operation Result Topic
Keygen mpc.mpc_keygen_result.*
Signing mpc.mpc_signing_result.complete
Reshare mpc.mpc_reshare_result.*

Because the result streams use WorkQueuePolicy (message deleted on ACK), consumers on the same stream compete for messages. Client A's keygen result may be delivered to Client B, and vice versa.

Architecture Diagram — Current (Broken with Multiple Clients)

┌──────────────┐         ┌──────────────┐
│  Client A    │         │  Client B    │
│  (Backend 1) │         │  (Backend 2) │
└──────┬───────┘         └──────┬───────┘
       │ CreateWallet("w1")     │ CreateWallet("w2")
       ▼                        ▼
┌─────────────────────────────────────────┐
│          NATS JetStream                 │
│                                         │
│  keygen_request stream                  │
│  ┌─────────┐  ┌─────────┐              │
│  │ req: w1 │  │ req: w2 │              │
│  └────┬────┘  └────┬────┘              │
│       └──────┬─────┘                   │
│              ▼                          │
│     MPC Cluster processes both          │
│              │                          │
│              ▼                          │
│  keygen_result stream (WorkQueuePolicy) │
│  ┌──────────────┐  ┌──────────────┐    │
│  │ result: w1   │  │ result: w2   │    │
│  └──────┬───────┘  └──────┬───────┘    │
│         │                 │             │
│         ▼                 ▼             │
│  ┌────────────────────────────────┐     │
│  │ consumer: "mpc-keygen-consumer"│     │
│  │ filter: mpc.mpc_keygen_result.*│     │
│  │ (SHARED by both clients!)      │     │
│  └────────────┬───────────────────┘     │
└───────────────┼─────────────────────────┘
                │
        ┌───────┴───────┐
        ▼               ▼
   Client A          Client B
   gets w2 ✗         gets w1 ✗    ← RACE! Wrong results delivered

Reproduction Scenario

  1. Client A (backend service 1) calls CreateWallet("wallet-A") and registers OnWalletCreationResult(callbackA).
  2. Client B (backend service 2) calls CreateWallet("wallet-B") and registers OnWalletCreationResult(callbackB).
  3. MPC cluster completes keygen for wallet-A and publishes result to mpc.mpc_keygen_result.wallet-A.
  4. Client B receives wallet-A's result (because both share the same durable consumer on WorkQueuePolicy).
  5. Client B's callback processes a wallet it never requested → unexpected behavior.
  6. Client A never receives its own result → timeout / lost event.

The same issue affects signing results (even worse since mpc.mpc_signing_result.complete is an exact single topic shared by all clients) and reshare results.


Proposed Solution — Client-Specific Result Routing

Introduce an optional clientID when constructing MPCClient. When set, the client subscribes to client-scoped result topics, and the MPC cluster routes results to the correct client-specific stream.

Architecture Diagram — Proposed (With Client ID)

┌───────────────────┐          ┌───────────────────┐
│  Client A         │          │  Client B         │
│  clientID: "svcA" │          │  clientID: "svcB" │
└──────┬────────────┘          └──────┬────────────┘
       │ CreateWallet("w1")           │ CreateWallet("w2")
       │ (header: clientID=svcA)      │ (header: clientID=svcB)
       ▼                              ▼
┌──────────────────────────────────────────────────────┐
│                    NATS JetStream                     │
│                                                      │
│  keygen_request stream                               │
│  ┌──────────────────┐  ┌──────────────────┐          │
│  │ req: w1          │  │ req: w2          │          │
│  │ hdr: clientID=A  │  │ hdr: clientID=B  │          │
│  └────────┬─────────┘  └────────┬─────────┘          │
│           └──────────┬──────────┘                    │
│                      ▼                               │
│           MPC Cluster processes both                 │
│                      │                               │
│           ┌──────────┴──────────┐                    │
│           ▼                     ▼                    │
│  Result for w1               Result for w2           │
│  topic: mpc.mpc_keygen_      topic: mpc.mpc_keygen_ │
│         result.svcA.w1              result.svcB.w2   │
│           │                         │                │
│           ▼                         ▼                │
│  ┌─────────────────────┐  ┌─────────────────────┐   │
│  │consumer: keygen-svcA│  │consumer: keygen-svcB│   │
│  │filter: ...result.   │  │filter: ...result.   │   │
│  │  svcA.*             │  │  svcB.*             │   │
│  └──────────┬──────────┘  └──────────┬──────────┘   │
└─────────────┼─────────────────────────┼──────────────┘
              ▼                         ▼
         Client A                  Client B
         gets w1 ✓                 gets w2 ✓     ← Correct!

Topic Scheme

Operation Without clientID (backward compat) With clientID
Keygen result mpc.mpc_keygen_result.<walletID> mpc.mpc_keygen_result.<clientID>.<walletID>
Signing result mpc.mpc_signing_result.complete mpc.mpc_signing_result.<clientID>.complete
Reshare result mpc.mpc_reshare_result.<sessionID> mpc.mpc_reshare_result.<clientID>.<sessionID>

Backward Compatibility

Scenario Behavior
NewMPCClient(signer, nats) — no clientID Works exactly as today. Same topics, same consumers. Zero breaking changes.
NewMPCClient(signer, nats, WithClientID("svcA")) — with clientID Client subscribes to mpc.mpc_keygen_result.svcA.* etc. Requests include clientID header. Results routed to client-specific topics.
Mixed deployment (some clients with ID, some without) Each operates independently. Legacy clients get legacy topics, ID-aware clients get scoped topics.

Implementation Scope

  1. pkg/client/client.go — Add WithClientID(id string) option. When set, modify topic subscriptions and include clientID header in all requests.
  2. pkg/event/ — Add topic builder functions that accept optional clientID (e.g., KeygenResultTopic(clientID, walletID) string).
  3. pkg/eventconsumer/event_consumer.go — When publishing results, check for clientID header on the original request. If present, publish to client-scoped topic. If absent, publish to legacy topic.
  4. pkg/messaging/message_queue.go — Support dynamic consumer/stream creation for client-scoped result queues.
  5. TypeScript client (mpcium-client-ts) — Mirror the same clientID option.

Impact

  • No changes required for existing single-client deployments.
  • Enables multi-tenant and multi-instance deployments safely.
  • Signing results benefit the most since all clients currently compete on the single mpc.mpc_signing_result.complete topic.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions