Gateway WebSocket handshake leaks tenant context onto the shared Netty event-loop thread

[nimsara66]

Description

InboundWebSocketProcessor.handleHandshake(...) opens a Carbon tenant flow for the handshake — it calls PrivilegedCarbonContext.startTenantFlow() and setTenantDomain() — but never calls PrivilegedCarbonContext.endTenantFlow(). Because the gateway's WebSocket inbound runs on a shared Netty event-loop thread pool, the tenant domain set during one handshake stays pinned to that thread after the handshake returns.

A later WebSocket frame authentication (InboundWebsocketProcessorUtil.authenticateToken(...), used by GraphQLRequestProcessor / RequestProcessor / GraphQLResponseProcessor / ResponseProcessor) did not establish its own tenant flow, so it executed under whatever tenant a previous, unrelated handshake had leaked onto that thread. Key-manager resolution is tenant-scoped, so the frame resolves the wrong tenant's key manager and a valid token is rejected — the connection is closed with WS code 4001 "Invalid JWT token" even though the token is legitimate.

Impact: intermittent, hard-to-diagnose authentication failures on WebSocket and GraphQL-subscription APIs in any multi-tenant deployment. A perfectly valid token fails at random depending on which handshake last touched the event-loop thread handling its frame. The gateway logs only a masked Invalid JWT token (900901 / 4001), giving no hint that the real cause is cross-tenant thread state. Same defect family as the RemoteServerLoggerData.getUsername() tenant-context NPE.

Affected component/version: org.wso2.carbon.apimgt.gateway — WebSocket inbound (inbound/websocket), observed on 9.33.162-SNAPSHOT (WSO2 API Manager 4.7.0-SNAPSHOT). Any release where handleHandshake starts a tenant flow without ending it is affected.

Root cause (code):

• .../inbound/websocket/InboundWebSocketProcessor.java → handleHandshake(...): startTenantFlow() is called with no matching endTenantFlow() in a finally.
• .../inbound/websocket/utils/InboundWebsocketProcessorUtil.java → authenticateToken(...): frame auth trusts the thread's ambient tenant instead of establishing its own from the request context.

Expected: a valid token is authenticated on its own API regardless of concurrent handshakes for other tenants; tenant context set during a handshake is always cleared before the thread is reused.
Actual: valid token intermittently rejected with 4001 "Invalid JWT token" when a concurrent other-tenant handshake poisons the shared event-loop thread.

Steps to Reproduce

1. Create a second tenant. As super-admin, create tenant tenant1.com with a tenant admin (admin@tenant1.com). (carbon.super already exists.)
2. Publish a GraphQL-subscription API in carbon.super (the victim).

• Create a GraphQL API with a schema that defines a Subscription type (e.g. subscription { liftStatusChange { name } }).
• Deploy and publish it. Note its gateway context (e.g. /gqlsub).

3. Publish a WebSocket API in tenant1.com (the leak source).

• As admin@tenant1.com, create and deploy a WebSocket API (any echo/streaming backend).
• Publish it. Note its full tenant context — for a tenant API this includes the prefix, e.g. /t/tenant1.com/wsecho.

4. Get a valid carbon.super token for the victim API.

• Create an application in carbon.super, subscribe it to the GraphQL-subscription API, generate keys, and obtain an OAuth2 access token. This token is valid and correctly scoped — it must succeed.

5. Open the victim GraphQL subscription over WebSocket to the carbon.super API's gateway WS endpoint using that token (graphql-transport-ws: send connection_init, then a subscribe/start frame). Do not complete it instantly — keep the connection in its pre-init/idle window.
6. Concurrently flood handshakes to the tenant1.com WS API. From several parallel clients, open ~100–200 WebSocket handshakes to /t/tenant1.com/wsecho in a tight burst. Each handshake reaches the gateway, runs handleHandshake → startTenantFlow() + setTenantDomain("tenant1.com") on an event-loop thread, then returns without endTenantFlow() — leaving tenant1.com pinned on those threads. (The flood handshakes themselves can even fail auth; they only need to reach the tenant-flow start.)
7. Observe the failure. The victim's carbon.super frame auth lands on a thread still pinned to tenant1.com, resolves tenant1.com's key manager for a carbon.super token, and the gateway closes the subscription with WS code 4001 "Invalid JWT token" — despite the token from step 4 being valid.

Gateway log signature (single incident):
ERROR - JWTValidator Invalid JWT token. {... "alg":"RS256"}
ERROR - OAuthAuthenticator 4001
org.wso2.carbon.apimgt.gateway.handlers.security.APISecurityException: Invalid JWT token

gqlws-repro-share.zip

Version

4.7.0

Environment Details (with versions)

No response