Skip to content

WebSocketServer SSL handshake gets stuck when client certificate is used #1500

@robert-s-ubi

Description

@robert-s-ubi

Describe the bug
When setting up a WebSocketServer with a WebSocketServerFactory that produces ServerSockets which want or need client certificate authentication, the WebSocketServer gets stuck in the handshake when a client with a valid client certificate connects.

What I could figure out so far: The code in SSLSocketChannel2 seems to get stuck, because the last read (presumably yielding the client certificate) leaves SSLEngine in the NEED_TASK state (presumably to validate the certificate), and the task leaves SSLEngine in the NEED_UNWRAP state. But since there is no further data to read, no further call to unwrap() is made, so the handshake never completes. Note that when the client certificate is invalid, the handshake is correctly rejected.

Hacking SSLSocketChannel2 to execute tasks synchronously in consumeDelegatedTasks() and after task execution checking for this state and calling sslEngine#unwrap() then makes the handshake complete, but I don't think that's a suitable solution.

Unfortunately, I don't understand the code well enough to come up with a proper solution.

To Reproduce
Steps to reproduce the behavior:

  1. Set up a WebSocketServer with a WebSocketServerFactory that produces ServerSockets which want/need a client certificate, and an SSLContext set up with a trust store for a client root certificate.
  2. Connect with a WebSocketClient with a WebSocketServerFactory that produces an SSLContext with a vald client certificate signed by the root certificate.
  3. Try to connect the WebSocketClient with the WebSocketServer
  4. Find that the handshake never completes

Example application to reproduce the issue
N/A

Expected behavior
The SSL handshake should successfully complete if the client sends a valid authentication certificate.

Debug log
N/A

Environment(please complete the following information):

  • Version used: 1.6.0
  • Java version: 8
  • Operating System and version: MacOS Tahoe 26.2
  • Endpoint Name and version: N/A
  • Link to your project: N/A

Additional context
N/A

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions