SIGSEGV when using password with server_awaiting_accept mode

[mjurincic]

We're using ex_libsrt v0.1.6 with membrane_srt_plugin v0.1.1 to build a multi-stream SRT ingest server. We share a single ExLibSRT.Server across multiple streams using the server_awaiting_accept pattern from the docs.

Everything works great without a password. But as soon as we pass a passphrase to Server.start_link/3, the BEAM crashes with a SIGSEGV (exit code 139) when a client connects and we accept the connection.

How to reproduce

Start the server with a password:

{:ok, server} = ExLibSRT.Server.start_link("0.0.0.0", 9000, "devpassphrase1234")

Then wait for a connection and accept it:

receive do
  {:srt_server_connect_request, _address, _stream_id} ->
    :ok = ExLibSRT.Server.accept_awaiting_connect_request(server)
end

Connect with GStreamer using the matching passphrase:

gst-launch-1.0 -e \
  videotestsrc is-live=true ! x264enc tune=zerolatency ! mpegtsmux ! \
  srtsink uri="srt://127.0.0.1:9000?streamid=test&passphrase=devpassphrase1234&pbkeylen=16" \
  wait-for-connection=true

BEAM dies immediately. No Elixir error, no crash log — just gone. Exit code 139.

What I think is happening

Looking at server.cpp, OnNewConnection() sets SRTO_PASSPHRASE on the socket inside srt_listen_callback, then blocks on a condition variable for up to 1 second waiting for the Elixir side to accept. My guess is the blocking callback messes with libsrt's crypto handshake state — the key exchange needs back-and-forth that can't happen while the callback is stuck waiting.

Without a password the same blocking-accept flow works fine, so it's specifically the combination of password + the deferred accept pattern.

Environment

• ex_libsrt v0.1.6, membrane_srt_plugin v0.1.1
• Elixir 1.19.5 / OTP 27
• macOS 15.5 arm64

Workaround

We disabled the passphrase for now and rely on network-level security.

[varsill]

Hello! Thanks for reporting that!
Unfortunately I was unable to reproduce this problem locally but I am totally aware that there is a problem with hanging on the callback registered with srt_listen_callback while waiting for the connection acceptence.

I wonder if the SIGSEGV could be caused by a tiny bug we had in the codebase, fixed by this PR (I am aware that in normal circumstances it shouldn't result in sefgault, but since the code runs as a NIF and memory is managed by BEAM, various things can happen :D)
Could you try updating to ex_libsrt v0.1.7 and see if the SIGSEGV is still thrown? If not, perhaps some new logs appear, which would help us debug further?

Waiting for a connection acceptance 'on Elixir side' is indeed quite problematic with SRT - once I even opened an issue in SRT repo: Haivision/srt#3178 concerning that but I didn't follow their suggestion to do this kind of "two phase connection" as I didn't want client to be forced to connect this way.

When it comes to the issue you reported I don't see how the handshake state could get corrupted during the interruped Key Material (KM) exchange, but I don't deny it either.
From what I understand, the SRT handshake is split into two phases: INDUCTION and CONCLUSION, and the flow looks like that:

1. Client -> Server: INDUCTION request
2. Server -> Client: INDUCTION response
3. Client -> Server: CONCLUSION request (with KMREQ). Upon receiving CONCLUSION message, the server calls newConnection(), with the following synchronious steps:
   1. socket is created
   2. callback registered with srt_listen_callback is invoked
   3. acceptAndRespond() is invoked where KMREQ is being processed
4. Server -> Client: CONCLUSION response (with KMRSP)

If server's receiver thread is blocked (e.g. waiting on conditional variable) and the CONCLUSION response is not received by the client within 250ms, the client retransmits CONCLUSION request (with KMREQ). Then the server has two CONCLUSION request messages (and therefore KMREQ messages as well) for the same connection, but looking at https://github.com/Haivision/srt/blob/c3cfa17e2a77ac6900b35030b283ec4613a46ba3/srtcore/api.cpp#L656 it seems that it should be more or less discarding the second one 🤔

I think that perhaps it would be better to always accept the connection in the callback registered with srt_listen_callback and then give a chance to "truly" accept or reject it 'on Elixir side' (and rejection would mean termination of the already estabilished connection) - it wasn't implemented this way as we thought that it's better not to complete the handshake than to complete it and drop immediately afterwards - but perhaps there is no better way.