fix(pty): use OpenSSL crypto backend on Windows for curve25519 KEX

[srstack]

Problem

SSH sessions from a Windows build of otty fail during the handshake against modern SSH servers (OpenSSH >= 10) with:

ssh error: [Session(-5)] Unable to exchange encryption keys

Root cause

The server-side log (OpenSSH 10.0, Debian 13) shows what a Windows build of libssh2 actually offers:

Unable to negotiate with <ip>: no matching key exchange method found.
Their offer: diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,
diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,
diffie-hellman-group1-sha1,diffie-hellman-group-exchange-sha1 [preauth]

The offer is missing curve25519-sha256 and ecdh-sha2-nistp*. Reason: in libssh2-sys's build.rs, Windows targets default to the WinCNG crypto backend unless the openssl-on-win32 feature is enabled. WinCNG only supports finite-field DH key exchange. OpenSSH 10 servers by default offer only mlkem768x25519, sntrup761x25519, curve25519 and ecdh-sha2-nistp* — zero intersection with WinCNG's offer, so the KEX negotiation fails.

This affects any Rust project using ssh2 with default features on Windows connecting to OpenSSH >= 10 servers, not just otty.

Fix

Enable ssh2's openssl-on-win32 feature. The crate already forces a vendored OpenSSL build (see the comment above the openssl dependency), so libssh2 now compiles against the same vendored OpenSSL on Windows and offers curve25519-sha256 / ecdh-sha2-nistp* again. On unix this feature is a no-op (the OpenSSL branch is already taken there).

Verification

• nm on the Windows libssh2.a after the change: contains _libssh2_curve25519_new / _libssh2_curve25519_gen_k and 70+ EVP_* symbol references (previously none — WinCNG build)
• End-to-end on a Windows host: otty now connects to an OpenSSH 10.0 server and reaches the shell (previously Session(-5) at handshake)
• cargo clippy -p otty-pty --all-targets --all-features -- -D warnings, cargo test -p otty-pty --all-features, cargo deny check all green

[Copilot]

Pull request overview

This PR fixes SSH handshake failures on Windows by ensuring the ssh2/libssh2-sys build uses the OpenSSL crypto backend (rather than the WinCNG backend), restoring support for modern key exchange algorithms required by OpenSSH >= 10 defaults.

Changes:

• Enable ssh2’s openssl-on-win32 feature for Windows builds.
• Add inline documentation explaining why this feature is required (OpenSSH >= 10 KEX compatibility).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[kemokempo]

hm, what do you mean when you tell windows build? otty does not have the windows support right now