Current Behavior
load_privkey() in src/eng_front.c calls ENGINE_set_default_string(engine, "PKEY_CRYPTO") to work around the OpenSSL segfault in openssl/openssl#23063 (3.0.12–3.0.13 / 3.1.4–3.1.5 / 3.2.0–3.2.1), but never undoes it. This makes the engine the process-wide global default for RSA/EC crypto for the rest of the process, even though the loaded key already has its own explicit engine binding a few lines later via EVP_PKEY_set1_engine().
Consequence: any unrelated code later asking OpenSSL for a generic "RSA" implementation via the modern provider API gets silently routed through this leftover engine instead, because int_ctx_new() checks the global engine table before providers or propquery. The engine doesn't implement the modern fromdata API, so it fails with:
error:03000096:digital envelope routines:fromdata_init:operation not supported for this keytype:../crypto/evp/pmeth_gn.c:354
I hit this with a TPM-backed PKCS#11 token in wpa_supplicant EAP-TLS: after this engine loads the WiFi client key, tpm2-pkcs11's own internal tpm2-tss code — needed to authorize the same TPM key for the TLS handshake — does its own unrelated Esys_StartAuthSession() RSA operation, which gets hijacked by the still-registered engine and fails, breaking the connection.
Related in spirit to #672/#675 (same theme: a PKCS#11 key load pollutes global OpenSSL state, also flagged there for wpa_supplicant + TPM2) but a different code path — those are about EVP_PKEY_meth_add0() in the newer pkcs11_pkey_method_rsa_new() (0.4.18+), while this is the older ENGINE_set_default_string() in load_privkey() (~0.4.13+, still shipped by Ubuntu's 0.4.12-based package). Neither issue mentions this code path, and #675's guard doesn't cover it.
Expected Behavior
Loading one private key via the ENGINE (to work around the OpenSSL segfault) should not leave the engine permanently registered as the global default for RSA/EC crypto for the rest of the process — the registration should be scoped to, at most, the window it's actually protecting (during ctx_load_privkey()), and undone once the key is safely loaded and has its own explicit engine binding.
Steps To Reproduce & Observed Output
- Full command line used:
nmcli connection up <eap-tls-wifi-profile> (NetworkManager driving wpa_supplicant for a corporate EAP-TLS 802.1x WiFi connection; equivalent to a plain wpa_supplicant config with key_mgmt=WPA-EAP, eap=TLS, private_key="pkcs11:model=...;object=Device Certificate;type=private", client_cert="/path/to/client.cer" — no explicit engine=1/engine_id set; the classic PKCS#11 engine is auto-selected by wpa_supplicant because private_key starts with pkcs11:)
- Full stdout/stderr output (from
journalctl -u wpa_supplicant):
wlp0s20f3: Associated with e6:55:b8:93:55:81
wlp0s20f3: CTRL-EVENT-EAP-STARTED EAP authentication started
wlp0s20f3: CTRL-EVENT-EAP-PROPOSED-METHOD vendor=0 method=13
Workaround for OpenSSL 3.0.13 30 Jan 2024 enabled
wlp0s20f3: CTRL-EVENT-EAP-METHOD EAP vendor 0 method 13 (TLS) selected
wlp0s20f3: CTRL-EVENT-EAP-PEER-CERT depth=2 subject='.../CN=Corp Root CA' ...
wlp0s20f3: CTRL-EVENT-EAP-PEER-CERT depth=1 subject='.../CN=Corp Issuing CA' ...
wlp0s20f3: CTRL-EVENT-EAP-PEER-CERT depth=0 subject='.../CN=radius.example.com' ...
ERROR:esys_crypto:src/tss2-esys/esys_crypto_ossl.c:723:iesys_cryptossl_pk_encrypt() ErrorCode (0x00070001) Could not create rsa key.
ERROR:esys:src/tss2-esys/esys_iutil.c:524:iesys_compute_encrypted_salt() During encryption. ErrorCode (0x00070001)
ERROR:esys:src/tss2-esys/api/Esys_StartAuthSession.c:226:Esys_StartAuthSession_Async() Error in parameter encryption. ErrorCode (0x00070001)
ERROR:esys:src/tss2-esys/api/Esys_StartAuthSession.c:113:Esys_StartAuthSession() Error in async function ErrorCode (0x00070001)
ERROR: Esys_StartAuthSession: esapi:Catch all for all errors not otherwise specified
ERROR: Could not start Auth Session with the TPM.
ERROR: Error unsealing wrapping key
SSL: SSL3 alert: write (local SSL3 detected an error):fatal:internal error
OpenSSL: openssl_handshake - SSL_connect error:03000096:digital envelope routines::operation not supported for this keytype
OpenSSL: pending error: error:03000096:digital envelope routines::operation not supported for this keytype
OpenSSL: pending error: error:43000005:PKCS#11 module::General Error
OpenSSL: pending error: error:0A080006:SSL routines::EVP lib
Note the server certificate chain (the RADIUS server's) validates fine — the failure is entirely on the client side, in the TPM-session-setup step needed to use the already-successfully-loaded client key.
To confirm the exact underlying OpenSSL failure (which tpm2-tss's own logging doesn't surface — it only logs "Could not create rsa key" without the OpenSSL error queue), I built a small LD_PRELOAD shim intercepting EVP_PKEY_CTX_new_from_name() / EVP_PKEY_fromdata_init() and dumping ERR_print_errors_fp() on failure:
[shim] EVP_PKEY_CTX_new_from_name(name=RSA, propquery=(null)) OK, libctx=0x6545320f6010
[shim] EVP_PKEY_fromdata_init FAILED (ret=-2), libctx=0x6545320f6010
[shim] providers active on this libctx:
[shim] active provider: default
[shim] full OpenSSL error queue:
error:03000096:digital envelope routines:fromdata_init:operation not supported for this keytype:../crypto/evp/pmeth_gn.c:354:
EVP_PKEY_CTX_new_from_name() succeeds and only the default provider is active on that library context — yet fromdata_init() fails, because the context's keymgmt ends up NULL. Reading crypto/evp/pmeth_lib.c's int_ctx_new() confirms why: ENGINE_get_pkey_meth_engine(NID_rsaEncryption) returns the globally-registered pkcs11 engine, so the entire provider-fetch branch (if (e == NULL && ...) { keymgmt = EVP_KEYMGMT_fetch(...); }) is skipped — this happens unconditionally, regardless of any explicit propquery passed to EVP_PKEY_CTX_new_from_name(). I confirmed this by re-testing with EVP_PKEY_CTX_new_from_name(libctx, "RSA", "provider=default") explicitly — identical failure, since propquery is never even consulted once an engine has claimed the NID.
Confirmed via /proc/<wpa_supplicant-pid>/maps during a live connection attempt that /usr/lib/x86_64-linux-gnu/engines-3/pkcs11.so (this library) is loaded in-process alongside libtpm2_pkcs11.so, libp11-kit.so, and opensc-pkcs11.so.
Environment
- Operating system and version: Ubuntu 24.04 (Noble)
- Architecture: x86_64
- PKCS#11 module used:
libtpm2_pkcs11.so.1.9.0 (tpm2-pkcs11)
- Token / HSM type: TPM 2.0, Nuvoton NPCT75x (fTPM/dTPM)
Versions
- libp11 built from:
- PKCS#11 module and version: tpm2-pkcs11 1.9.0-0.2build4 (
libtpm2-pkcs11-1), built against tpm2-tss (libtss2-esys-3.0.2-0t64) 4.0.1-7.1ubuntu5.1
openssl version -a:
OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024)
built on: Wed Jul 29 16:55:30 2026 UTC
platform: debian-amd64
OPENSSLDIR: "/usr/lib/ssl"
ENGINESDIR: "/usr/lib/x86_64-linux-gnu/engines-3"
MODULESDIR: "/usr/lib/x86_64-linux-gnu/ossl-modules"
I haven't been able to test against current upstream master yet — happy to if it'd help, though the specific ENGINE_set_default_string code and its version-range gate (0x300000c0–0x300000d0 etc.) still appears to be present there as of the #675 discussion, unaffected by the PKCS11_FLAG_NO_METHODS guard (which the maintainers note only covers the provider path, not the engine path).
Configuration / Settings
- No custom OpenSSL configuration is involved — I confirmed this by testing with
default_properties = provider=default and by explicitly activating both default and a pkcs11 OpenSSL provider (unrelated to this engine) in openssl.cnf; neither had any effect, which is expected since the engine-table check in int_ctx_new() happens before any of that is consulted.
wpa_supplicant/NetworkManager config: EAP-TLS, private_key set to a pkcs11: URI, no explicit engine=1/engine_id (auto-selected as pkcs11 by wpa_supplicant itself since the private_key string has the pkcs11: prefix).- No
OPENSSL_CONF override; system default /etc/ssl/openssl.cnf.
Files / Artifacts
Fix (see attached eng_front_unregister_default.patch): in load_privkey(), right before return pkey; (after the key has been loaded and given its own explicit engine binding via EVP_PKEY_set1_engine()), add calls to ENGINE_unregister_pkey_meths(engine) and ENGINE_unregister_pkey_asn1_meths(engine), gated behind the same OpenSSL-version check as the ENGINE_set_default_string() call above it, and run regardless of whether the key load succeeded. This undoes the temporary global default registration once it's no longer needed.
I built this against Ubuntu's exact libp11 0.4.12-1.1ubuntu1 source (in a clean container matching the target system) and confirmed end-to-end on real hardware: the WiFi connection now succeeds — full EAP-TLS handshake, DHCP lease, live network reachability — with no regression to the original segfault protection (that crash window is entirely before these new lines run).
Happy to attach the built .deb or open a PR with this patch if useful.
Anything else
Related: #672, #675 (same underlying theme — a PKCS#11 key load leaving global OpenSSL crypto-method state polluted — different code path/mechanism, and #675's fix doesn't cover this one).
Upstream OpenSSL bug this workaround exists for: openssl/openssl#23063.
eng_front_unregister_default.patch
Current Behavior
load_privkey()insrc/eng_front.ccallsENGINE_set_default_string(engine, "PKEY_CRYPTO")to work around the OpenSSL segfault in openssl/openssl#23063 (3.0.12–3.0.13 / 3.1.4–3.1.5 / 3.2.0–3.2.1), but never undoes it. This makes the engine the process-wide global default for RSA/EC crypto for the rest of the process, even though the loaded key already has its own explicit engine binding a few lines later viaEVP_PKEY_set1_engine().Consequence: any unrelated code later asking OpenSSL for a generic
"RSA"implementation via the modern provider API gets silently routed through this leftover engine instead, becauseint_ctx_new()checks the global engine table before providers orpropquery. The engine doesn't implement the modernfromdataAPI, so it fails with:I hit this with a TPM-backed PKCS#11 token in
wpa_supplicantEAP-TLS: after this engine loads the WiFi client key,tpm2-pkcs11's own internaltpm2-tsscode — needed to authorize the same TPM key for the TLS handshake — does its own unrelatedEsys_StartAuthSession()RSA operation, which gets hijacked by the still-registered engine and fails, breaking the connection.Related in spirit to #672/#675 (same theme: a PKCS#11 key load pollutes global OpenSSL state, also flagged there for wpa_supplicant + TPM2) but a different code path — those are about
EVP_PKEY_meth_add0()in the newerpkcs11_pkey_method_rsa_new()(0.4.18+), while this is the olderENGINE_set_default_string()inload_privkey()(~0.4.13+, still shipped by Ubuntu's 0.4.12-based package). Neither issue mentions this code path, and #675's guard doesn't cover it.Expected Behavior
Loading one private key via the ENGINE (to work around the OpenSSL segfault) should not leave the engine permanently registered as the global default for RSA/EC crypto for the rest of the process — the registration should be scoped to, at most, the window it's actually protecting (during
ctx_load_privkey()), and undone once the key is safely loaded and has its own explicit engine binding.Steps To Reproduce & Observed Output
nmcli connection up <eap-tls-wifi-profile>(NetworkManager drivingwpa_supplicantfor a corporate EAP-TLS 802.1x WiFi connection; equivalent to a plainwpa_supplicantconfig withkey_mgmt=WPA-EAP,eap=TLS,private_key="pkcs11:model=...;object=Device Certificate;type=private",client_cert="/path/to/client.cer"— no explicitengine=1/engine_idset; the classic PKCS#11 engine is auto-selected bywpa_supplicantbecauseprivate_keystarts withpkcs11:)journalctl -u wpa_supplicant):Note the server certificate chain (the RADIUS server's) validates fine — the failure is entirely on the client side, in the TPM-session-setup step needed to use the already-successfully-loaded client key.
To confirm the exact underlying OpenSSL failure (which
tpm2-tss's own logging doesn't surface — it only logs "Could not create rsa key" without the OpenSSL error queue), I built a smallLD_PRELOADshim interceptingEVP_PKEY_CTX_new_from_name()/EVP_PKEY_fromdata_init()and dumpingERR_print_errors_fp()on failure:EVP_PKEY_CTX_new_from_name()succeeds and only thedefaultprovider is active on that library context — yetfromdata_init()fails, because the context'skeymgmtends up NULL. Readingcrypto/evp/pmeth_lib.c'sint_ctx_new()confirms why:ENGINE_get_pkey_meth_engine(NID_rsaEncryption)returns the globally-registeredpkcs11engine, so the entire provider-fetch branch (if (e == NULL && ...) { keymgmt = EVP_KEYMGMT_fetch(...); }) is skipped — this happens unconditionally, regardless of any explicitpropquerypassed toEVP_PKEY_CTX_new_from_name(). I confirmed this by re-testing withEVP_PKEY_CTX_new_from_name(libctx, "RSA", "provider=default")explicitly — identical failure, sincepropqueryis never even consulted once an engine has claimed the NID.Confirmed via
/proc/<wpa_supplicant-pid>/mapsduring a live connection attempt that/usr/lib/x86_64-linux-gnu/engines-3/pkcs11.so(this library) is loaded in-process alongsidelibtpm2_pkcs11.so,libp11-kit.so, andopensc-pkcs11.so.Environment
libtpm2_pkcs11.so.1.9.0(tpm2-pkcs11)Versions
libengine-pkcs11-openssl0.4.12-1.1ubuntu1 (source packagelibp110.4.12-1.1ubuntu1, with Canonical'spkcs11-engine-openssl3-load-privkey-segfault-workaround.patchbackporting theENGINE_set_default_stringworkaround from upstream 0.4.13)libtpm2-pkcs11-1), built against tpm2-tss (libtss2-esys-3.0.2-0t64) 4.0.1-7.1ubuntu5.1openssl version -a:I haven't been able to test against current upstream
masteryet — happy to if it'd help, though the specificENGINE_set_default_stringcode and its version-range gate (0x300000c0–0x300000d0etc.) still appears to be present there as of the #675 discussion, unaffected by thePKCS11_FLAG_NO_METHODSguard (which the maintainers note only covers the provider path, not the engine path).Configuration / Settings
default_properties = provider=defaultand by explicitly activating bothdefaultand apkcs11OpenSSL provider (unrelated to this engine) inopenssl.cnf; neither had any effect, which is expected since the engine-table check inint_ctx_new()happens before any of that is consulted.wpa_supplicant/NetworkManager config: EAP-TLS,private_keyset to apkcs11:URI, no explicitengine=1/engine_id(auto-selected aspkcs11bywpa_supplicantitself since the private_key string has thepkcs11:prefix).OPENSSL_CONFoverride; system default/etc/ssl/openssl.cnf.Files / Artifacts
Fix (see attached
eng_front_unregister_default.patch): inload_privkey(), right beforereturn pkey;(after the key has been loaded and given its own explicit engine binding viaEVP_PKEY_set1_engine()), add calls toENGINE_unregister_pkey_meths(engine)andENGINE_unregister_pkey_asn1_meths(engine), gated behind the same OpenSSL-version check as theENGINE_set_default_string()call above it, and run regardless of whether the key load succeeded. This undoes the temporary global default registration once it's no longer needed.I built this against Ubuntu's exact
libp110.4.12-1.1ubuntu1 source (in a clean container matching the target system) and confirmed end-to-end on real hardware: the WiFi connection now succeeds — full EAP-TLS handshake, DHCP lease, live network reachability — with no regression to the original segfault protection (that crash window is entirely before these new lines run).Happy to attach the built
.debor open a PR with this patch if useful.Anything else
Related: #672, #675 (same underlying theme — a PKCS#11 key load leaving global OpenSSL crypto-method state polluted — different code path/mechanism, and #675's fix doesn't cover this one).
Upstream OpenSSL bug this workaround exists for: openssl/openssl#23063.
eng_front_unregister_default.patch