Add SHA3 support

[jls5177]

This commit adds SHA3 support for use with the RSA PKCS#1 crypto algorithms. This is an odd PR as SoftHSM2 does not yet support SHA3 and Thales HSMs only support PKCS#1 v1.5 (no PSS support).

My company makes hardware devices that will support SHA3 hash algorithms with RSASSA-PSS to be future compatible. This change adds native support for RSASSA-PKCS1-v1_5 since the algorithm IDs are prefixed to the hash before sending to request to the HSM.

However, RSASSA-PSS is a little more involved. The solution I came up with is a hybrid approach where the EMSA-PSS encoding is done in solftware but the encryption/decryption of the encrypted message is performed by the HSM.

I validated this change using SoftHSM and Go 1.12.5 and all of the test cases passed locally.

[dmjones]

Thanks for the PR. Which HSM/libraries have you tested this with?

[jls5177]

I tested using SoftHSM2 v2.5 and my company uses the Thales nShield Connect XC HSM in production. We have been running this patchset on our production HSMs for over 8 months now without any issues.

[jls5177]

Sorry for the out of hand PR. This change is now based ontop of PR #41 to prevent merge conflict. However, there is no local branch so it looks like this change is pulling in additional commits from #41. Please ignore these additional commits and just focus on the following 2 commits:

78da155 feat: add RSA Hybrid solution to support SHA3 hash functions
f8a6ddc feat: add SHA3 hash support to RSA APIs

This will get cleaned up once PR #41 gets merged.

I created a local PR that will show just the 2 commits in my forked repo:
https://github.com/jls5177/crypto11/pull/1/files

[dmjones]

I created a local PR that will show just the 2 commits in my forked repo:
https://github.com/jls5177/crypto11/pull/1/files

That's helpful. I will add some comments there.

[Nicolas-Peiffer]

Thanks for this PR, and apologies it sat unanswered for so long — that's on us, not on
the contribution.

We're going to close it without merging.

The blocker is the hybrid PSS path, not SHA3. The PKCS#1 v1.5 half of this PR is
uncontroversial: the DigestInfo prefix is assembled host-side already, so adding the
SHA3 OIDs is just another table entry. The PSS half does something categorically
different — it performs EMSA-PSS encoding (MGF1, salt, DB masking) in Go and then
hands the finished encoded message to the token as a raw CKM_RSA_PKCS operation.

That inverts the property people use this library for. crypto11 is a thin binding
whose value is that the token performs the signature, under the token's own
mechanism policy. Once the padding is built in process memory:

• the salt and the encoded message are attacker-reachable on a compromised host, and
  a raw-signing primitive over caller-supplied blocks is a much broader capability
  than "sign with PSS";
• the key must be usable for raw CKM_RSA_PKCS, so the HSM can no longer distinguish
  a PSS signature from an arbitrary block signature. Any per-mechanism key policy on
  the token is defeated by construction — and many deployments (ours included)
  specifically forbid raw-sign keys;
• the library would be quietly making a security tradeoff on the caller's behalf,
  with no signal at the API boundary that this key's signatures were not produced the
  way the token thinks they were.

We'd rather return an error for a mechanism the token doesn't implement than silently
emulate it in software.

The good news: the native answer now exists. When this PR was written (2019),
SHA3 mechanisms weren't broadly available and SoftHSM2 had no SHA3 at all, so the
hybrid was a reasonable workaround for a real gap. That gap has since closed.
PKCS#11 v3.0 defines CKM_SHA3_224/256/384/512, the CKM_SHA3_*_RSA_PKCS[_PSS]
mechanisms, and CKG_MGF1_SHA3_*, and the binding this repo depends on
(eclipse-keypont/pkcs11-go) already exports all of them. So on a token with SHA3
support the change is small and stays entirely on the HSM:

• extend hashToPKCS11 in rsa.go with crypto.SHA3_224/256/384/512 →
  CKM_SHA3_* + CKG_MGF1_SHA3_*, which gets both PSS and OAEP for free;
• add the four SHA3 DigestInfo prefixes (OIDs 2.16.840.1.101.3.4.2.7–10) to
  pkcs1Prefix for the v1.5 path;
• gate the new tests with skipIfMechUnsupported, as the existing PSS tests do, so
  they no-op on tokens without SHA3 rather than failing the suite.

And crypto11@v2 will implement support for PKCS#11 v3.2.

---

In 2026 the big project for crypto11 is to add the support of PKCS#11 v3.2 and the support of the ML-KEM PQC algorithm:

• add support for pkcs11 v3.2 ml-kem #131
• We work on this branch mainly: pkcs11-v3.2-ml-kem.
• It will lead to a crypto11@v2, as there are many changes, including we are using a new PKCS11 implementation eclipse-keypont/pkcs11-go.
• Summary of the changes here: pkcs11-v3.2-ml-kem/CHANGELOG.md

The other big project for 2026 is giving + moving the crypto11 project under the Eclipse Foundation.