Fenrir fixes - WolfSSLKeyStore, WolfCryptPKIXCertPathValidator

[cconlon]

This PR resolves a batch of 8 findings from Fenrir. Changes are DoS hardening, input validation, integrity ordering, and a correctness fix. No public API changes. One new optional Security property, wolfjce.wks.maxEntrySize, is added to bound WKS KeyStore load allocations.

Changes

KeyStore load input validation and allocation bounds

• F-4354: bound the encoded entry length when loading a WKS KeyStore in engineLoad()
• F-4355: enforce the exact HMAC length when loading a WKS KeyStore in engineLoad()
• F-4357 / F-4358 / F-4360 / F-4361: bound the variable-length allocations in the WKS per-entry decoders (private key, secret key, chain certificates, certificate)

KeyStore integrity ordering

• F-4356: commit parsed WKS entries into the KeyStore only after the HMAC integrity check passes in engineLoad()

FIPS provider check consistency

• F-4363: use value equality for the FIPS signature provider check in WolfCryptPKIXCertPathValidator.engineValidate()

[Copilot]

Pull request overview

This PR hardens the WKS KeyStore load path against malformed inputs (allocation bounding + integrity-ordering) to address Fenrir findings, and fixes a correctness issue in the FIPS signature-provider check in WolfCryptPKIXCertPathValidator, without introducing public API changes (aside from one new optional Security property).

Changes:

• Add wolfjce.wks.maxEntrySize and enforce encoded-entry/inner-field length bounds during WKS load/decoding.
• Parse WKS entries into a temporary map and only commit them after the HMAC integrity check succeeds.
• Add regression tests for malformed WKS streams and update docs/JNI constants accordingly; fix String value-equality in FIPS signature provider check.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/test/java/com/wolfssl/provider/jce/test/WolfSSLKeyStoreTest.java	Adds regression tests for malformed WKS streams and integrity-ordering behavior.
src/main/java/com/wolfssl/provider/jce/WolfSSLKeyStore.java	Implements entry-size bounds, integrity-ordering (commit-after-HMAC), and per-entry decoder allocation limits.
src/main/java/com/wolfssl/provider/jce/WolfCryptPKIXCertPathValidator.java	Fixes FIPS signature-provider check to use value equality.
README_JCE.md	Documents the new wolfjce.wks.maxEntrySize Security property.
jni/include/com_wolfssl_provider_jce_WolfSSLKeyStore.h	Updates generated constant for default max entry size.
docs/design/WolfSSLKeyStore.md	Documents the new wolfjce.wks.maxEntrySize Security property in design notes.

Files not reviewed (1)

• jni/include/com_wolfssl_provider_jce_WolfSSLKeyStore.h: Generated file

Comments suppressed due to low confidence (2)

src/main/java/com/wolfssl/provider/jce/WolfSSLKeyStore.java:2435

• DataInputStream.read(byte[]) is not guaranteed to fill the buffer in one call, so this can spuriously fail (or behave inconsistently across InputStream types). Use readFully() to reliably read the entire encoded entry or throw EOFException.

                encodedEntry = new byte[encodedLen];
                bytesRead = dis.read(encodedEntry);
                if (bytesRead != encodedLen) {
                    throw new IOException(
                        "Unable to read total encoded entry byte array");
                }

src/main/java/com/wolfssl/provider/jce/WolfSSLKeyStore.java:2499

• Same issue as above: DataInputStream.read(byte[]) is not guaranteed to read the full HMAC in one call. Use readFully() to avoid short-read false failures on non-buffered streams.

            hmac = new byte[hmacLen];
            hmacLen = dis.read(hmac);
            if (hmacLen != hmac.length) {
                throw new IOException(
                    "Failed to read entire HMAC from WKS stream");
            }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.