fix: free EVP_MD and EVP_PKEY_CTX before switch in verify()

[toddr-bot]

What

Free EVP_MD and EVP_PKEY_CTX objects before the switch statement in verify() so they are actually reached.

Why

On OpenSSL 3.x, EVP_MD_free(md) and EVP_PKEY_CTX_free(ctx) were placed after the switch, but every branch exits the function: XSRETURN_YES, XSRETURN_NO, or croak via CHECK_OPEN_SSL(0). The cleanup was dead code — every verify() call leaked both objects.

How

Capture the EVP_PKEY_verify() return value into verify_result, call EVP_MD_free(md) and EVP_PKEY_CTX_free(ctx), then switch on the stored result. The legacy (pre-3.x) RSA_verify path is unaffected — it doesn't allocate these objects.

Testing

Full test suite passes (318 tests, 10 test files).

🤖 Generated with Claude Code

---

Quality Report

Changes: 1 file changed, 4 insertions(+), 5 deletions(-)

Code scan: clean

Tests: passed (OK)

Branch hygiene: clean

Generated by Kōan post-mission quality pipeline

[atoomic]

@toddr-bot rebase

[atoomic]

@Koan-Bot review

[Koan-Bot]

PR Review — fix: free EVP_MD and EVP_PKEY_CTX before switch in verify()

The fix is correct: EVP_MD_free(md) and EVP_PKEY_CTX_free(ctx) were dead code after the switch because every branch (XSRETURN_YES, XSRETURN_NO, CHECK_OPEN_SSL(0)) performs a longjmp. Capturing the result into verify_result and freeing before the switch is the right approach. The rebased code on disk goes further by adding THROW/goto err for error-path cleanup and moving the declaration to PREINIT, which resolves the C89 compatibility concern in the original diff. Merge-ready after confirming the rebase is reflected in the final diff.

---

🟡 Important

1. Mid-block int declaration breaks C89 compatibility (RSA.xs, L1077)
The diff introduces int verify_result = EVP_PKEY_verify(...) mid-block inside the #if directive. XS-generated C code may be compiled in C89 mode on older toolchains, where declarations must precede statements.

Move the declaration into the PREINIT: section (which the rebased code on disk already does at line 1101). If the rebase resolved this, this comment can be disregarded — but the diff as shown still contains the mid-block form.

int verify_result = EVP_PKEY_verify(ctx, sig, sig_length, digest, get_digest_length(p_rsa->hashMode));

🟢 Suggestions

1. Pre-existing error-path leaks from CHECK_OPEN_SSL not addressed (RSA.xs, L1077-1080)
The CHECK_OPEN_SSL calls before EVP_PKEY_verify (e.g., CHECK_OPEN_SSL(EVP_PKEY_CTX_set_rsa_pss_saltlen(...))) croak immediately on failure, leaking both ctx and md. This is a pre-existing issue and out of scope for this PR, but worth noting. The rebased code on disk uses the THROW/goto err pattern with proper cleanup (lines 1123-1148), which resolves this — confirming the rebase incorporated fixes from PR #81.

---

Checklist

• Resource leaks on success path fixed
• No hardcoded secrets
• Error-path cleanup — addressed by rebase (THROW/goto err pattern)
• C89 declaration compatibility — warning quietened critic to -3, added test #1 — resolved by rebase moving decl to PREINIT

---

Summary

The fix is correct: EVP_MD_free(md) and EVP_PKEY_CTX_free(ctx) were dead code after the switch because every branch (XSRETURN_YES, XSRETURN_NO, CHECK_OPEN_SSL(0)) performs a longjmp. Capturing the result into verify_result and freeing before the switch is the right approach. The rebased code on disk goes further by adding THROW/goto err for error-path cleanup and moving the declaration to PREINIT, which resolves the C89 compatibility concern in the original diff. Merge-ready after confirming the rebase is reflected in the final diff.

---

Automated review by Kōan

[toddr-bot]

Rebase: fix: free EVP_MD and EVP_PKEY_CTX before switch in verify()

Branch koan.toddr.bot/fix-verify-memleak rebased onto main and force-pushed.

Review feedback was analyzed and applied.

Actions

• Resolved merge conflicts (1 round(s))
• Rebased koan.toddr.bot/fix-verify-memleak onto upstream/main
• Force-pushed koan.toddr.bot/fix-verify-memleak to origin

---

Automated by Kōan