Skip to content

fix: multiple OpenSSL 3.x memory leaks#75

Merged
atoomic merged 1 commit into
cpan-authors:mainfrom
atoomic:koan.atoomic/fix-memleak-get-key-params
Mar 14, 2026
Merged

fix: multiple OpenSSL 3.x memory leaks#75
atoomic merged 1 commit into
cpan-authors:mainfrom
atoomic:koan.atoomic/fix-memleak-get-key-params

Conversation

@Koan-Bot
Copy link
Copy Markdown
Contributor

Summary

Fixes three distinct memory leak families in OpenSSL >= 3.0.0 code paths:

  • _get_key_parameters(): EVP_PKEY_get_bn_param() allocates new BIGNUMs (unlike pre-3.x getters which return internal const pointers), but cor_bn2sv() duplicates them via BN_dup() without freeing the originals — 8 BIGNUMs leaked per call
  • verify(): XSRETURN_NO/XSRETURN_YES returned immediately, bypassing EVP_MD_free() and EVP_PKEY_CTX_free() cleanup — leaked on every verify call
  • _new_key_from_parameters(): EVP_PKEY_CTX (pctx) was never freed on any path, test_ctx from EVP_PKEY_check() was never freed, and OSSL_PARAM_BLD/OSSL_PARAM were not freed in the public-key-only branch

Root cause

The OpenSSL 3.x migration changed ownership semantics: pre-3.x getter functions return internal pointers (caller must not free), while 3.x functions like EVP_PKEY_get_bn_param() allocate new objects (caller must free). The original migration missed several of these ownership changes.

Test plan

  • All 278 tests pass (including bignum tests that exercise _get_key_parameters())
  • Compiled with OpenSSL 3.6.1 on macOS
  • CI will validate against OpenSSL 1.1.1 (bullseye), 3.0.x (bookworm), 3.4.x (trixie)

🤖 Generated with Claude Code

… construction

Three distinct memory leak fixes for OpenSSL >= 3.0.0 code paths:

1. _get_key_parameters(): EVP_PKEY_get_bn_param() allocates new BIGNUMs
   (unlike pre-3.x getters which return internal pointers). cor_bn2sv()
   duplicates them via BN_dup() but the originals were never freed,
   leaking 8 BIGNUMs on every call.

2. verify(): XSRETURN_NO/XSRETURN_YES returned immediately, bypassing
   EVP_MD_free(md) and EVP_PKEY_CTX_free(ctx) cleanup. Restructured to
   capture verify result, free resources, then switch on the result.

3. _new_key_from_parameters():
   - EVP_PKEY_CTX (pctx) was never freed on any path (success or error)
   - EVP_PKEY_CTX from EVP_PKEY_check() (test_ctx) was never freed
   - OSSL_PARAM_BLD and OSSL_PARAM were not freed in the else branch
     (public-key-only path)
   Moved pctx/params_build/params declarations to PREINIT for proper
   scope, added cleanup on both success and error paths.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@timlegge
Copy link
Copy Markdown
Member

looks like there is a newer version of this one in another PR

@atoomic atoomic marked this pull request as ready for review March 12, 2026 02:19
@atoomic atoomic requested a review from timlegge March 12, 2026 02:19
@timlegge timlegge self-assigned this Mar 13, 2026
@Koan-Bot
Copy link
Copy Markdown
Contributor Author

@timlegge — correct. PR #75 has been superseded by individual, more focused PRs:

However, one fix from this PR is not yet covered elsewhere: the _get_key_parameters() BIGNUM freeing. On OpenSSL 3.x, EVP_PKEY_get_bn_param() allocates new BIGNUMs (unlike pre-3.x RSA_get0_*() which returns internal pointers), so the 8 BIGNUMs (n, e, d, p, q, dmp1, dmq1, iqmp) need to be freed after use. I'll track this separately.

This PR can be closed once that remaining leak is addressed in its own PR.

Copy link
Copy Markdown
Member

@timlegge timlegge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did

s/>= 0x30000000L/>= 0x90000000L/g
s/< 0x30000000L/< 0x50000000L/g

to force the code to take the old version of the api and I have to change padding.t to do :

if $major lt '5.0' && $pad eq 'sslv23'

The old code path works fine.

Approvgving

Comment thread RSA.xs
EVP_PKEY_CTX *pctx = EVP_PKEY_CTX_new_from_name(NULL, "RSA", NULL);
pctx = EVP_PKEY_CTX_new_from_name(NULL, "RSA", NULL);
CHECK_OPEN_SSL(pctx != NULL);
CHECK_OPEN_SSL(EVP_PKEY_fromdata_init(pctx) > 0);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be THROW(EVP_PKEY_fromdata_init(pctx) > 0);

Comment thread RSA.xs
BN_free(dmp1);
BN_free(dmq1);
BN_free(iqmp);
#endif
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch

Copy link
Copy Markdown
Member

@timlegge timlegge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - The code duplication with the switch should be sorted later though

@atoomic atoomic merged commit 2ac8fa5 into cpan-authors:main Mar 14, 2026
50 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants