fix: canonical two-pass buffer allocation in rsa_crypt()#131
Merged
timlegge merged 1 commit intoMar 21, 2026
Merged
Conversation
On the OpenSSL 3.x path, rsa_crypt() pre-allocated the output buffer using EVP_PKEY_get_size() before calling the sizing pass. This meant: 1. A wasted alloc+free on the PSS padding early-exit path 2. A latent buffer overflow if the sizing call ever returned a value larger than EVP_PKEY_get_size() (not currently possible for RSA, but violates the API contract) 3. Inconsistency with sign(), which already uses the correct pattern Move the allocation into the version-specific blocks: - 3.x: allocate after the sizing call using the returned length - Pre-3.x: allocate with EVP_PKEY_get_size() as before (no sizing API) Initialize `to = NULL` so Safefree() in the error path is safe before allocation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
timlegge
approved these changes
Mar 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Use OpenSSL 3.x's two-pass allocation pattern in
rsa_crypt()— allocate after the sizing call, not before.Why
The function pre-allocated the output buffer with
EVP_PKEY_get_size()before calling the sizing pass. This:EVP_PKEY_get_size()sign(), which already uses the correct two-pass patternHow
CHECK_NEWinto the pre-3.x#elseblock (where there's no sizing API)Newx()after the sizing call using the returnedto_lengthto = NULLsoSafefree()in the error path is safe before allocationSafefree(to)before the PSS croakTesting
All 497 tests pass. The existing encrypt/decrypt/private_encrypt/public_decrypt tests with all padding modes exercise this code path thoroughly.
🤖 Generated with Claude Code
Quality Report
Changes: 1 file changed, 4 insertions(+), 3 deletions(-)
Code scan: clean
Tests: passed (OK)
Branch hygiene: clean
Generated by Kōan post-mission quality pipeline