Address CVE-2023-39327, stop excessive loop errors

[xcawolfe-amzn]

Attempts to resolve #1472

Prevents excessive looping when consuming packets when errors are hit.

Validated with the steps below

Testing

create bigloop file to validate the issue

ÿOÿQ�,�����d����������	��ü������������������B�ÿR����ÿ��������"ÿ\��@@HHPÿd�-��Creator: AV-J2K (���Vator: AV-J2K (c) iotÿ��
��������ÿ�ÿÿOÿQ

Baseline

This establishes that there is an issue and that it breaks when the loop is excessive

Create a docker container from amazon linux 2023

Test script - test_baseline.sh

#!/bin/bash
set -e

echo "=== CVE-2023-39327 Baseline Test ==="
echo "Testing AL2023 baseline openjpeg2 with malicious PoC file"
echo ""

echo "[1] Installed version:"
rpm -q openjpeg2 openjpeg2-tools

# Test with the PoC file
echo ""
echo "[2] Testing with CVE-2023-39327 PoC file..."
echo "Running: timeout 10 opj2_decompress -i /test/bigloop -o /tmp/output.raw 2>&1 | wc -l"
echo ""

# Count total lines of output in 10 seconds
LINES=$(timeout 10 opj2_decompress -i /test/bigloop -o /tmp/output.raw 2>&1 | wc -l || true)

echo "Total output lines in 10 seconds: $LINES"
echo ""

if [ "$LINES" -gt 10000 ]; then
    echo "VULNERABLE: Excessive output detected ($LINES lines in 10s)"
    echo "The baseline version exhibits CVE-2023-39327 - infinite loop printing warnings."
    echo "This can consume CPU and fill logs indefinitely."
else
    echo "Output: $LINES lines in 10 seconds"
fi

Dockerfile

FROM public.ecr.aws/amazonlinux/amazonlinux:2023

# Install baseline openjpeg2 from AL2023 repos (vulnerable version)
RUN dnf install -y openjpeg2 openjpeg2-tools

# Copy test files
COPY bigloop /test/bigloop
COPY test_baseline.sh /test/test_baseline.sh

# Make script executable
RUN chmod +x /test/test_baseline.sh

# Run the test
CMD ["/test/test_baseline.sh"]

Run the container

# Build baseline test container
docker build -f Dockerfile.baseline -t openjpeg2-baseline-test .
# Run baseline test (should timeout showing vulnerability)
docker run --rm openjpeg2-baseline-test

see that there are many loop iterations and time out is hit

Fixed

Test scipt to validate fix - test_cve_fix.sh

#!/bin/bash
set -e

echo "=== CVE-2023-39327 Test Script ==="
echo "Testing openjpeg2 with malicious PoC file"
echo ""

# Install the patched RPM
echo "[1] Installing patched openjpeg2 package..."
dnf install -y /rpms/openjpeg2-2.5.2-*.x86_64.rpm /rpms/openjpeg2-tools-2.5.2-*.x86_64.rpm 2>&1 | tail -5

# Check version
echo ""
echo "[2] Installed version:"
rpm -q openjpeg2 openjpeg2-tools

# Test with the PoC file
echo ""
echo "[3] Testing with CVE-2023-39327 PoC file (should fail fast, not loop)..."
echo "Running: timeout 10 opj2_decompress -i /test/bigloop -o /tmp/output.raw"
echo ""

# Run with timeout - should fail quickly with error, not timeout
if timeout 10 opj2_decompress -i /test/bigloop -o /tmp/output.raw 2>&1; then
    echo "ERROR: Command succeeded when it should have failed!"
    exit 1
else
    EXIT_CODE=$?
    if [ $EXIT_CODE -eq 124 ]; then
        echo "FAIL: Command timed out - infinite loop detected!"
        echo "The CVE fix is NOT working properly."
        exit 1
    else
        echo "SUCCESS: Command failed quickly (exit code: $EXIT_CODE)"
        echo "The CVE fix is working - no infinite loop detected."
        exit 0
    fi
fi

Docker file

FROM public.ecr.aws/amazonlinux/amazonlinux:2023

# Copy test files
COPY bigloop /test/bigloop
COPY test_cve_fix.sh /test/test_cve_fix.sh
COPY ../<path_to_rpms_with_fix>/ /rpms/

# Make script executable
RUN chmod +x /test/test_cve_fix.sh

# Run the test
CMD ["/test/test_cve_fix.sh"]

Run and build the tests

# Build test container with patched RPMs
docker build -f Dockerfile.test -t openjpeg2-cve-test .

# Run test (should fail fast with loop detection)
docker run --rm openjpeg2-cve-test

See that the timeout is not hit

[rouault]

From where does the hardcoded 100000 packet limit comes from ? Couldn't it be reached for large valid images ?

[xcawolfe-amzn]

From where does the hardcoded 100000 packet limit comes from ? Couldn't it be reached for large valid images ?

It was a "big" number that seemed reasonable, but yes in theory it could be reached for very large images....
hmm maybe make it bigger? or configurable?

[rouault]

I doubt making this configurable would be meaningful to 95% + of users who have no idea of the internals of JPEG2000 and in particular what a packet is. As far as making it bigger, there is perhaps some correlation with other JPEG2000 parameters (image or probably tile size, number of resoutions, number of quality layers), but that should be carefully studied. More globally there are tons of ways where JPEG2000 parameters can be abused to create huge RAM or CPU consumption. Ideally we'd have some global strategy to deal with that, but I can't think of one

[xcawolfe-amzn]

there is perhaps some correlation with other JPEG2000 parameters (image or probably tile size, number of resoutions, number of quality layers), but that should be carefully studied. More globally there are tons of ways where JPEG2000 parameters can be abused to create huge RAM or CPU consumption.

This makes sense to me, hmmm we'll have to take a looks and see what might work. Ill convert this PR back to draft and take a look at what parameters we could measure and how to intake that data in a secure and effective manner. Thanks for taking a look @rouault!

[sebras]

@rouault

More globally there are tons of ways where JPEG2000 parameters can be abused to create huge RAM or CPU consumption. Ideally we'd have some global strategy to deal with that, but I can't think of one

One way to deal with hugh RAM consumption is to provide API support for a custom allocator.

If OpenJPEG had an interface similar to the one below, then opj_malloc(), opj_realloc(), opj_free() etc, could be implemented in terms of it. Naturally, OpenJPEG should provide a default allocator based on malloc(), realloc() and free() in case the program doesn't call opj_set_allocator() at all.

typedef struct opj_alloc_private {
        void *user;

        /* Standard allocator interface */
        void *(*malloc)(void * p_user, size_t size);
        void *(*calloc)(void * p_user, size_t num, size_t size);
        void *(*realloc)(void * p_user, void * p_old_ptr, size_t size);
        void (*free)(void * p_user, void * p_ptr);

        /* Optional interface for aligned allocations, opj_aligned_malloc() and friends 
         * may gracefully fall back to an implementation based on the standard interface
         * similar to opj_aligned_alloc_n()'s generic aligned malloc implementation.
         */
        void *(*aligned_malloc)(void * p_user, size_t alignment, size_t size);
        void *(*aligned_calloc)(void * p_user, size_t alignment, size_t num, size_t size);
        void *(*aligned_realloc)(void * p_user, size_t alignment, void * p_old_ptr, size_t size);
        void (*aligned_free)(void * p_user, size_t alignment, void * p_ptr);
}
opj_alloc_private_t;

OPJ_BOOL opj_set_allocator(opj_codec_t * p_codec, opj_alloc_t * p_alloc);

The user and p_user pointers are just pointers for the custom allocator; in case it needs a handle to its state or other data structure.

This way, if an program wants to restrict memory usage, it can implement its own allocator where malloc(), realloc(), etc. return NULL if a chosen memory threshold is reached. That threshold would be decided by the program, not by the OpenJPEG library itself. It wouldn't require messing with ulimit() and such. And the program can decide to limit memory usage related to J2K decoding but allow more (or less) memory to be dedicated to other matters.

You have probably consider this before, but it would still be interesting to hear your thoughts on it in that case. :)