Skip to content

Fix unchecked integer overflow in encoder UV plane allocation#3233

Open
jortles wants to merge 1 commit into
AOMediaCodec:mainfrom
jortles:fix/encoder-overflow-checks
Open

Fix unchecked integer overflow in encoder UV plane allocation#3233
jortles wants to merge 1 commit into
AOMediaCodec:mainfrom
jortles:fix/encoder-overflow-checks

Conversation

@jortles
Copy link
Copy Markdown
Contributor

@jortles jortles commented May 29, 2026

Summary

  • Add pre-multiplication overflow checks to codec_avm.c and codec_svt.c for UV plane size computations, consistent with the existing safe pattern in avifImageAllocatePlanes() (avif.c:435-441)
  • Add overflow check in avifImageCopyProperties() before numProperties allocation, consistent with avifImagePushProperty() (avif.c:387)

Details

Three sites compute allocation sizes via unchecked integer multiplication, unlike their parallel functions which validate before multiplying:

codec_avm.c:919channelSize * monoUVWidth (uint32_t) and monoUVHeight * monoUVRowBytes (size_t) used for avifAlloc() without overflow checks. The parallel code in avifImageAllocatePlanes() checks width > UINT32_MAX / channelSize and height > PTRDIFF_MAX / fullRowBytes before the same operations.

codec_svt.c:286uvWidth * bytesPerPixel (uint32_t) and uvHeight * uvRowBytes (size_t) computed without pre-multiplication validation. The existing post-hoc check (uvSize > UINT32_MAX / 2) cannot detect a wrap that already occurred.

avif.c:237numProperties * sizeof(...) passed to avifAlloc() without an overflow guard. avifImagePushProperty() in the same file already checks numProperties < SIZE_MAX / sizeof(avifImageItemProperty) before an identical allocation.

Test plan

  • Builds cleanly with cmake --build . --target avif_obj (zero warnings)
  • Existing test suite passes (CI)

Add pre-multiplication overflow checks to codec_avm.c and codec_svt.c
for UV plane size computations, consistent with the existing safe
pattern in avifImageAllocatePlanes() (avif.c:435-441). Also add an
overflow check in avifImageCopyProperties() before the numProperties
allocation, consistent with avifImagePushProperty() (avif.c:387).

Without these checks, crafted image dimensions can silently wrap the
uint32_t multiplication, leading to undersized allocations.

Signed-off-by: Anthony Hurtado <amhurtado@pm.me>
Copilot AI review requested due to automatic review settings May 29, 2026 18:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant