Skip to content

Reach the >4 GiB match arm with a test [#331] - #337

Merged
iderex merged 1 commit into
mainfrom
issue/331
Aug 9, 2026
Merged

Reach the >4 GiB match arm with a test [#331]#337
iderex merged 1 commit into
mainfrom
issue/331

Conversation

@iderex

@iderex iderex commented Aug 9, 2026

Copy link
Copy Markdown
Owner

Closes #331.

The arm and why it was unreachable

src/lz4_decode.cuh runs the closed-form overlap gather at two arithmetic
widths and picks the 32-bit one under seq.match_len <= UINT32_MAX. Nothing
in the suite allocated the 4 GiB a match past 2^32 needs, so nothing ever
selected the 64-bit arm, and the guard could be deleted without a single test
noticing.

What lands

tests/huge_match_gpu.cu. One valid LZ4 block: three literals, one match of
2^32 + 4096 bytes at offset 3, a five-byte literals-only tail. Destination
capacity 4294971400, source 16843037 bytes, almost all of it the match-length
continuation run.

No 4 GiB comparison is needed. The output is ABC repeating and 2^32 mod 3
== 1, so a gather that truncates its loop index to 32 bits shifts the phase of
the window at the 4 GiB mark. A 64-byte window at [4294967267, 4294967331)
carries the whole verdict; the head and the tail are checked too, so a decode
that wrote nothing cannot pass by agreeing with poison.

The block's well-formedness is the reference's verdict rather than an
assertion about the format. The same construction at match length 4096 goes
through LZ4_decompress_safe first, and the large block is built only if the
oracle agrees byte for byte.

How it runs, and how it does not

Built by every CUDA configure, so the target cannot rot; registered as a ctest
entry only under -DCUDEC_HUGE_TESTS=ON. ctest has no build-side way to keep
an entry out of a bare ctest run, and both plausible mechanisms were tried
rather than assumed:

add_test(NAME optin ...)
set_tests_properties(optin PROPERTIES LABELS huge CONFIGURATIONS huge)
$ ctest
2/2 Test #2: optin ......................   Passed

CONFIGURATIONS is ignored when no -C is given, and DISABLED would hide
the entry from -L as well. So the registration is the gate.

The label is huge, not gpu: scripts/sanitize-gpu.sh discovers its
targets with ctest -L gpu -N and runs four tools over every one of them, and
a 4 GiB single-warp decode is not that sweep's business.

cmake -B build-huge -DCUDEC_ENABLE_CUDA=ON -DCUDEC_HUGE_TESTS=ON
cmake --build build-huge -j
ctest --test-dir build-huge -L huge --output-on-failure

CONTRIBUTING gains that block, in a subsection beside the container command,
with the rule the next capacity-selected arm should follow.

Refused, never skipped

A device that cannot hold the allocation fails with the shortfall in the
message. Reached by raising the requirement in a throwaway edit, not in the
tree:

FAIL tests/huge_match_gpu.cu:127: free_bytes >= needed | this test needs
17247257748 bytes on the device and 9513730048 are free of 10736893952
total - it is refused rather than skipped, because it is the only test
that reaches the >4 GiB match arm

A skip on the only test that reaches an arm reports the same silence as a
pass, which is how the arm got here.

Proof that it bites

RTX 3080, driver 560.94, in the pinned container
(nvidia/cuda:12.6.2-devel-ubuntu24.04, nvcc V12.6.77, -arch=sm_86).

Guard present, the whole opt-in tree:

ctest --test-dir build-huge --no-tests=error --output-on-failure
100% tests passed, 0 tests failed out of 37
Label Time Summary:
gpu     =   5.37 sec*proc (8 tests)
huge    =  19.33 sec*proc (1 test)

The arm made unconditional (if (true) in place of the test, not in the
tree):

ctest --test-dir build-huge -L huge --output-on-failure
1/1 Test #31: huge_match_gpu ..............***Failed   10.69 sec
FAIL tests/huge_match_gpu.cu:187: differing == 0 | window
[4294967267, 4294967331): 32 of 64 bytes differ

ctest --test-dir build-huge -LE huge
(the other 36 entries: green)

32 of 64 differing is the count the issue recorded, so the shape reproduces
rather than merely failing.

The default tree is unchanged in what it runs:

cmake -B build-cuda -DCUDEC_ENABLE_CUDA=ON
ctest --test-dir build-cuda -N | grep -c 'Test *#'
36
ctest --test-dir build-cuda -N | grep -i huge   # exit 1, no match

ctest --test-dir build-cuda --no-tests=error --output-on-failure
100% tests passed, 0 tests failed out of 36

Formatting:

npx prettier@3 --check "**/*.{md,yml,yaml}"
All matched files use Prettier code style!

Not covered

The Compute Sanitizer sweep over this change has NOT been run. The four tools
cannot attach to the device on this route, which issue #258 holds; this test
carries the huge label and would be outside the sweep's discovery in any
case.

No second person read this change. The evidence above stands in place of one.

src/lz4_decode.cuh picks a 32-bit overlap gather under
seq.match_len <= UINT32_MAX and keeps the 64-bit one above it. Nothing in the
suite allocated the 4 GiB a match past 2^32 needs, so nothing selected the
64-bit arm, and deleting the test left every run green - a guard the suite
could not prove.

tests/huge_match_gpu.cu reaches it. One valid LZ4 block: three literals, one
match of 2^32 + 4096 bytes at offset 3, a five-byte literals-only tail. The
output is "ABC" repeating, and 2^32 mod 3 == 1, so a gather that truncates its
loop index to 32 bits shifts the phase of the window at the 4 GiB mark; a
64-byte window astride that mark carries the verdict without a 4 GiB compare.
The block's well-formedness is liblz4's verdict rather than an assertion about
the format: the same construction at match length 4096 is decoded by
LZ4_decompress_safe first, and the large one is only built if that agrees.

Registered only under -DCUDEC_HUGE_TESTS=ON, and built by every CUDA configure
so the target cannot rot. ctest has no build-side way to keep an entry out of
a bare `ctest` run - CONFIGURATIONS is ignored when no -C is given, DISABLED
hides the entry from -L as well - so the registration is the gate. Its label
is `huge` rather than `gpu`, which keeps a 4 GiB single-warp decode out of the
four-tool sanitizer sweep, and CONTRIBUTING carries the invocation beside the
container command.

A device that cannot hold the allocation FAILS with the shortfall in the
message rather than skipping, because a skip on the only test that reaches an
arm reports the same silence as a pass.

Measured on the RTX 3080 in the pinned container. With the guard present the
window is clean and the run takes 19s; with the arm made unconditional the
test reds at "window [4294967267, 4294967331): 32 of 64 bytes differ" while
the other 36 entries stay green, which is the shape the issue recorded. The
capacity refusal was reached the same way, by raising the requirement.
@iderex
iderex merged commit ffd0789 into main Aug 9, 2026
7 checks passed
@iderex
iderex deleted the issue/331 branch August 9, 2026 20:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The >4 GiB match arm has no test that reaches it, so the guard selecting it cannot be proven by the suite

1 participant