Add the Silesia Zstd corpus path and CPU denominator to the bench harness - #347
Merged
Conversation
…ness [#227] bench_zstd could only build its hand-constructed worst case. The standard corpora had no path through it at all, so the M5 record carried no ratio and no CPU denominator, and a later device number would have had nothing to be read against. The new path cuts a fetched corpus into independent frames at both endpoints of the 64 KiB to 512 KiB granularity range the batch model names, across levels 1, 3 and 19, and times single-thread ZSTD_decompress over each cell. It consumes MakeZstdBatchFrames from tests/zstd_corpus.h rather than adding a second generator, and the forced-mode fixtures beside it are run as a coverage report that states in its own output that it carries no number. Two guards, because the corpus can stop being the recorded one in two ways that both still round-trip. The frames are decoded back and their concatenation compared against the source, so a frame set that dropped or reordered part of the corpus fails instead of reporting a plausible ratio over different bytes. And each cell's corpus digest is pinned in the selfcheck, so a moved compressor pin or a changed granularity reds CI. Both were checked by breaking them: shrinking the selfcheck source reds all six digest cells, and dropping one frame per corpus trips the reconstruction check. The pin table and the run grid are indexed by the same two loop variables, so a level added to one and not the other is refused at compile time rather than read off the end of the array inside the selfcheck. docs/BENCHMARKS.md gains the M5 section with all six recorded cells and their methodology. The measurement it reports and the Snappy entry above it disagree: cutting to 64 KiB frames costs libzstd 8 to 14 percent where it cost snappy nothing measurable, so a device figure has to be quoted against the denominator at its own granularity.
Both sides added an entry to the bench selfcheck timeout list and a section to the benchmark record. Kept both in each: the parse-bound Snappy corpus from the mainline and the Zstd corpus path from this branch. Verified after the merge in the pinned container against the local device: 41/41 ctest, 0 failed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #227.
bench_zstdcould only build its hand-constructed worst case. The standard corpora had no path through it, so the M5 record carried no ratio and no CPU denominator, and a later device number would have had nothing to be read against.What this adds
A standard corpus path that cuts a fetched corpus into independent frames at both endpoints of the 64 KiB to 512 KiB granularity range the batch model names, across levels 1, 3 and 19, and times single-thread
ZSTD_decompressover each of the six cells. It consumesMakeZstdBatchFramesfromtests/zstd_corpus.hrather than adding a second generator, which is what the issue asks for.The forced-mode fixtures are run by
bench_zstd --coverage. They report which decode surfaces were reached and state in their own output that they carry no number, because fixtures sized to reach one surface each cannot carry a throughput figure.Recorded numbers
Measured in the digest-pinned
nvidia/cuda:12.6.2-devel-ubuntu24.04container on an AMD Ryzen 9 5950X, libzstd 1.5.7, over the twelve Silesia files at 211.94 MB. Reproduce withbench_zstd --warmup 3 --runs 30 bench/corpora/silesia/*.docs/BENCHMARKS.mdcarries all six blocks with their full methodology rather than this table alone.The reading worth carrying forward is that this disagrees with the Snappy entry above it. Cutting to 64 KiB frames costs libzstd 8 to 14 percent, where the same cut cost snappy 0.18 percent and sat inside its own run-to-run spread. So a device figure has to be quoted against the denominator at its own granularity; reading a 64 KiB batch decode against the 512 KiB row would credit the device with a difference that is the corpus shape.
The two guards, and the proof that each bites
A corpus can stop being the recorded one in two ways that both still round-trip, so there are two guards and both were checked by breaking them.
The frames are decoded back and their concatenation compared against the source before anything is timed. Dropping one frame from every built corpus:
Each cell's corpus digest is pinned in the selfcheck. Moving the selfcheck source from 3 MiB to 2 MiB:
All six cells red, not one, because the grid reports every digest before judging any of them. Reverting both mutations returns the selfcheck to
exit=0.The pin table and the run grid are indexed by the same two loop variables. A level added to one and not the other is now refused at compile time by a
static_assertrather than read off the end of the array inside the selfcheck, which is the one place a wrong answer would look like a verdict.Gate
Full build and
ctestin the digest-pinned container against the local RTX 3080, driver 560.94, CUDA 12.6.2:40 rather than the previous 38:
bench_zstd_selfcheckandbench_zstd_coverage_selfcheck. Both are CPU-only so they run on the GPU-less runner, and both carry the finiteTIMEOUTthatcudec_assert_test_timeouts()requires.npx prettier@3 --check "**/*.{md,yml,yaml}"reports all matched files use Prettier code style.What this does not claim
No second reader has looked at this change. What stands in place of one is the evidence above: the two guards were each mutated and watched to fail, the numbers carry the commands that produced them, and the gate output is quoted rather than summarised.
The compute-sanitizer net did not run over this change and nothing here asserts it is clean under those tools. That gate cannot attach to a device on this route, which is recorded on #258, and this change adds no device code.
No claim is made about why level 19 decodes slowest while compressing best. The per-phase split that would answer it is a separate entry.