Add the Snappy CPU denominator to the bench harness - #344
Merged
Conversation
There was no Snappy path in the bench harness at all, so the M3 device work had no number to be read against and any later GPU figure would have arrived without a baseline to divide by. bench_snappy builds the Silesia corpus in the two shapes that disagree about what a unit is - one stream per file, and one stream per 64 KiB page, which is what the columnar formats emit and what the batch API decodes - and times snappy::RawUncompress alone over each. Every stream is round-trip verified against the reference before anything is timed, and the destination buffers and declared lengths are taken outside the timed region so the number is the decoder's rather than the allocator's. Each report carries a digest of the corpus that was actually built, folded over the produced streams. A moved compressor pin, a changed chunk size or a generator that lost its noise source all move that digest while leaving the round trip intact, so the selfcheck asserts it and CI reds on drift the round trip alone would pass. Proven by cutting kChunkBytes to 32768: the selfcheck fails with the two digests printed, and passes again when it is restored. HostCpuName moves into bench_stats.h. Every report block names the host, and that header exists precisely so a promise kept by hand-matched copies does not become a promise about the copies. Recorded in docs/BENCHMARKS.md under M3.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Closes #164.
The bench harness had no Snappy path, so the M3 device work had nothing to be
read against.
bench_snappybuilds the Silesia corpus in both shapes the issuenames and records an honest single-thread CPU denominator for each.
The two shapes are there because the format and the batch API disagree about
what a unit is. Snappy's own framing is one stream per file; the shape this
library decodes is a page, which the columnar formats emit at 64 KiB. A
denominator taken on one does not transfer to the other unless that is
measured, so both are built from the same bytes and reported separately.
There is no Snappy kernel yet, and the report says so in its own decoder line
rather than leaving a reader to infer it from a missing GPU row.
The means is C++ in
bench/, linking the oracle targettests/already pinsand the fixture library that is the single provenance for every Snappy stream
in this project. A bench that compressed its own streams would be a second
authority on what a valid Snappy stream is, and nobody reconciles two.
Type of change
Engineering checklist
refuses a corpus rather than timing it: a file that contributes no
streams, a stream the reference will not decode, a stream that does not
round-trip, or a stream with no readable declared length each stop the
run before any timing. A failed decode inside the timed loop returns a
negative duration, which cannot be reported as a fast one.
on validity here, and it passes verdict on every stream before timing.
bench_snappylinks no CUDA at all.end.
Review record
CONFIRMED 0 / PLAUSIBLE 2.
snappy::RawUncompressand nothingelse; allocation and the declared-length parse are hoisted out and run once.
The digest fold is order-, length- and content-sensitive by construction.
No finding.
PrintReportindexessizes.front(), so an emptycorpus would be an out-of-bounds read. Disposition FIX, already in the diff:
RunCorpusrefuses a corpus of zero bytes before the report is reachable,and
AppendFilerefuses per file rather than over the accumulation, whichis the check that goes vacuous from the second argument on.
PrintReportratherthan cached. Disposition DECLINE. It runs once per report over 16 bytes per
stream, which is 52 KB for the 3239-stream corpus, and it is outside every
timed region.
bench/is consumer-only by rule and stays so: it linkscudec,cudec_test_fixturesandsnappy_oracleand modifies none ofthem. The new ctest entry carries the mandatory finite timeout and passes
cudec_assert_test_timeouts(). No finding.The corpus lock, and the proof that it bites
The issue asks for the built corpus to be hash-locked so its shape cannot
drift silently. The digest is a fold over the produced streams, not over the
inputs, because the inputs are already pinned by the manifest
bench/get-corpora.shwrites and two authorities on one fact is what thatavoids. Each stream contributes its length and its own XXH64, little-endian,
in corpus order, and the reported digest is the XXH64 of that array.
XXH64 and not SHA-256, stated so nothing more is read into it than is there:
this detects drift in data the harness just built. It is not a defence against
a chosen collision, and it is the hash already in the tree rather than a new
dependency for a bench.
The selfcheck asserts the digest of both shapes against recorded constants, so
a moved compressor pin, a changed chunk size or a generator that lost its noise
source reds CI. None of those would stop the corpus round-tripping, which is
why the round trip alone is not the check.
Proven by breaking it, in the container, on this branch.
kChunkBytescut from65536 to 32768,
bench_snappyrebuilt, nothing else touched:The failure names both digests and says what it means, rather than reporting a
mismatch and leaving the reader to work out which corpus was built.
GPU sanitizer gate
bench_snappyis aplain C++ target with no CUDA in it, and no
.cufile is modified.So the block is not read as an answered one: #258 records that no route to a
device the sanitizer can attach to is available on this machine.
Performance checklist
this diff.
and this adds the first Snappy entry rather than moving one.
Recorded inside the digest-pinned container, host AMD Ryzen 9 5950X, 3 warmup
plus 30 measured runs, over the pinned Silesia corpus:
Two readings worth having in the record, both written into
docs/BENCHMARKS.md with the full methodology blocks.
Cutting the corpus into 64 KiB pages costs the reference decoder nothing
measurable, 0.18% apart and well inside either row's own p50-to-p99 spread.
That is what keeps a later device number honest: quoted against the chunked
denominator it is not being flattered by a handicapped baseline, because the
two baselines are the same number.
Snappy's reference decoder is about 2.8x slower than liblz4's on these bytes at
the same ratio, 1.197 GB/s against the 3.410 GB/s the LZ4 CPU-oracle row in the
same file reports over the same 211.94 MB. Both are single-thread wall clock on
this host with the timed region held to the decode call, so that compares the
two references and says nothing about either GPU path.
Quality checklist
HostCpuNamemoves intobench_stats.hrather than beingcopied: every report block names the host, and that header exists
because a promise kept by hand-matched copies is a promise about the
copies.
bench_lz4.cpploses the local definition and calls the sharedone.
concatenating, why the zero-contribution check is per file, and what the
hash choice is and is not.
corpus is the one the numbers were recorded on, is locked by the
selfcheck above.
Verification
Full gate in the pinned container against the local RTX 3080, at this branch's
head:
-Wall -Wextra -Werror.ctestgreen, 37 of 37. The count was 36 before this branch; the newentry is
bench_snappy_selfcheck, and it carries no gpu label, so itruns in the CI selection (
ctest -LE gpu) rather than being deferred tothe local gate.
**/*.{md,yml,yaml}.blocks verbatim and the command that reproduces them.
One reproducibility note that belongs in the record rather than in a footnote:
the digest is order-sensitive, so the two constants in the M3 section belong to
the file order a glob produces. Running the harness with the files in another
order gives the same timings and a different digest, which is the lock working
rather than failing.
Notes
This change had no second reader. The review record above is one reader over
four lenses, and the pasted gate output, the break-and-restore proof and the
recorded numbers are the evidence in place of one.
#166 builds the adversarial max-element-density corpus in this same file and
follows this landing rather than sharing it, so each has its own reason to
exist and its own definition of done.