Skip to content

Add the Snappy CPU denominator to the bench harness - #344

Merged
iderex merged 2 commits into
mainfrom
issue/164-snappy-bench
Aug 10, 2026
Merged

Add the Snappy CPU denominator to the bench harness#344
iderex merged 2 commits into
mainfrom
issue/164-snappy-bench

Conversation

@iderex

@iderex iderex commented Aug 10, 2026

Copy link
Copy Markdown
Owner

What & why

Closes #164.

The bench harness had no Snappy path, so the M3 device work had nothing to be
read against. bench_snappy builds the Silesia corpus in both shapes the issue
names and records an honest single-thread CPU denominator for each.

The two shapes are there because the format and the batch API disagree about
what a unit is. Snappy's own framing is one stream per file; the shape this
library decodes is a page, which the columnar formats emit at 64 KiB. A
denominator taken on one does not transfer to the other unless that is
measured, so both are built from the same bytes and reported separately.

There is no Snappy kernel yet, and the report says so in its own decoder line
rather than leaving a reader to infer it from a missing GPU row.

The means is C++ in bench/, linking the oracle target tests/ already pins
and the fixture library that is the single provenance for every Snappy stream
in this project. A bench that compressed its own streams would be a second
authority on what a valid Snappy stream is, and nobody reconciles two.

Type of change

  • Build / CI / supply chain
  • Docs
  • Decode kernel / device code
  • Format support (LZ4 / Snappy / GDeflate / Zstd)
  • API surface
  • Performance
  • Bug fix
  • Refactor / code quality

Engineering checklist

  • Fail-closed preserved. Nothing here parses untrusted input. The harness
    refuses a corpus rather than timing it: a file that contributes no
    streams, a stream the reference will not decode, a stream that does not
    round-trip, or a stream with no readable declared length each stop the
    run before any timing. A failed decode inside the timed loop returns a
    negative duration, which cannot be reported as a fast one.
  • Oracle coverage: the pinned snappy 1.2.2 decoder is the sole authority
    on validity here, and it passes verdict on every stream before timing.
  • Determinism preserved. Nothing in the library moves.
  • No CUDA call is added; bench_snappy links no CUDA at all.
  • Adversarial review: one reader, four lenses. See the disclosure at the
    end.
  • Review record below.

Review record

CONFIRMED 0 / PLAUSIBLE 2.

  • Correctness. The timed region holds snappy::RawUncompress and nothing
    else; allocation and the declared-length parse are hoisted out and run once.
    The digest fold is order-, length- and content-sensitive by construction.
    No finding.
  • Robustness. PLAUSIBLE: PrintReport indexes sizes.front(), so an empty
    corpus would be an out-of-bounds read. Disposition FIX, already in the diff:
    RunCorpus refuses a corpus of zero bytes before the report is reachable,
    and AppendFile refuses per file rather than over the accumulation, which
    is the check that goes vacuous from the second argument on.
  • Performance. PLAUSIBLE: the digest is recomputed inside PrintReport rather
    than cached. Disposition DECLINE. It runs once per report over 16 bytes per
    stream, which is 52 KB for the 3239-stream corpus, and it is outside every
    timed region.
  • Integration. bench/ is consumer-only by rule and stays so: it links
    cudec, cudec_test_fixtures and snappy_oracle and modifies none of
    them. The new ctest entry carries the mandatory finite timeout and passes
    cudec_assert_test_timeouts(). No finding.

The corpus lock, and the proof that it bites

The issue asks for the built corpus to be hash-locked so its shape cannot
drift silently. The digest is a fold over the produced streams, not over the
inputs, because the inputs are already pinned by the manifest
bench/get-corpora.sh writes and two authorities on one fact is what that
avoids. Each stream contributes its length and its own XXH64, little-endian,
in corpus order, and the reported digest is the XXH64 of that array.

XXH64 and not SHA-256, stated so nothing more is read into it than is there:
this detects drift in data the harness just built. It is not a defence against
a chosen collision, and it is the hash already in the tree rather than a new
dependency for a bench.

The selfcheck asserts the digest of both shapes against recorded constants, so
a moved compressor pin, a changed chunk size or a generator that lost its noise
source reds CI. None of those would stop the corpus round-tripping, which is
why the round trip alone is not the check.

Proven by breaking it, in the container, on this branch. kChunkBytes cut from
65536 to 32768, bench_snappy rebuilt, nothing else touched:

=== the digest lock, proven by breaking it ===
0% tests passed, 1 tests failed out of 1
The following tests FAILED:
	 36 - bench_snappy_selfcheck (Failed)
--- restoring ---
100% tests passed, 0 tests failed out of 1

The failure names both digests and says what it means, rather than reporting a
mismatch and leaving the reader to work out which corpus was built.

GPU sanitizer gate

  • Not applicable: this change touches no device code. bench_snappy is a
    plain C++ target with no CUDA in it, and no .cu file is modified.

So the block is not read as an answered one: #258 records that no route to a
device the sanitizer can attach to is available on this machine.

Performance checklist

  • Measured, not reasoned. Every number below came out of the harness in
    this diff.
  • No regression against docs/BENCHMARKS.md. Nothing in the library changes,
    and this adds the first Snappy entry rather than moving one.

Recorded inside the digest-pinned container, host AMD Ryzen 9 5950X, 3 warmup
plus 30 measured runs, over the pinned Silesia corpus:

bench_snappy --warmup 3 --runs 30 bench/corpora/silesia/*
shape streams original compressed ratio p50 wall p50
whole-file 12 211.94 MB 101.35 MB 0.478 177.088 ms 1.197 GB/s
64 KiB-chunked 3239 211.94 MB 101.36 MB 0.478 177.413 ms 1.195 GB/s

Two readings worth having in the record, both written into
docs/BENCHMARKS.md with the full methodology blocks.

Cutting the corpus into 64 KiB pages costs the reference decoder nothing
measurable, 0.18% apart and well inside either row's own p50-to-p99 spread.
That is what keeps a later device number honest: quoted against the chunked
denominator it is not being flattered by a handicapped baseline, because the
two baselines are the same number.

Snappy's reference decoder is about 2.8x slower than liblz4's on these bytes at
the same ratio, 1.197 GB/s against the 3.410 GB/s the LZ4 CPU-oracle row in the
same file reports over the same 211.94 MB. Both are single-thread wall clock on
this host with the timed region held to the decode call, so that compares the
two references and says nothing about either GPU path.

Quality checklist

  • Minimal. HostCpuName moves into bench_stats.h rather than being
    copied: every report block names the host, and that header exists
    because a promise kept by hand-matched copies is a promise about the
    copies. bench_lz4.cpp loses the local definition and calls the shared
    one.
  • Self-documenting. The comments say why the digest folds instead of
    concatenating, why the zero-contribution check is per file, and what the
    hash choice is and is not.
  • Conformance tests pass. The new structural property, that the built
    corpus is the one the numbers were recorded on, is locked by the
    selfcheck above.

Verification

Full gate in the pinned container against the local RTX 3080, at this branch's
head:

cmake -B build-cuda -DCUDEC_ENABLE_CUDA=ON && cmake --build build-cuda -j 12
ctest --test-dir build-cuda --output-on-failure

100% tests passed, 0 tests failed out of 37
Label Time Summary:
gpu    =   4.90 sec*proc (8 tests)
Total Test time (real) =   8.72 sec
  • Builds clean under -Wall -Wextra -Werror.
  • ctest green, 37 of 37. The count was 36 before this branch; the new
    entry is bench_snappy_selfcheck, and it carries no gpu label, so it
    runs in the CI selection (ctest -LE gpu) rather than being deferred to
    the local gate.
  • Prettier clean over **/*.{md,yml,yaml}.
  • Docs synced. docs/BENCHMARKS.md gains the M3 section with both report
    blocks verbatim and the command that reproduces them.

One reproducibility note that belongs in the record rather than in a footnote:
the digest is order-sensitive, so the two constants in the M3 section belong to
the file order a glob produces. Running the harness with the files in another
order gives the same timings and a different digest, which is the lock working
rather than failing.

Notes

This change had no second reader. The review record above is one reader over
four lenses, and the pasted gate output, the break-and-restore proof and the
recorded numbers are the evidence in place of one.

#166 builds the adversarial max-element-density corpus in this same file and
follows this landing rather than sharing it, so each has its own reason to
exist and its own definition of done.

iderex added 2 commits August 10, 2026 11:09
There was no Snappy path in the bench harness at all, so the M3 device work
had no number to be read against and any later GPU figure would have arrived
without a baseline to divide by.

bench_snappy builds the Silesia corpus in the two shapes that disagree about
what a unit is - one stream per file, and one stream per 64 KiB page, which
is what the columnar formats emit and what the batch API decodes - and times
snappy::RawUncompress alone over each. Every stream is round-trip verified
against the reference before anything is timed, and the destination buffers
and declared lengths are taken outside the timed region so the number is the
decoder's rather than the allocator's.

Each report carries a digest of the corpus that was actually built, folded
over the produced streams. A moved compressor pin, a changed chunk size or a
generator that lost its noise source all move that digest while leaving the
round trip intact, so the selfcheck asserts it and CI reds on drift the round
trip alone would pass. Proven by cutting kChunkBytes to 32768: the selfcheck
fails with the two digests printed, and passes again when it is restored.

HostCpuName moves into bench_stats.h. Every report block names the host, and
that header exists precisely so a promise kept by hand-matched copies does
not become a promise about the copies.

Recorded in docs/BENCHMARKS.md under M3.
@iderex iderex added enhancement New feature or request area:bench Benchmarks and baselines priority:p2 Next: current milestone, not blocking labels Aug 10, 2026
@iderex iderex self-assigned this Aug 10, 2026
@iderex iderex added this to the M3 - Snappy milestone Aug 10, 2026
@iderex
iderex merged commit 535ac48 into main Aug 10, 2026
7 checks passed
@iderex
iderex deleted the issue/164-snappy-bench branch August 10, 2026 09:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:bench Benchmarks and baselines enhancement New feature or request priority:p2 Next: current milestone, not blocking

Projects

None yet

Development

Successfully merging this pull request may close these issues.

M3 bench: the Snappy corpus path (Silesia, both shapes, hash-locked) and the CPU denominator

1 participant