MeshCore: versioned shared compression dictionary (#23) - #166
Merged
Conversation
The shared zstd dictionary is baked from a fixed corpus, so today every node derives byte-identical bytes - but nothing carried the dictionary version on the wire. The moment two nodes ran different dictionaries (a retrained corpus), the receiver would feed zstd a mismatched dictionary and either throw or, worse, silently produce corrupt bytes. This makes the scheme versioned and safe. - Wire: a compressed frame now carries the dictionary version in header byte1 ([flags][version][fragment]); uncompressed frames are unchanged ([flags][fragment]). Worst case 2 + 160 = 162 B, inside the 165 B firmware limit, so no extra fragmentation. - DappsCompression: a version -> dictionary registry that retains superseded versions (so a node can still decompress peers that haven't upgraded), CurrentDictionaryVersion for outbound, version-keyed Decompress, and IsKnownVersion. Retraining = add a registry entry + bump the current version; never mutate an existing version's bytes. - Receiver: a fully-reassembled frame whose dictionary version we don't hold is returned as Kind.Unsupported and dropped (never decompressed with the wrong dictionary) - MeshCoreInbound logs it so an operator sees the version gap, and doesn't ACK, so the sender keeps trying. A mixed-version fleet degrades to "can't read newer peers yet" instead of corrupting payloads. Validated on air (radio1/radio2, compression on): 6/6 delivered, 0% loss, reliability confirmed - compressed frames with the version byte round-trip on real hardware. Tests: MeshCoreCompressionVersionTests (version stamped, unknown-version dropped-not-delivered, round-trip, Decompress/IsKnownVersion); existing 20 transport tests still green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KLbwvhE2cKCe8WPZNg8k17
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes the MeshCore shared-dictionary compression versioned so a mixed-version fleet degrades safely instead of corrupting payloads.
Before: the dictionary version was a
constthat never reached the wire. The dictionary is baked from a fixed corpus, so today all nodes derive identical bytes — but the instant two nodes ran different dictionaries (a retrained corpus), the receiver would feed zstd a mismatched dictionary and either throw or silently produce corrupt bytes, with no way to detect it.How
[flags][version][fragment]. Uncompressed frames are unchanged ([flags][fragment]), so that path stays byte-identical. Worst case2 + 160 = 162 B, inside the firmware's 165 B channel-data limit — no extra fragmentation.DappsCompression— aversion → dictionaryregistry that retains superseded versions (a node can still decompress peers that haven't upgraded),CurrentDictionaryVersionfor outbound, version-keyedDecompress(throwsNotSupportedExceptionon unknown version), andIsKnownVersion. Retraining = add a registry entry + bump the current version; never mutate an existing version's bytes.Kind.Unsupportedand dropped (never decompressed with the wrong dictionary).MeshCoreInboundlogs it so an operator sees the version gap, and doesn't ACK — so the sender keeps trying. Mixed-version fleet degrades to "can't read newer peers yet."Validation
On air (radio1 ↔ radio2, compression on): 6/6 delivered, 0% loss, reliability confirmed — compressed frames with the version byte round-trip on real hardware within the 165 B limit.
Tests —
MeshCoreCompressionVersionTests(6): version stamped at byte 1; unknown version dropped-as-Unsupported and never delivered; current-version round-trip; uncompressed has no version byte;Decompressrecovers known / throws unknown;IsKnownVersion. Existing 20 transport tests (round-trips + shrink) still green.Note: this changes the compressed wire format; the bearer is pre-deployment (bench only), so this is the right moment to bake versioning in.
🤖 Generated with Claude Code