feat: add msgpack_strip_nulls() to shrink blobs by dropping nil map keys - #16
Merged
Conversation
New SQL function msgpack_strip_nulls(mp) recursively rebuilds a msgpack
value, dropping any map key whose value is nil. Recurses into nested
maps, including maps nested inside array elements. Array elements
themselves (even literal nulls) are always preserved, so array
length/order never changes -- only map keys are dropped. Scalars and
non-map/array roots pass through unchanged. NULL input -> NULL;
malformed msgpack -> error (never a crash).
Useful for shrinking sparse wide records (e.g. many optional columns
encoded as msgpack_object(...) with NULL for absent fields) before
storing them.
Implementation mirrors the mpMergePatch rebuild style: a recursive
mpStripNulls() walks the source buffer, buffering filtered map pairs
in a temp MpBuf before writing the final header (since the pair count
isn't known until nil keys are excluded), and appending array elements
directly since their count never changes. Map/array header counts are
bounds-checked against the remaining buffer before use, consistent
with the mpMergePatch fix in the previous release.
Testing:
- 9 new regression checks (9.1-9.9) in test_spec_p5_mutation.c: top-level
and recursive nil-stripping, array-element preservation (including a
nested map inside an array), all-nil map -> {}, scalar passthrough,
NULL -> NULL, deep multi-level nesting, size-shrink assertion, and a
malformed MAP32-count rejection check.
- Added msgpack_strip_nulls to the fuzz_msgpack.c harness so it's
exercised by both fuzz_corpus (ctest) and ad-hoc libFuzzer runs.
- Full ctest suite: 16/16 passing.
- 45s libFuzzer+ASan run across the whole harness (now including
msgpack_strip_nulls): 102,261 execs, 0 crashes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new SQL function
msgpack_strip_nulls(mp)that recursively removes map keys whose value isnil, shrinking the encoded size — useful for compacting sparse wide records (e.g. many optional columns encoded viamsgpack_object(...)withNULLfor absent fields).Semantics
nullinside an array stays, and array length/order never changes. Only map keys are dropped.NULLinput →NULL; malformed msgpack → SQL error (never a crash).Implementation
Mirrors the
mpMergePatchrebuild style added in the recent security fix: a recursivempStripNulls()walks the source buffer, buffers filtered map pairs in a tempMpBuf(since the final pair count isn't known until nils are excluded) before writing the map header, and appends array elements directly into the output since array element count never changes. Map/array header counts are bounds-checked against the remaining buffer before use — same hardening pattern as thempMergePatchfix.Testing
test_spec_p5_mutation.c: top-level and recursive nil-stripping, array-element preservation (including a nested map inside an array), all-nil map →{}, scalar passthrough,NULL→NULL, deep multi-level nesting, a size-shrink assertion, and a malformed-MAP32-count rejection check.msgpack_strip_nullsto thefuzz_msgpack.charness so it's exercised by bothfuzz_corpus(ctest) and ad-hoc libFuzzer runs going forward.ctestsuite: 16/16 passing.msgpack_strip_nulls): 102,261 execs, 0 crashes.Docs
README updated with a new
#### msgpack_strip_nulls(mp)section alongside the other mutation functions.Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com