Skip to content

feat: add msgpack_strip_nulls() to shrink blobs by dropping nil map keys - #16

Merged
khanaffan merged 1 commit into
mainfrom
feat/strip-nulls
Jul 2, 2026
Merged

feat: add msgpack_strip_nulls() to shrink blobs by dropping nil map keys#16
khanaffan merged 1 commit into
mainfrom
feat/strip-nulls

Conversation

@khanaffan

Copy link
Copy Markdown
Owner

Summary

Adds a new SQL function msgpack_strip_nulls(mp) that recursively removes map keys whose value is nil, shrinking the encoded size — useful for compacting sparse wide records (e.g. many optional columns encoded via msgpack_object(...) with NULL for absent fields).

Semantics

  • Recurses into nested maps, including maps nested inside array elements.
  • Array elements are always preserved — even a literal null inside an array stays, and array length/order never changes. Only map keys are dropped.
  • Scalars and non-map/array roots pass through unchanged.
  • NULL input → NULL; malformed msgpack → SQL error (never a crash).
SELECT msgpack_to_json(
  msgpack_strip_nulls(msgpack_from_json('{"a":1,"b":null,"c":{"x":null,"y":2}}'))
);
-- {"a":1,"c":{"y":2}}

SELECT msgpack_to_json(msgpack_strip_nulls(msgpack_array(1, null, 3)));
-- [1,null,3]   (array elements untouched)

Implementation

Mirrors the mpMergePatch rebuild style added in the recent security fix: a recursive mpStripNulls() walks the source buffer, buffers filtered map pairs in a temp MpBuf (since the final pair count isn't known until nils are excluded) before writing the map header, and appends array elements directly into the output since array element count never changes. Map/array header counts are bounds-checked against the remaining buffer before use — same hardening pattern as the mpMergePatch fix.

Testing

  • 9 new regression checks (9.1–9.9) in test_spec_p5_mutation.c: top-level and recursive nil-stripping, array-element preservation (including a nested map inside an array), all-nil map → {}, scalar passthrough, NULLNULL, deep multi-level nesting, a size-shrink assertion, and a malformed-MAP32-count rejection check.
  • Added msgpack_strip_nulls to the fuzz_msgpack.c harness so it's exercised by both fuzz_corpus (ctest) and ad-hoc libFuzzer runs going forward.
  • Full ctest suite: 16/16 passing.
  • 45s libFuzzer+ASan run across the whole harness (now including msgpack_strip_nulls): 102,261 execs, 0 crashes.

Docs

README updated with a new #### msgpack_strip_nulls(mp) section alongside the other mutation functions.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

New SQL function msgpack_strip_nulls(mp) recursively rebuilds a msgpack
value, dropping any map key whose value is nil. Recurses into nested
maps, including maps nested inside array elements. Array elements
themselves (even literal nulls) are always preserved, so array
length/order never changes -- only map keys are dropped. Scalars and
non-map/array roots pass through unchanged. NULL input -> NULL;
malformed msgpack -> error (never a crash).

Useful for shrinking sparse wide records (e.g. many optional columns
encoded as msgpack_object(...) with NULL for absent fields) before
storing them.

Implementation mirrors the mpMergePatch rebuild style: a recursive
mpStripNulls() walks the source buffer, buffering filtered map pairs
in a temp MpBuf before writing the final header (since the pair count
isn't known until nil keys are excluded), and appending array elements
directly since their count never changes. Map/array header counts are
bounds-checked against the remaining buffer before use, consistent
with the mpMergePatch fix in the previous release.

Testing:
- 9 new regression checks (9.1-9.9) in test_spec_p5_mutation.c: top-level
  and recursive nil-stripping, array-element preservation (including a
  nested map inside an array), all-nil map -> {}, scalar passthrough,
  NULL -> NULL, deep multi-level nesting, size-shrink assertion, and a
  malformed MAP32-count rejection check.
- Added msgpack_strip_nulls to the fuzz_msgpack.c harness so it's
  exercised by both fuzz_corpus (ctest) and ad-hoc libFuzzer runs.
- Full ctest suite: 16/16 passing.
- 45s libFuzzer+ASan run across the whole harness (now including
  msgpack_strip_nulls): 102,261 execs, 0 crashes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@khanaffan
khanaffan merged commit e5a49d2 into main Jul 2, 2026
6 checks passed
@khanaffan
khanaffan deleted the feat/strip-nulls branch July 2, 2026 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant