perf: batch msgpack_set/remove/patch for O(map+edits) wide-record edits - #11
Merged
Conversation
msgpack_set() and msgpack_remove() rebuilt the entire blob once per (path,value) pair, and msgpack_patch()'s merge matched keys with a linear scan of the patch — all O(nPairs * mapSize) per call. On wide maps this made bulk edits super-linear and much slower than the SQLite JSON equivalents. Apply all top-level "$.<key>" edits in a single rebuild pass using a small open-addressing hash (stack-allocated for small edits, heap for large), and hash the pre-scanned patch keys in mpMergePatch. Each call becomes O(mapSize + nPairs) while producing byte-for-byte identical output; any non-simple path (nested/array) transparently falls back to the generic path. Measured on 10k-row updates: set 34,107 -> 196 us/row @ 2000 keys (~174x, now faster than jsonb_set) remove 783 -> 8 us/row @ 400 keys (~95x, now faster than jsonb_remove) patch 138 -> 19 us/row @ 400 keys (~7x, ~31x faster than jsonb_patch) Validated with 300k+ randomized cases (byte-identical to the previous implementation) plus new msgpack_spec_p5 batch/remove/patch regression checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
msgpack_set(),msgpack_remove(), andmsgpack_patch()were O(nPairs × mapSize) per call on maps:msgpack_set/msgpack_removerebuilt the entire blob once per path/value pair.msgpack_patch(mpMergePatch) matched each target key against the patch with a linear scan (the code claimed O(n+m) but the match loop was O(target × patch)).On wide records this made bulk edits super-linear and slower than the SQLite JSON equivalents.
Change
Apply all top-level
$.<key>edits in a single rebuild pass using a small open-addressing hash (stack-allocated for small edit counts, heap for large), and hash the pre-scanned patch keys inmpMergePatch. Each call becomes O(mapSize + nPairs).$.a.b, array$.a[0], mixed) transparently falls back to the existing generic path.nullset/drop, duplicate-key last-wins, remove-all-occurrences.Measured (10,000-row updates, Optimized build, SQLite 3.53.2)
msgpack_setjsonb_setmsgpack_removejsonb_removemsgpack_patchjsonb_patchAll three are now linear in map size.
Validation
tests/test_spec_p5_mutation.c(§8.1–8.9) covering wide batch set, set semantics, batch remove (incl. remove-all →{}), and recursive patch merge.ctestsuite: 16/16 passing. No new compiler warnings.