Skip to content

perf: batch msgpack_set/remove/patch for O(map+edits) wide-record edits - #11

Merged
khanaffan merged 1 commit into
mainfrom
perf/batch-mutation-fastpath
Jul 1, 2026
Merged

perf: batch msgpack_set/remove/patch for O(map+edits) wide-record edits#11
khanaffan merged 1 commit into
mainfrom
perf/batch-mutation-fastpath

Conversation

@khanaffan

Copy link
Copy Markdown
Owner

Summary

msgpack_set(), msgpack_remove(), and msgpack_patch() were O(nPairs × mapSize) per call on maps:

  • msgpack_set / msgpack_remove rebuilt the entire blob once per path/value pair.
  • msgpack_patch (mpMergePatch) matched each target key against the patch with a linear scan (the code claimed O(n+m) but the match loop was O(target × patch)).

On wide records this made bulk edits super-linear and slower than the SQLite JSON equivalents.

Change

Apply all top-level $.<key> edits in a single rebuild pass using a small open-addressing hash (stack-allocated for small edit counts, heap for large), and hash the pre-scanned patch keys in mpMergePatch. Each call becomes O(mapSize + nPairs).

  • Output is byte-for-byte identical to the previous implementation.
  • Any non-simple path (nested $.a.b, array $.a[0], mixed) transparently falls back to the existing generic path.
  • Semantics preserved: replace-existing, append-new in first-appearance order, null set/drop, duplicate-key last-wins, remove-all-occurrences.

Measured (10,000-row updates, Optimized build, SQLite 3.53.2)

op keys before after speedup vs JSON
msgpack_set 2000 34,107 µs/row 196 ~174× now faster than jsonb_set
msgpack_remove 400 783 µs/row 8 ~95× now faster than jsonb_remove
msgpack_patch 400 138 µs/row 19 ~7× ~31× faster than jsonb_patch

All three are now linear in map size.

Validation

  • 300,000+ randomized fuzz cases comparing the patched output against the previous implementation — byte-identical across existing/absent/duplicate keys, null values, nested values, and fallback paths.
  • New regression checks added to tests/test_spec_p5_mutation.c (§8.1–8.9) covering wide batch set, set semantics, batch remove (incl. remove-all → {}), and recursive patch merge.
  • Full ctest suite: 16/16 passing. No new compiler warnings.

msgpack_set() and msgpack_remove() rebuilt the entire blob once per
(path,value) pair, and msgpack_patch()'s merge matched keys with a linear
scan of the patch — all O(nPairs * mapSize) per call. On wide maps this made
bulk edits super-linear and much slower than the SQLite JSON equivalents.

Apply all top-level "$.<key>" edits in a single rebuild pass using a small
open-addressing hash (stack-allocated for small edits, heap for large), and
hash the pre-scanned patch keys in mpMergePatch. Each call becomes
O(mapSize + nPairs) while producing byte-for-byte identical output; any
non-simple path (nested/array) transparently falls back to the generic path.

Measured on 10k-row updates:
  set    34,107 -> 196 us/row @ 2000 keys  (~174x, now faster than jsonb_set)
  remove    783 ->   8 us/row @  400 keys  (~95x,  now faster than jsonb_remove)
  patch     138 ->  19 us/row @  400 keys  (~7x,   ~31x faster than jsonb_patch)

Validated with 300k+ randomized cases (byte-identical to the previous
implementation) plus new msgpack_spec_p5 batch/remove/patch regression checks.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@khanaffan
khanaffan merged commit 77b6b8d into main Jul 1, 2026
6 checks passed
@khanaffan
khanaffan deleted the perf/batch-mutation-fastpath branch July 1, 2026 19:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant