perf: speed up msgpack_extract/lookup on the map hot path - #12
Merged
Conversation
mpLookup() skipped over each map key with a full mpSkipOne() even though the key header was already decoded, and mpSkipOneD() tested fixstr — the most common element type (every map key plus short strings) — only after the whole switch and two further branches. Derive the value offset directly from the decoded key header (dropping the redundant per-key skip) and check fixstr before the switch. Output is unchanged; only the traversal is cheaper. Extracting every property of a 256-key map (per-row, us): int 176 -> 131 real 213 -> 161 text 223 -> 180 blob 198 -> 150 msgpack_extract is now faster-or-equal to json_extract/jsonb_extract across all value types (text was ~12% slower and blob slower; both are now parity/faster), a 19-26% improvement. Validated with 40k randomized extract/msgpack_to_json cases (identical to the prior implementation) and the full ctest suite (spec p4 extract + fuzz corpus). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
khanaffan
force-pushed
the
perf/faster-extract-lookup
branch
from
July 1, 2026 20:04
257fd7b to
0bb0fe4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
msgpack_extract()(and everything built onmpLookup:WHERE, index expressions) was slower thanjson_extract/jsonb_extractfor text and blob values, which dragged mixed-type full-row reads below jsonb. Two causes on the map traversal hot path:mpLookupskipped over each map key with a fullmpSkipOne()call — even though the key header (type + length) had just been decoded in the same loop iteration. The value offset can be computed directly.mpSkipOneDtested fixstr last (after the entireswitchplus two more branches), yet fixstr is the most common element type — every map key, plus short strings.Change
mpSkipOne().switch.Output is byte/value-identical — this only makes the traversal cheaper. Non-string keys still fall back to
mpSkipOne().Results
Extracting every property of a 256-key map (µs per row):
19–26% faster extraction;
msgpack_extractis now faster-or-equal to the SQLite JSON functions across all value types. The gain holds at larger maps (P=512 text: 836 → 630 µs/row).Validation
msgpack_extract/msgpack_to_jsoncases — results identical to the previous implementation (nested maps, bins, nulls, absent keys, all scalar types).ctestsuite (16/16), includingmsgpack_spec_p4(extract) and the fuzz corpus. No new warnings.Independent of #11 (that PR targets the mutation functions); this touches only
mpLookup/mpSkipOneD.