perf: single-pass multi-path msgpack_extract (O(K*N) -> O(N)) - #14
Merged
Conversation
msgpack_extract(mp, p1, ..., pK) previously resolved each path with an
independent mpLookup() scan from the start of the container — O(K*N) for K
paths over an N-element map/array. When the root is a container and every
path is a single component ($.key or $[idx]) and K >= 16, gather all
requested elements in ONE linear pass:
* mpScanMap — open-addressing hash of the requested keys (chained for
duplicate paths); each map entry matched in O(1).
* mpScanArray — direct index table (idx -> chain), bounded by a size guard.
Any other case (few paths, non-container root, non-simple path, malformed
structure, OOM) falls back to the original per-path loop, so the returned
bytes are identical in every case.
Paths are parsed locally per call rather than cached in per-argument
sqlite3_set_auxdata(): SQLite's auxdata lookup is O(#entries), so per-path
caching would make a K-path call O(K^2).
Also hardens the map key decode against a malformed STR32 key whose declared
length near 0xFFFFFFFF would wrap the 32-bit value offset past the valOff<=n
guard and drive an out-of-bounds read when hashing the key; the length is now
validated against the available bytes first.
Benchmarks (10k rows, one call/row): extracting all N fields ~40-46x faster
at N=400 (map and array); a 16-field subset of an N=400 record ~100-119x
faster; small path counts unchanged. Validated with differential tests vs the
original (byte-identical) and ASan/UBSan clean, incl. adversarial malformed
blobs. Adds Phase 4.11 multi-path tests.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
msgpack_extract(mp, p1, …, pK)(the multi-path form that returns a msgpack array) currently resolves each path with an independentmpLookup()scan from the start of the container — O(K·N) for K paths over an N-element map/array. This PR gathers all requested elements in one linear O(N) pass when the root is a container and every path is a single component ($.keyor$[idx]) andK ≥ 16.How
mpScanMap— builds an open-addressing hash of the requested keys (with per-key chains for duplicate paths); each map entry is matched in O(1).mpScanArray— builds a direct index table (idx → chain), bounded by a size guard for sparse indices.mpExtractMultidispatches to the fast path; any other case (few paths, non-container root, non-simple/nested path, malformed structure, OOM) falls back to the original per-path loop, so the returned bytes are identical in every case.sqlite3_set_auxdata()— SQLite's auxdata lookup is O(#entries), so per-path caching would make a K-path call O(K²) (this was measured, and regressed array extraction). Documented in-code.Security hardening
Fixes a potential out-of-bounds read in the new map key decode: a malformed
STR32key whose declared length is near0xFFFFFFFFwould wrap the 32-bit value offset past thevalOff <= nguard and drive an OOB read when hashing the key. The length is now validated against the available bytes first. (The originalmpLookupnever hashed the stored key, so this is specific to the fast path and is guarded from the outset.)Benchmarks
10k rows, one
msgpack_extractcall per row (standalonesqlite3.c+msgpack.c, -O2):Validation
STR32overflow repro.ctest16/16 pass; adds Phase 4.11 multi-path tests totests/test_spec_p4_extract.c(map/array single-pass, nested-path fallback, malformed-STR32no-overflow) — 75/75 in that file, clean under ASan.Notes
msgpack_extract(mp,'$.pA','$.pB',…)(ormsgpack_each); N separate single-path calls are unaffected.