Skip to content

Add Python/JS/Rust/Go ports and modularize the C++ Blob library into cpp/ - #8

Merged
khanaffan merged 4 commits into
mainfrom
feat/blob-multi-language-ports
Jun 22, 2026
Merged

Add Python/JS/Rust/Go ports and modularize the C++ Blob library into cpp/#8
khanaffan merged 4 commits into
mainfrom
feat/blob-multi-language-ports

Conversation

@khanaffan

Copy link
Copy Markdown
Owner

Summary

Refactors the standalone C++ Blob API into a self-contained cpp/ package and adds four native, zero-dependency portsPython, TypeScript/JS, Rust, and Go — each producing byte-identical MessagePack output. Blobs now round-trip freely between the SQLite extension, C++, Python, JavaScript, Rust and Go.

The SQLite extension is untouched (src/msgpack.c, src/sqlite3.c, include/sqlite3*.h).

What changed

C++ library (cpp/)

  • Split the monolithic src/msgpack_blob.cpp (~2,250 lines) into five logical modules — decode, encode, json, mutate, iterate — sharing private internals via msgpack_blob_detail.hpp. Public API (msgpack_blob.hpp) is unchanged.
  • Moved the library into its own self-contained cpp/ package (sibling of python/, js/, rust/, go/) with a CMakeLists.txt that builds standalone (cmake -B build cpp) or via add_subdirectory(cpp) from the root. The root build keeps the C++ ↔ SQLite interop test.

New language ports

Port Runtime Tests
python/ Python ≥ 3.7, stdlib only 30
js/ Node ≥ 18, ESM + .d.ts, 64-bit ints via bigint 19
rust/ Rust ≥ 1.70, no deps 22
go/ Go ≥ 1.21, stdlib only 20

Each mirrors the same Blob / Builder / Value / Iterator API.

Cross-language byte-identity

A C++ generator (cpp/tests/gen_blob_vectors.cpp, CTest target blob_vectors_gen) emits a shared vector file (tests/vectors/blob_vectors.json) that every port replays — covering from_json, to_json/pretty, typed encoding, mutation, merge-patch, extraction, array_length, and iteration.

Notable engineering

  • Float formatting (%g) was the hardest part — no native %g in Rust/JS. Reproduced C's %.17g/%.7g exactly: JS via BigInt round-half-to-even, Rust via {:.*e}, Go via strconv. Validated 0 mismatches across 35k+ values per language, plus a 50k-value end-to-end to_json cross-check vs the C++ reference.
  • 64-bit integers round-trip losslessly (JS bigint, Rust i64/u64, Go native).
  • Non-UTF-8 string bytes are preserved byte-exactly in every port.
  • Extra fuzzing vs the C++ reference: 16k from_json numbers, 15k merge_patch, 15k set/remove, and a 50k-record adversarial panic-fuzz through every public method (0 panics).

Bugs found & fixed during review

  • JS to_json corrupted non-UTF-8 string bytes (lossy TextDecoder) — replaced with a byte-preserving codec.
  • JS %g exponent off-by-one on values like 1e-7 — rewritten with exact BigInt arithmetic.
  • Rust & Go could panic on a truncated map key (out-of-bounds slice) where C++ bails gracefully — added the missing bounds guard, with regression tests.

Testing

  • C++: ctest 16/16 under both Makefiles and Ninja (CI generator); standalone cpp/ build 2/2.
  • Python 30, JS 19, Rust 22, Go 20 — all green.
  • Shared vectors reproduce byte-for-byte from blob_vectors_gen.

Docs

Main README (Language libraries section + C++ reference row), per-language READMEs, CONTRIBUTING.md, SECURITY.md, and .gitignore all updated for the cpp/ layout and new ports.

Affan Khan and others added 4 commits June 22, 2026 12:50
… package

Split the monolithic src/msgpack_blob.cpp (~2,250 lines) into five logical translation units — decode, encode, json, mutate, iterate — sharing private internals via msgpack_blob_detail.hpp. The public API (msgpack_blob.hpp) is unchanged and the SQLite extension is untouched.

The C++ Blob library now lives in its own self-contained cpp/ package (the C++ sibling of the python/, js/, rust/ and go/ ports) with a CMakeLists.txt that builds standalone (cmake -B build cpp) or via add_subdirectory(cpp) from the repository root. The root build keeps the C++ <-> SQLite interop test.

Also adds the cross-language test-vector generator (cpp/tests/gen_blob_vectors.cpp, CTest target blob_vectors_gen) and the generated tests/vectors/blob_vectors.json that every language port is validated against.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Native, zero-dependency re-implementations of the C++ Blob API in four languages, each exposing the same Blob / Builder / Value / Iterator API and producing byte-identical msgpack output, so blobs round-trip freely between SQL, C++, Python, JavaScript, Rust and Go.

- python/  pure stdlib (30 tests)
- js/      TypeScript, ESM + .d.ts, 64-bit ints via bigint (19 tests)
- rust/    no deps, exact %g float formatting via {:.*e} (22 tests)
- go/      stdlib only (20 tests)

Every port replays the shared tests/vectors/blob_vectors.json generated from the C++ reference, plus API/round-trip and non-UTF-8 regression tests. Float formatting and 64-bit integer handling were validated against the C reference across tens of thousands of randomized values.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Update the main README (new Language libraries section with a C++ reference row), the per-language READMEs, CONTRIBUTING.md and SECURITY.md to reflect the cpp/ package layout and the new ports, and extend .gitignore for Node, TypeScript, Rust and Go build artifacts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CI previously built and tested only the C/C++ targets via CTest. Add dedicated parallel jobs so every language port is exercised on each push and pull request:

- python — python -m unittest discover -s tests
- js     — npm ci, tsc type-check + build, node --test
- rust   — cargo fmt --check, cargo clippy -D warnings, cargo test
- go     — gofmt check, go vet, go test ./...

Also adds a standalone cpp/ package build job and a step that regenerates the shared vectors and fails if tests/vectors/blob_vectors.json has drifted from the C++ reference. Reformats one regression test so cargo fmt --check passes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@khanaffan
khanaffan merged commit b96d11d into main Jun 22, 2026
6 checks passed
@khanaffan
khanaffan deleted the feat/blob-multi-language-ports branch June 22, 2026 17:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant