Idea: canonical / deterministic serialization (RFC 8785 JCS)

[nlohmann]

Status: brainstorming, not a plan

This is an idea to move forward on, not a sanctioned TODO or a commitment. It is a hypothesis to investigate, measure, and discuss. Treat this as a starting point for discussion.

Motivation

Signing, hashing, content-addressing, reproducible builds, and merkle/blockchain use cases need two serializations of equal documents to be byte-identical. There is currently no way to get that without post-processing the output. RFC 8785 (JSON Canonicalization Scheme, JCS) is the relevant standard.

Of the net-new ideas, this one has the best effort-to-impact ratio: it's a self-contained serializer variant with no core/ABI changes.

What JCS mandates, and where it rubs against dump()

1. Object keys sorted by UTF-16 code unit. The default object_t is std::map, sorted by std::string::operator< — i.e. UTF-8 byte order. For BMP characters that matches UTF-16 order, but for characters ≥ U+10000 it does not (UTF-8 byte order sorts by code point; UTF-16 sorts surrogate pairs, so U+E000–U+FFFF sort after supplementary characters). Canonical mode therefore cannot rely on the container's ordering and must sort keys itself with a UTF-16-code-unit comparator. ordered_json (insertion order) needs the same treatment.

2. Number formatting = ECMAScript Number::toString. Our dtoa already produces shortest round-trip output (≈90% there), but JCS pins the exact rules: plain notation for 1e-6 ≤ |x| < 1e21, exponential outside, specific e+/e- formatting, integers up to 2^53, no trailing .0. This needs an audit against the spec's reference, not an assumption of compatibility.

3. Only finite numbers; UTF-8 output; minimal escaping. NaN/Inf must be a hard error in canonical mode (we currently emit null), and escaping must be the minimal JSON set.

Sketch

A serializer variant, not a tree transform — e.g. dump_canonical() or dump(..., serialization_policy::rfc8785). During traversal it sorts each object's keys into a temporary std::vector<const_iterator> with the UTF-16 comparator and swaps in the ECMAScript number formatter. It never mutates the document, so there's no ABI impact.

std::string canonical = j.dump_canonical();   // illustrative name

Implementation notes

• Lives entirely in the serializer (include/nlohmann/detail/output/serializer.hpp); no changes to basic_json storage.
• UTF-16-code-unit key comparator: compare by transcoding UTF-8 → UTF-16 code units (or an equivalent code-unit-order comparison directly on UTF-8) — this is the easy-to-get-wrong part that passes casual tests but fails above the BMP.
• ECMAScript number formatting: verify the exponent thresholds and integer handling against the RFC 8785 reference vectors; reject non-finite values with a clear error.
• Ship with the official JCS test vectors (RFC 8785 appendix / reference implementation test suite).

Open questions

• API shape: a dedicated method vs. a flag/policy on dump(); how it composes with indent (canonical implies no insignificant whitespace, so indent must be ignored or rejected).
• Where to draw the line vs. full RFC 8785 (e.g. Unicode normalization is explicitly not required by JCS — confirm we don't over-implement).
• Behavior for binary values (not representable in JCS) — error.

Dependencies / interactions

• Deliberately overrides the lossless-number idea: raw number tokens must be re-canonicalized (parse → ECMAScript format), not echoed.
• Independent of the borrowed-view and arena ideas.

[0rkz]

Two spec-precision notes on the "What JCS mandates" list:

1. Lone surrogates are a mandated hard error (§3.2.2.2), symmetric with NaN/Infinity.
RFC 8785 §3.2.2.2: "Since invalid Unicode data like 'lone surrogates' (e.g., U+DEAD) may lead to interoperability issues including broken signatures, occurrences of such data MUST cause a compliant JCS implementation to terminate with an appropriate error." The list currently treats surrogates only as the key-sorting concern (point 1) and hard-errors only NaN/Infinity (point 3). In C++ most of this already falls out of UTF-8 validation: the decoder's DFA rejects ED A0..BF, so under the default error_handler_t::strict a UTF-8-encoded lone surrogate already throws. The canonical-mode ask is to pin that rather than inherit it — replace (U+FFFD) and ignore would silently repair exactly the input the RFC requires to fail, so canonical mode probably wants to reject those handlers outright rather than honor them. A U+DEAD negative vector is worth adding alongside the number vectors: the official JCS testdata is positive input/output pairs only, so shipping those vectors won't exercise this clause.

2. "integers up to 2^53" conflates an application-level SHOULD with the serializer's duty.
That bound comes from Appendix B, Note 1, which is an application-interpretation SHOULD — true integers "SHOULD be in the range -9007199254740991 to 9007199254740991" — and which closes by saying how numbers are used in applications "does not affect the JCS algorithm." The serializer's duty (§3.2.2.3) is exact ECMAScript Number::toString serialization of every IEEE-754 double — Appendix B's own sample table includes ~2^68. The C++-specific bite: number_integer_t is int64_t, which holds integers no double can represent (e.g. 2^53+1 = 9007199254740993). Emitting those digits verbatim produces output no ECMAScript-derived implementation can ever emit; converting through double silently changes the value. Canonical mode probably needs that to be a third hard error (or an explicitly documented conversion) — same shape as the NaN/Infinity decision.

Both are cheap to pin now and expensive to retrofit once canonical output bytes become a compatibility promise.

(Disclosure: I work on PayPerByte, where cross-implementation byte-identical JSON is load-bearing for signed payloads — this comment is about the spec text only.)