to_msgpack() silently emits corrupt output (or reads out of bounds) above the documented 4294967295 size limit

[nlohmann]

Description

docs/mkdocs/docs/features/binary_formats/messagepack.md states that strings, byte strings, arrays, and objects with more than 4294967295 elements/bytes "can not be converted to a MessagePack value". Nothing in the writer enforces that, and the same page also promises "Any MessagePack output created by to_msgpack can be successfully parsed by from_msgpack".

All four size-prefixed cases in write_msgpack end their if/else if chain at std::uint32_t with no final else:

value type	site	largest branch
string	binary_writer.hpp:554-587	0xDB str 32
array	binary_writer.hpp:589-609	0xDD array 32
binary	binary_writer.hpp:619-686	0xC6/0xC9 bin 32 / ext 32
object	binary_writer.hpp:702-722	0xDF map 32

When N exceeds UINT32_MAX no branch runs, so no header byte is written at all and the writer proceeds straight to the payload. Two failure modes:

• array / object — the elements are written through real iterators, so to_msgpack() returns successfully with a headerless, corrupt document.
• string / binary — write_characters(data(), N) is then called with the oversized N and reads past the end of the buffer, crashing the process.

CBOR does not have this problem: its chains end in a uint64_t branch (binary_writer.hpp:254-260, 292-298, 355-361, 394-400). UBJSON does not either: it routes sizes through write_number_with_ubjson_prefix, which reaches the 64-bit L marker. MessagePack is the only format whose wire size fields top out at 32 bits, and the only one with no diagnostic.

This is the same defect class as #5314 (to_bson() and INT32_MAX).

Reproduction steps

Serialize a container that reports a size above UINT32_MAX. Using the size-reporting technique from #5314, an array type keeps real iterators so the corrupt output is observable without any out-of-bounds access.

Expected vs. actual results

Expected: a thrown exception, as to_bson() will do after #5314.

Actual, for an array reporting size() == 2^33 with three real elements:

array reports size() = 8589934592

to_msgpack (3 bytes): 01 02 03           <-- no length prefix at all
to_cbor   (12 bytes): 9b 00 00 00 02 00 00 00 00 01 02 03
to_ubjson (17 bytes): 5b 23 4c 00 00 00 02 00 00 00 00 69 01 69 02 69 03

CBOR emits 0x9B + a 64-bit count, UBJSON emits [ # L + a 64-bit count. MessagePack emits the elements with no header. Reading that back:

from_msgpack(01 02 03) strict:     THROW: [json.exception.parse_error.110] parse error at byte 2:
    syntax error while parsing MessagePack value: expected end of input; last byte: 0x02
from_msgpack(01 02 03) non-strict: 1

so in non-strict mode the corruption is silent in both directions.

For value_t::binary and value_t::string the same input instead terminates the process with SIGSEGV inside write_characters.

Minimal code example

#include <nlohmann/json.hpp>
#include <iostream>
#include <vector>

// Reports an oversized size() but iterates its real elements, so the writer
// takes the >UINT32_MAX path with no out-of-bounds access.
template<typename T, typename A = std::allocator<T>>
struct huge_array : std::vector<T, A>
{
    using std::vector<T, A>::vector;
    std::size_t size() const noexcept { return std::size_t{1} << 33; }
};

using huge_json = nlohmann::basic_json<
    std::map, huge_array, std::string, bool, std::int64_t, std::uint64_t,
    double, std::allocator, nlohmann::adl_serializer, std::vector<std::uint8_t>, void>;

int main()
{
    huge_json j = huge_json::array();
    j.push_back(1); j.push_back(2); j.push_back(3);

    const auto v = huge_json::to_msgpack(j);        // returns normally
    std::cout << v.size() << " bytes\n";            // 3 -- the elements, no header
}

Swapping huge_array for a BinaryType that reports an oversized size() reproduces the out-of-bounds read instead.

Error messages

None at serialization time — that is the bug.

Compiler and operating system

g++ 13.3.0, Ubuntu 24.04, -std=c++11

Library version

develop @ 8ec98e2

Would the fix be breaking?

No API/ABI change: no signature, type, or enumerator is affected, and the header stays source- and binary-compatible.

• What changes: to_msgpack() gains a throwing path for inputs that today return corrupt output or crash. Since the docs already say these values cannot be converted, the throw enforces a stated limitation rather than introducing a new one. Nobody can be depending on the current behavior in any useful way — the output is unparseable, and half the cases don't even return.
• Exception safety is preserved. to_msgpack() documents a strong guarantee; throwing before any bytes are committed keeps that, and the JSON value is never modified.
• Coordinate with to_bson() silently emits corrupt documents when a length exceeds INT32_MAX #5314. That PR introduces out_of_range.412 for the BSON INT32_MAX case and a to_bson_length() helper. The MessagePack fix should reuse the same exception id and a parallel helper rather than minting a second one, so the two formats report the same way. If to_bson() silently emits corrupt documents when a length exceeds INT32_MAX #5314 lands first this becomes a small follow-up.
• Docs. The "Size constraints" warning stays accurate; the "Complete mapping" note that promises from_msgpack can parse any to_msgpack output only becomes true once this is fixed.
• Testing. As to_bson() silently emits corrupt documents when a length exceeds INT32_MAX #5314 found, reproducing this for real needs multi-gigabyte values, so the practical approach is a file-local type that reports an oversized size() without allocating — the array variant above is the safe one to test with, since the string/binary variants read out of bounds on unpatched code.

The only judgement call is whether an over-large value should throw or be silently truncated to a shorter encoding; truncation would corrupt data just as badly, so throwing seems clearly right and matches #5314.

[CHIRANJEET1729DAS]

Hi @nlohmann! I'd like to work on this if it's still open.

My plan: add an else branch after the uint32_t check in each of the
four size-prefixed cases in write_msgpack() (string, array, binary,
object), throwing out_of_range.412 via detail::value_in_range_ofstd::uint32_t
before any bytes are written -- reusing the exception id and helper
pattern from #5314 as suggested, rather than introducing a new one.

Happy to adjust based on how #5314's review lands. Could I get this
assigned to me?