operator>> does not restore the character that terminates a number

[nlohmann]

Summary

operator>> is documented to leave the stream "positioned right after" the value it parsed, so that it can be called repeatedly on a stream of concatenated JSON values. That holds for strings, arrays, objects and literals, but not for numbers: a number is only terminated by the character following it, and that character is consumed from the stream and never put back. The next extraction therefore starts one byte too late.

This came up while reviewing #5326, which documents the parse() vs operator>> strictness difference and repeats the positioning guarantee from operator>>'s notes. Before we document the guarantee more prominently, we should decide whether to make it true or to qualify it.

Reproduction

#include <nlohmann/json.hpp>
#include <iostream>
#include <sstream>

int main()
{
    std::istringstream input("1true");
    nlohmann::json j1;
    input >> j1;                       // j1 == 1

    std::string rest;
    char c;
    while (input.get(c)) { rest += c; }
    std::cout << j1 << " | rest = \"" << rest << "\"\n";
}

Output:

1 | rest = "rue"

Expected (per the documented guarantee): rest = "true", and a second input >> j2 yielding true. Instead the second extraction throws:

[json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - invalid literal; last read: 'r'

More cases (current behaviour):

input	first value	remaining stream
1true	1	rue ❌
1 true	1	true ✔ (the eaten byte was whitespace)
truefalse	true	false ✔
[1][2]	[1]	[2] ✔
null null	null	null ✔

So the guarantee only breaks when a number is immediately followed by a non-whitespace character — 1true, 1[2], 1{}, 1"a". Concatenated values that are whitespace-separated, or whose first value is not a number, work as documented.

Cause

lexer::scan_number() reads one character past the number and ungets it:

json/include/nlohmann/detail/input/lexer.hpp

Lines 1277 to 1280 in dd24e2d

	scan_number_done:
	// unget the character after the number (we only read it to know that
	// we are done scanning a number)
	unget();

But lexer::unget() is deliberately simulated — it only rewinds the lexer's own bookkeeping, not the input:

We implement unget by setting variable next_unget to true. The input is not changed - we just simulate ungetting by modifying chars_read_total, chars_read_current_line, and token_string.

and input_stream_adapter::get_character() consumes via sbumpc() with no corresponding sungetc():

json/include/nlohmann/detail/input/input_adapters.hpp

Lines 127 to 136 in dd24e2d

	std::char_traits<char>::int_type get_character()
	{
	auto res = sb->sbumpc();
	// set eof manually, as we don't use the istream interface.
	if (JSON_HEDLEY_UNLIKELY(res == std::char_traits<char>::eof()))
	{
	is->clear(is->rdstate() | std::ios::eofbit);
	}
	return res;
	}

The simulated unget is correct and sufficient for parse()/accept(), which require the input to end after the value anyway. It is only observable when the caller keeps using the stream afterwards — i.e. operator>>, and sax_parse with strict == false.

Options

1. Fix the adapter. Give input_stream_adapter a real unget (sungetc(), or sputbackc() for the character just read) and have lexer::unget() propagate it to the adapter when the adapter supports it. This makes the documented guarantee hold. It is a behaviour change: code that today relies on the extra byte being swallowed would see it again. sungetc() can also fail if the streambuf has no putback room, which needs a decision (currently sbumpc() is used precisely to avoid depending on std::istream state).
2. Qualify the documentation. Keep the implementation as is and state in operator>>'s notes and in features/parsing that concatenated values must be separated by whitespace when the preceding value is a number. Cheapest, no behaviour change, but leaves a sharp edge.
3. Both: qualify the docs now (so docs: document standards compliance and parse() vs operator>> strictness #5326 can land accurately) and treat the adapter fix as a candidate for the next major release.

I lean towards 3, but since option 1 changes observable behaviour of operator>>, marking this for discussion.

Which version of the library did you use?

develop (dd24e2d)

---

This issue was written by Claude Code.

[Patrick10199]

Option 3 seems safest. Propagating lexer::unget() directly to input_stream_adapter would not be sufficient because next_unget already replays the cached character on the next lexer read, potentially causing the terminator to be consumed twice. A full fix appears to need an explicit parser-finalization path that restores only a still-pending character, plus defined behavior when sungetc() fails. That seems better suited to the next major release it's a pretty big one; the documentation should be qualified now.

[nlohmann]

I created #5343 as a short-term documentation fix. A complete fix is drafted in #5344 - but it would change the behavior of operator>>.