perf: make ReindentFilter._get_offset linear in tokens instead of quadratic

[zhangj23]

ReindentFilter._get_offset builds the text preceding a token by flattening the entire statement from the start, then keeps only the last line of it. It is called once per token that needs an offset, so reindenting is quadratic in the token count.

This walks backwards from the token instead, collecting only as much preceding text as is guaranteed to contain that final line. Groups are yielded whole rather than descended into, so str() on the result still spells the same text.

Numbers

time.process_time, alternating base and patched subprocesses, min of 3, on nested function calls (SELECT COALESCE(a,b,0) x n FROM t):

n	master	this branch	speedup	master growth	branch growth
200	134.19 ms	45.32 ms	2.96x	x3.00	x2.03
400	462.69 ms	95.31 ms	4.85x	x3.45	x2.10
800	1669.31 ms	217.01 ms	7.69x	x3.61	x2.28

The growth columns are the substance: master grows about 3.6x per doubling while this grows about 2x, so the quadratic term is gone and the speedup keeps increasing with statement size. A 400-row INSERT ... VALUES measures about 2.7x.

Not every statement benefits. Statements whose tokens mostly do not need an offset, such as a long flat WHERE ... AND chain, a large IN list, chained UNIONs, or a wide flat SELECT column list, measure 0.92x to 1.08x, which is unchanged within noise. The win is specifically for nested and grouped constructs, since those are what call _get_offset.

The trade-off, measured

Cost is now proportional to the length of the preceding line multiplied by the nesting depth, because a deeply nested group re-walks the same long preceding line once per level. So there is a shape that regresses.

A 200 KB literal followed by 30 flat grouped expressions is actually 1.27x faster. Nesting those same expressions 40 deep after a 200 KB literal measures 0.57x, roughly 2x slower than master. That is the honest worst case, and I would rather state it than have it found later.

Behaviour

Three details here are load-bearing and each took a correction to get right.

str.splitlines() ignores a trailing boundary, so "a\nb\n" ends on the line "b", and a token ending in a break does not start a fresh line. The walk therefore stops after the third token containing a boundary rather than the second, which guarantees the two boundaries needed even when a \r\n is split across a token edge and counts only once. Trimming each token to its own last line looks like a further win but is wrong for exactly this reason: it changed 219 of 20,435 differential cases when I tried it.

A token spliced into the tree without a parent cannot be located by walking parents. StripCommentsFilter does exactly that, so the walk signals the condition and falls back to flattening. Treating it as "nothing precedes this token" instead mis-indented a CASE containing a stripped comment:

sqlparse.format('select case/*\nc\n*/when a=1 then 2 else 3 end from t',
                reindent=True, strip_comments=True)

master:   'select case\n           when a=1 then 2\n           else 3\n       end\nfrom t'
unfixed:  'select case\n      when a=1 then 2\n      else 3\n       end\nfrom t'

_flatten_up_to_token is effectively part of the surface, so if it has been replaced by a subclass, on the class, or on the instance, the original path is used instead of this walk. All three are verified to still see the override called, with byte-identical output.

Differential testing against master: 20,435 comparisons, being 4,087 SQL statements crossed with 5 formatting configs, 0 mismatches. The corpus covers every line boundary splitlines() recognises, boundaries inside string literals and comments, leading and trailing boundaries, runs of three or more, \r\n split across token edges, deep paren nesting, 200-column selects, and 4,000 randomly assembled fragments.

Checks

pytest gives 494 passed, 2 xfailed, 1 xpassed, exit code 0, matching master exactly. ruff check sqlparse/ is clean.

One small unrelated cleanup in this diff: the HAS_LINE_BREAK character class uses \u2028 and \u2029 escapes rather than the literal separator characters, since literal U+2028 and U+2029 in source are invisible and easy to mangle.