fix(fieldNorm): count tabs and newlines as word separators#836
Conversation
|
Thanks @chatman-media, good catch. Same class of bug as #830 and the fix is the right shape. Confirmed it's real: with only charCode 32 counting as a separator, Tests pass and the dist rebuild is clean. Merging. One thing I'll tidy in a follow-up, not on you: the comment says "any whitespace character" but the predicate is ASCII whitespace + NBSP, not the full |
The word counter added in krisk#830 only checked for charCode 32 (plain ASCII space), so any field that separated words with a tab or newline instead was scored as a single long word. That gives it an artificially better field-length norm than a same-content field using regular spaces - e.g. 'foo\tbar\tbaz' scores 20x better than 'foo bar baz' even though they're the same three words. Added a small isWordSeparator helper covering the usual whitespace range (tab/LF/VT/FF/CR, space, NBSP) instead of the single charCode check, plus a test that fails without the fix.
krisk
force-pushed
the
fix/fieldnorm-whitespace-separators
branch
from
01aeb1e to
a692bbd
Compare
…mment Follow-up to #836. The isWordSeparator comment claimed "any whitespace character" but the predicate only covers common ASCII whitespace plus NBSP, not the full Unicode \s set (ideographic space, en/em spaces, U+2028/2029, etc.). Narrow the wording so the scope reads as deliberate rather than an oversight, and add assertions for CR, VT, FF, and NBSP to lock in the set the predicate actually claims.
2 participants
Following up on #830 - the word counter still only checks charCode 32 (plain space), so tabs and newlines between words don't count as separators. A field like
'foo\tbar\tbaz'gets scored as one giant word instead of three, which makes its field-length norm ~20x better than the equivalent space-separated text. Anyone with multi-line descriptions or tab-formatted fields is getting skewed rankings for no good reason.Fix is the same shape as #830's: swap the single charCode check for a small helper covering the normal whitespace range (tab/LF/VT/FF/CR, space, NBSP). Added a test asserting tab/newline-separated text scores the same as space-separated text with the same words - it fails on main and passes with the fix. Full suite (372 tests), typecheck, lint and format:check all still pass.