fix(scoring): use normalised key weights in _searchObjectList#834
fix(scoring): use normalised key weights in _searchObjectList#834JSap0914 wants to merge 1 commit into
Conversation
_searchObjectList built its key list from this._myIndex.keys, which
stores the raw user-specified weights (before the normalisation that
KeyStore performs). KeyStore normalises weights so they sum to 1, but
_searchObjectList never read from KeyStore — it used the original
values directly.
This caused two bugs:
1. Score underflow for extreme weight values.
A single key always normalises to weight=1.0 (only key = 100%),
so 'weight:1' and 'weight:100' must produce the same score.
With the raw weight in the exponent the formula
Math.pow(Number.EPSILON, 100 * fieldNorm)
underflows to exactly 0, whereas the normalised exponent (1)
gives ~2.2e-16.
2. Score inconsistency between query forms.
Plain string queries went through _searchObjectList (raw weights);
logical Expression queries went through _searchLogical which
already called this._keyStore.get(keyId) (normalised weights).
Identical searches produced different absolute scores depending
on how the query was expressed.
Fix: replace 'const { keys, records } = this._myIndex' with
'const keys = this._keyStore.keys()' so the keys seen by
_findMatches — and therefore by computeScoreSingle — carry
the same normalised weights that KeyStore guarantees.
Closes krisk#833
|
Thanks, the diagnosis is spot on. I landed a slightly wider fix on
Your tests carried over almost as-is. Closing in favor of the version on |
Problem
_searchObjectListbuilt its key list fromthis._myIndex.keys, which stores the raw user-specified weights — before the normalisation thatKeyStoreperforms (weights sum to 1)._searchObjectListnever read fromKeyStore, so it applied the original values directly.This caused two observable bugs:
1. Score underflow for extreme weight values
A single key always normalises to
weight = 1.0(only key = 100 % share). With a raw weight of100, the exact-match formula becomes:whereas the normalised exponent (1.0) gives
~2.2e-16.2. Score inconsistency between query forms
_searchObjectList→ raw weightsExpressionqueries →_searchLogical→this._keyStore.get(keyId)→ normalised weightsIdentical searches against identical data produced different absolute scores depending solely on how the query was expressed.
Fix
Replace
with
KeyStoreandFuseIndexboth iteratethis.options.keysin the same order, so the positional indexingitem[keyIndex]is unaffected.Tests
Three new tests in
test/key-weight-normalization.test.js(import from source):weight:1andweight:2(single key) produce the same scoreweight:100does not underflow to 0All 374 existing tests continue to pass.
Closes #833