Improve LoCoMo source-turn recall confidence - #334
Draft
okJiang wants to merge 4 commits into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a guarded source-turn evidence bonus for LoCoMo recall confidence. The boost applies only to insight candidates with same-speaker source turns, exact/general query shapes, and bounded confidence thresholds; it skips duration, frequency, modal/inference, location-like, non-insight, and already-high-confidence candidates.
Benchmark Evidence
20260526T161100/Users/jiangxianjie/code/okjiang/harness/mem9-locomo/20260526T161100/benchmark-results.jsonOverall LLM (micro):0.639610389620260526T213228/Users/jiangxianjie/code/okjiang/harness/mem9-locomo/20260526T213228/benchmark-results.jsonOverall LLM (micro):0.6487012987+0.0090909091>= 0.64461038961986total QA,1540LLM-judged QA, no sample/category filters, modelqwen3.6-plus, dataset/Users/jiangxianjie/code/mem9-locomo-workspace/mem9-benchmark/locomo/data/locomo10.json59wins /45losses /1436tiesCategory LLM deltas versus baseline:
0.3971631206(+0.0000000000)0.7445482866(+0.0000000000)0.3958333333(+0.0208333333)0.7253269917(+0.0142687277)Known non-primary regressions:
0.6123243694(-0.0016045183)0.6888497265(-0.0064467109)0.6580717489(-0.0269058296)Source Binding
codex/locomo-20260526T213228-source-turn-boost-v2ca32790c0368c0e775757b23c55f8998e6c93b32bb0a522194275c2c425f920d5928e5e579d9c612mem9 origin/main:fd0e7d38f19d3d84550978e43922284d391182e269d0d9c8fe665259bd3234fe9d16a256af266d34Local Checks
git diff --checkgo test ./internal/handler -run 'SourceTurn|RecallConfidence|RecallCandidateOptions|AnswerEvidence|QuestionSpecific'go test ./internal/handler ./internal/serviceIntegrity Notes
The benchmark dataset, gold answers, judge prompts, scoring logic, and category labels were not changed. The run used the Harness
./locomo.shfull benchmark with all samples and all categories.