Skip to content

Improve LoCoMo source-turn recall confidence - #334

Draft
okJiang wants to merge 4 commits into
mainfrom
codex/locomo-20260526T213228-source-turn-boost-v2
Draft

Improve LoCoMo source-turn recall confidence#334
okJiang wants to merge 4 commits into
mainfrom
codex/locomo-20260526T213228-source-turn-boost-v2

Conversation

@okJiang

@okJiang okJiang commented May 26, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a guarded source-turn evidence bonus for LoCoMo recall confidence. The boost applies only to insight candidates with same-speaker source turns, exact/general query shapes, and bounded confidence thresholds; it skips duration, frequency, modal/inference, location-like, non-insight, and already-high-confidence candidates.

Benchmark Evidence

  • Baseline run: 20260526T161100
  • Baseline artifact: /Users/jiangxianjie/code/okjiang/harness/mem9-locomo/20260526T161100/benchmark-results.json
  • Baseline Overall LLM (micro): 0.6396103896
  • Candidate run: 20260526T213228
  • Candidate artifact: /Users/jiangxianjie/code/okjiang/harness/mem9-locomo/20260526T213228/benchmark-results.json
  • Candidate Overall LLM (micro): 0.6487012987
  • Absolute delta: +0.0090909091
  • Promotion target: >= 0.6446103896
  • Counts/config: 1986 total QA, 1540 LLM-judged QA, no sample/category filters, model qwen3.6-plus, dataset /Users/jiangxianjie/code/mem9-locomo-workspace/mem9-benchmark/locomo/data/locomo10.json
  • Ordered-row LLM flips versus baseline: 59 wins / 45 losses / 1436 ties

Category LLM deltas versus baseline:

  • Cat1: 0.3971631206 (+0.0000000000)
  • Cat2: 0.7445482866 (+0.0000000000)
  • Cat3: 0.3958333333 (+0.0208333333)
  • Cat4: 0.7253269917 (+0.0142687277)
  • Cat5: N/A

Known non-primary regressions:

  • Overall F1: 0.6123243694 (-0.0016045183)
  • Overall Evidence Recall: 0.6888497265 (-0.0064467109)
  • Cat5 Evidence Recall: 0.6580717489 (-0.0269058296)

Source Binding

  • Candidate product branch: codex/locomo-20260526T213228-source-turn-boost-v2
  • Candidate product commit: ca32790c0368c0e775757b23c55f8998e6c93b32
  • Comparable baseline product HEAD: bb0a522194275c2c425f920d5928e5e579d9c612
  • Baseline observed mem9 origin/main: fd0e7d38f19d3d84550978e43922284d391182e2
  • Benchmark repo commit: 69d0d9c8fe665259bd3234fe9d16a256af266d34

Local Checks

  • git diff --check
  • go test ./internal/handler -run 'SourceTurn|RecallConfidence|RecallCandidateOptions|AnswerEvidence|QuestionSpecific'
  • go test ./internal/handler ./internal/service

Integrity Notes

The benchmark dataset, gold answers, judge prompts, scoring logic, and category labels were not changed. The run used the Harness ./locomo.sh full benchmark with all samples and all categories.

@okJiang okJiang added the codex label May 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant