Description
In \openagent_eval/diagnosis/chunking.py, the content gap detection uses naive substring matching that silently ignores non-ASCII characters (Unicode, accented characters, CJK, emoji, etc.).
Current Behavior
If the retrieved context contains non-ASCII characters, the gap detection might miss them or produce false positives because string matching doesn't account for Unicode normalization.
Expected Behavior
Content gap detection should handle non-ASCII characters correctly using Unicode-aware string comparison (\unicodedata.normalize\ or similar).
Files to Look At
- \openagent_eval/diagnosis/chunking.py\ — the chunking quality analyzer
- \ ests/unit/test_diagnosis/test_chunking.py\ — existing tests
Good First Issue ✅
A contained fix with clear before/after behavior. Good introduction to Unicode handling in Python.
Description
In \openagent_eval/diagnosis/chunking.py, the content gap detection uses naive substring matching that silently ignores non-ASCII characters (Unicode, accented characters, CJK, emoji, etc.).
Current Behavior
If the retrieved context contains non-ASCII characters, the gap detection might miss them or produce false positives because string matching doesn't account for Unicode normalization.
Expected Behavior
Content gap detection should handle non-ASCII characters correctly using Unicode-aware string comparison (\unicodedata.normalize\ or similar).
Files to Look At
Good First Issue ✅
A contained fix with clear before/after behavior. Good introduction to Unicode handling in Python.