fix(eval): improve rubric text normalization for judge-garbled output#6080
Open
tottenjordan wants to merge 3 commits into
Open
fix(eval): improve rubric text normalization for judge-garbled output#6080tottenjordan wants to merge 3 commits into
tottenjordan wants to merge 3 commits into
Conversation
Author
|
@surajksharma07 PR is up per your suggestion in #6072. Includes the NFKC normalization, smart-char mapping, and uniqueness guard on the substring fallback. 46 tests pass (44 existing + 2 new). |
Collaborator
|
/adk-pr-analyze |
Collaborator
|
Hi @tottenjordan , Thank you for your contribution! We appreciate you taking the time to submit this pull request. Please fix formatting errors. |
rohityan
added
the
request clarification
Author
|
@googlebot I signed it. |
tottenjordan
force-pushed
the
fix/rubric-text-normalization
branch
2 times, most recently
from
67faae3 to
e29c0f3
Compare
Replace _normalize_text's simple lower().strip() with NFKC unicode normalization, smart-quote/dash translation, and markdown artifact stripping. Add substring fallback with uniqueness guard to convert_auto_rater_response_to_score for cases where normalization alone isn't sufficient. Fixes google#6072
Address reviewer feedback on google#6072: - Guard `if not rubric and normalized_rubric_text:` prevents empty judge Property: lines from matching every rubric via substring - Guard `if ct and` prevents empty rubric keys from matching - Add logger.debug when substring fallback rescues a match to track judge drift in eval logs - Add test_empty_property_text_does_not_match test case
tottenjordan
force-pushed
the
fix/rubric-text-normalization
branch
from
e29c0f3 to
72ece22
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #6072
_normalize_textcurrently only does.lower().strip(), so judge-model garbling (markdown bullets, smart quotes, bold formatting, extra whitespace) causes exact rubric match failures. Rubric scores get silently dropped with only a warning log.Changes:
_normalize_textwith NFKC unicode normalization, smart-quote/dash translation, and markdown artifact strippingconvert_auto_rater_response_to_score— accepts a match only when exactly one rubric candidate matches, preventing ambiguous cross-matchingGarbling patterns handled:
- The response correctly uses toolsthe response correctly uses tools* **The response correctly uses tools**the response correctly uses tools"The response correctly uses tools"(smart quotes)the response correctly uses tools— The response correctly uses tools(em dash)the response correctly uses tools– The response correctly uses tools(en dash)the response correctly uses tools• The response correctly uses tools(unicode bullet)the response correctly uses toolsThe response correctly uses tools(double spaces)the response correctly uses toolsThe response… uses tools(ellipsis)the response... uses toolsréponse(accented chars)réponse(preserved)Per @surajksharma07's suggestion in #6072: uses NFKC normalization instead of ascii-ignore (preserves non-English rubrics), and adds uniqueness guard on the substring fallback.
Validation
test_rubric_based_evaluator.pygepa-run-8fb68a8f52-20260611-115752) with 4 rubric-based criteria, gemini-2.5-pro judge — zero "not found in rubrics" warnings across all generationsTest plan
pytest tests/unittests/evaluation/test_rubric_based_evaluator.py -v— all 46 passTestNormalizeTextcovers all garbling patterns from issueTestSubstringFallbackUniquenessGuardverifies unique match accepted, ambiguous match rejected