Evaluation Tool Improvements - Human Reviewer Findings (Dec 2025)

[mmcky]

Opus 4.5 Evaluation Tool - Human Reviewer Findings

Date: December 4, 2025
Reviewer: @HumphreyYang
PRs Reviewed: 24 translation PRs in QuantEcon/test-translation-sync.zh-cn
PR Range: #361 - #384

---

Executive Summary

HumphreyYang reviewed all 24 translation PRs and the corresponding Opus 4.5 evaluation comments. Overall, the evaluation tool performs well, with accurate assessments and helpful suggestions.

Key Findings - ALL RESOLVED ✅

Category	Finding	Status
✅ Strengths	Assessments generally accurate, summaries helpful, glossary compliance well-checked	N/A
✅ Fixed	Suggestions now focus on changed sections only	05a2e23
✅ Fixed	Configurable max suggestions with improved prompt	0a3ca1f
✅ Fixed	Markdown syntax validation in prompts	7710457
✅ Fixed	File rename handling - transfers translation, deletes old file	403fd63
✅ Fixed	PR #381 - "Changed Sections" list bug	ffa2b02
✅ Fixed	Glossary additions for game theory terms	c451963
ℹ️ Expected	Same suggestions repeated across multiple PRs (test suite uses similar documents)	N/A

---

Improvements Implemented (v0.6.1)

1. Focus Suggestions on Changed Content ✅

Commit: 05a2e23

The evaluator now computes changed sections by comparing before/after content and instructs Claude to focus suggestions ONLY on changed content.

2. Configurable Max Suggestions ✅

Commit: 0a3ca1f

Allows 0-5 suggestions by default (was ~2). Configurable via --max-suggestions CLI flag.

3. Markdown Syntax Validation ✅

Commit: 7710457

LLM-based syntax checking in translator and evaluator prompts. Deterministic tool proposed: QuantEcon/meta#268

4. File Rename Handling ✅

Commit: 403fd63

Detects status: 'renamed' files, transfers existing translation to new filename, deletes old file.

5. Changed Sections Bug Fix ✅

Commit: ffa2b02

Fixed bug where "Changed Sections" list included non-existent sections.

6. Glossary Additions ✅

Commit: c451963

Added game theory terms (357 total, was 355):

• "folk theorem" → "无名氏定理"
• "grim trigger strategy" → "冷酷策略"

---

Remaining Items (Low Priority)

• Build myst-lint deterministic linting tool (PROJECT: myst-lint - MyST Markdown Linting Tool meta#268)
• Consider document-level vs PR-level review modes

---

Summary Statistics

Metric	Count
Total PRs Reviewed	24
Issues Identified	6
Issues Fixed	6 ✅
Remaining	0 (2 low-priority future items)

---

Full report: tool-test-action-on-github-reviewer-2025-12-04.md

[mmcky]

Validation Checklist for v0.6.1 Changes

Once we run the GitHub test suite (./tool-test-action-on-github/test-action-on-github.sh), we need to validate the following fixes:

1. File Rename Handling (Test #20)

Source PR: Renames lecture.md → linear-algebra.md + updates _toc.yml

• Target PR creates linear-algebra.md with translated content
• Target PR deletes lecture.md (not left orphaned)
• Existing translation is preserved/transferred (not full re-translation)
• Heading-map is preserved in the new file
• _toc.yml is updated correctly
• PR description shows correct file operations (rename detected)

2. Markdown Syntax Validation

Any test with translation output

• No ####NoSpace errors (space after # in headings)
• Evaluator reports syntax errors if any exist
• PR comments show 🔴 markers for syntax issues (if any)

3. Changed Sections Focus (All tests)

Evaluator suggestions

• Suggestions focus only on changed content
• No suggestions for unchanged sections
• Changed sections list is accurate (no phantom sections)

4. Configurable Max Suggestions

Evaluator behavior

• Default allows 0-5 suggestions
• Excellent translations can have empty suggestions array
• No invented/filler issues

---

Test Commands

# Run all GitHub tests
./tool-test-action-on-github/test-action-on-github.sh

# Run evaluation on resulting PRs
cd tool-test-action-on-github/evaluate
npm run evaluate

# Check specific PR for rename test (#20)
gh pr view <PR_NUMBER> --repo QuantEcon/test-translation-sync.zh-cn

Key PRs to Watch

• Test Phase 3b: Forward resync command #20 (rename): Most critical for file rename fix validation
• Test LLM makes beneficial improvements to existing translations during updates #1-8: Basic scenarios for syntax/suggestion validation

[mmcky]

thanks @HumphreyYang I am about to re-run the tests using https://github.com/QuantEcon/action-translation-sync/releases/tag/v0.6.1. Really appreciate your detailed review. It has been a HUGE help.