test: adversarial overlap fixtures for tool dispatch#124
test: adversarial overlap fixtures for tool dispatch#124clean6378-max-it wants to merge 2 commits into
Conversation
Add test_tool_dispatch_adversarial.py with overlap JSON for every ORDERING_INVARIANTS pair, checking winners through _parse_tool_result. Share ORDERING_INVARIANT_IDS with the ordering test module. Tighten the sync guard and add a monkeypatch case that shows plan/file_write misclassification when dispatch order is wrong.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThis PR adds adversarial tool-dispatch tests that exercise overlapping ChangesAdversarial Dispatch Overlap Tests
Estimated code review effort: 2 (Simple) | ~12 minutes Possibly related issues
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
🧹 Nitpick comments (1)
tests/test_tool_dispatch_adversarial.py (1)
137-141: 🎯 Functional Correctness | 🔵 TrivialSync check verifies set/count equality, not positional alignment.
This won't catch a case where
ORDERING_INVARIANTSandORDERING_INVARIANT_IDShave matching counts/keys but a shifted order (e.g., an invariant inserted mid-list without its ID inserted at the same index), which would mislabel theids=in the ordering file's parametrize. Given the current fixed 4-entry lists, this is a low-probability edge case.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_tool_dispatch_adversarial.py` around lines 137 - 141, The current test only checks counts and set equality, so it can miss positional drift between ORDERING_INVARIANTS and ORDERING_INVARIANT_IDS. Update test_ordering_invariants_have_adversarial_coverage (and, if needed, the related ORDERING_INVARIANTS/ORDERING_INVARIANT_IDS assertions) to verify the two sequences align by index, not just by membership, so a mid-list insertion or reorder is detected and the ids= mapping stays correct.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@tests/test_tool_dispatch_adversarial.py`:
- Around line 137-141: The current test only checks counts and set equality, so
it can miss positional drift between ORDERING_INVARIANTS and
ORDERING_INVARIANT_IDS. Update
test_ordering_invariants_have_adversarial_coverage (and, if needed, the related
ORDERING_INVARIANTS/ORDERING_INVARIANT_IDS assertions) to verify the two
sequences align by index, not just by membership, so a mid-list insertion or
reorder is detected and the ids= mapping stays correct.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 806dc56a-ddbe-4f15-a185-2bb57afd26c5
📒 Files selected for processing (2)
tests/test_tool_dispatch_adversarial.pytests/test_tool_dispatch_ordering.py
Closes #116
Summary
The ordering test only checks that predicates sit in the right place in the dispatch table. This adds behavioral tests for JSON blobs that match more than one predicate at the same time.
Each ordering invariant pair gets an overlap fixture. Tests run the blob through parse and check which result type wins, plus a couple of fields that would differ if the wrong handler ran.
Parametrize ids are shared between the ordering and adversarial test files now, so they cannot drift apart. A small sync test keeps fixture keys lined up with those ids.
Also covered: retrieval with no task_id or message (narrow path only), and a monkeypatch that swaps plan below file_write. Wrong order yields file_write; the plan assertion fails.
Test plan
Week 28 item #4. Blocks Thursday dispatch simplification (#6).
Summary by CodeRabbit