Skip to content

test: adversarial overlap fixtures for tool dispatch#124

Open
clean6378-max-it wants to merge 2 commits into
masterfrom
test/adversarial-dispatch-fixtures
Open

test: adversarial overlap fixtures for tool dispatch#124
clean6378-max-it wants to merge 2 commits into
masterfrom
test/adversarial-dispatch-fixtures

Conversation

@clean6378-max-it

@clean6378-max-it clean6378-max-it commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Closes #116

Summary

The ordering test only checks that predicates sit in the right place in the dispatch table. This adds behavioral tests for JSON blobs that match more than one predicate at the same time.

Each ordering invariant pair gets an overlap fixture. Tests run the blob through parse and check which result type wins, plus a couple of fields that would differ if the wrong handler ran.

Parametrize ids are shared between the ordering and adversarial test files now, so they cannot drift apart. A small sync test keeps fixture keys lined up with those ids.

Also covered: retrieval with no task_id or message (narrow path only), and a monkeypatch that swaps plan below file_write. Wrong order yields file_write; the plan assertion fails.

Test plan

  • adversarial + ordering pytest
  • dispatch sync + real session fixtures
  • ruff on both test files
  • mypy on tests (mypy-tests.ini, follow-imports skip)
  • swap two predicates in dispatch locally, confirm adversarial test fails, revert

Week 28 item #4. Blocks Thursday dispatch simplification (#6).

Summary by CodeRabbit

  • Tests
    • Added adversarial coverage for tool-result dispatch using overlapping payload shapes to verify the documented “first match wins” classification.
    • Added boundary and regression tests to ensure correct selection for narrow vs. broader retrieval/task-message variants and to confirm behavior flips when dispatch precedence is inverted.
    • Consolidated ordering test parameter IDs into a shared list to keep coverage consistent across related test suites.

Add test_tool_dispatch_adversarial.py with overlap JSON for every
ORDERING_INVARIANTS pair, checking winners through _parse_tool_result.
Share ORDERING_INVARIANT_IDS with the ordering test module. Tighten the
sync guard and add a monkeypatch case that shows plan/file_write
misclassification when dispatch order is wrong.
@coderabbitai

coderabbitai Bot commented Jul 8, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 3523a415-9df9-4c2c-a4dc-2f1117e2e45e

📥 Commits

Reviewing files that changed from the base of the PR and between e0a9054 and ed7f91a.

📒 Files selected for processing (1)
  • tests/test_tool_dispatch_adversarial.py
🚧 Files skipped from review as they are similar to previous changes (1)
  • tests/test_tool_dispatch_adversarial.py

📝 Walkthrough

Walkthrough

This PR adds adversarial tool-dispatch tests that exercise overlapping toolUseResult payloads against _parse_tool_result, and updates the existing ordering test to reuse a shared ORDERING_INVARIANT_IDS list.

Changes

Adversarial Dispatch Overlap Tests

Layer / File(s) Summary
Shared ordering invariant IDs
tests/test_tool_dispatch_ordering.py
Adds ORDERING_INVARIANT_IDS and reuses it in the existing parametrized ordering test.
Adversarial fixtures and assertion helpers
tests/test_tool_dispatch_adversarial.py
Adds overlapping fixture dicts, imports dispatch helpers, and defines typed assertion helpers for expected parsed fields.
Invariant-behavior mapping and parametrized/boundary tests
tests/test_tool_dispatch_adversarial.py
Maps invariant IDs to fixture/assertion pairs, then checks the documented winner behavior and the narrow retrieval boundary case.
Regression and coverage-consistency tests
tests/test_tool_dispatch_adversarial.py
Adds a monkeypatch regression test for predicate reordering and a coverage check tying invariants to the adversarial fixtures.

Estimated code review effort: 2 (Simple) | ~12 minutes

Possibly related issues

Possibly related PRs

Suggested reviewers: timon0305, wpak-ai

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is concise and accurately describes the new adversarial tool dispatch fixtures.
Linked Issues check ✅ Passed The PR adds adversarial _parse_tool_result tests for overlapping predicates, boundary shapes, and a reordering regression as requested.
Out of Scope Changes check ✅ Passed The changes stay focused on tool dispatch behavioral tests and a small ordering-test ID refactor.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch test/adversarial-dispatch-fixtures

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
tests/test_tool_dispatch_adversarial.py (1)

137-141: 🎯 Functional Correctness | 🔵 Trivial

Sync check verifies set/count equality, not positional alignment.

This won't catch a case where ORDERING_INVARIANTS and ORDERING_INVARIANT_IDS have matching counts/keys but a shifted order (e.g., an invariant inserted mid-list without its ID inserted at the same index), which would mislabel the ids= in the ordering file's parametrize. Given the current fixed 4-entry lists, this is a low-probability edge case.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/test_tool_dispatch_adversarial.py` around lines 137 - 141, The current
test only checks counts and set equality, so it can miss positional drift
between ORDERING_INVARIANTS and ORDERING_INVARIANT_IDS. Update
test_ordering_invariants_have_adversarial_coverage (and, if needed, the related
ORDERING_INVARIANTS/ORDERING_INVARIANT_IDS assertions) to verify the two
sequences align by index, not just by membership, so a mid-list insertion or
reorder is detected and the ids= mapping stays correct.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tests/test_tool_dispatch_adversarial.py`:
- Around line 137-141: The current test only checks counts and set equality, so
it can miss positional drift between ORDERING_INVARIANTS and
ORDERING_INVARIANT_IDS. Update
test_ordering_invariants_have_adversarial_coverage (and, if needed, the related
ORDERING_INVARIANTS/ORDERING_INVARIANT_IDS assertions) to verify the two
sequences align by index, not just by membership, so a mid-list insertion or
reorder is detected and the ids= mapping stays correct.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 806dc56a-ddbe-4f15-a185-2bb57afd26c5

📥 Commits

Reviewing files that changed from the base of the PR and between a725bd2 and e0a9054.

📒 Files selected for processing (2)
  • tests/test_tool_dispatch_adversarial.py
  • tests/test_tool_dispatch_ordering.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

claude-code-chat-browser: Adversarial predicate-overlap fixtures for tool dispatch classification

1 participant