Skip to content

Add deterministic route contract verification script#58

Closed
Disaster-Terminator wants to merge 1 commit intoreview/route-contract-hardening-2026-05-05from
add-contract-verification-script-for-routes-rsmzko
Closed

Add deterministic route contract verification script#58
Disaster-Terminator wants to merge 1 commit intoreview/route-contract-hardening-2026-05-05from
add-contract-verification-script-for-routes-rsmzko

Conversation

@Disaster-Terminator
Copy link
Copy Markdown
Owner

Motivation

  • Provide a fast, deterministic local checker to detect route_id / target_model drift and misconfigured eval expectations before public readiness.
  • Ensure route configuration references (fallbacks, hard_rules, sample expectations) are validated without any external embedding or LiteLLM calls.

Description

  • Add scripts/verify_route_contract.py, a standalone verifier that defaults to config/routes.yaml and config/eval_cases.yaml and returns a non-zero exit on failure.
  • The verifier checks that the route_model/entry_model is not used as a route_id, fallback_route_id/default_route exists, every hard_rules route_id exists, every route resolves to a non-empty target_model, and no resolved target_model equals the entry model, and that eval expect values reference route_id (not target_model).
  • The script prints readable error lines for each problem and succeeds silently with a short success message when no issues are found.
  • Add tests in tests/test_verify_route_contract.py including a passing config fixture and a failing fixture that exercise drift and missing-reference errors.

Testing

  • Ran the full test suite with uv run python -m pytest -q and all tests passed: 152 passed, 1 warning (4.11s).
  • Ran the verifier with uv run python scripts/verify_route_contract.py against repository configs and it succeeded with a pass message.
  • Ran the evaluator with uv run python scripts/eval_routes.py --mock-embeddings and observed all sample eval cases pass (10/10).

Codex Task

Copy link
Copy Markdown

@sourcery-ai sourcery-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Disaster-Terminator! 👋

Your private repo does not have access to Sourcery.

Please upgrade to continue using Sourcery ✨

@qodo-code-review
Copy link
Copy Markdown

ⓘ You've reached your Qodo monthly free-tier limit. Reviews pause until next month — upgrade your plan to continue now, or link your paid account if you already have one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant