Skip to content

Add redacted route quality review runbook, synthetic fixture, and tests#57

Closed
Disaster-Terminator wants to merge 1 commit intoreview/route-contract-hardening-2026-05-05from
improve-redacted-route-review-runbook-mjpb2y
Closed

Add redacted route quality review runbook, synthetic fixture, and tests#57
Disaster-Terminator wants to merge 1 commit intoreview/route-contract-hardening-2026-05-05from
improve-redacted-route-review-runbook-mjpb2y

Conversation

@Disaster-Terminator
Copy link
Copy Markdown
Owner

Motivation

  • Provide operators a concise, auditable workflow for running redacted route decision reviews without relying on raw prompts or private logs.
  • Ensure imports and evaluations enforce the contract that expect is a product-level route_id and not a deployment target_model.
  • Supply a tiny synthetic redacted fixture so tooling and CI can validate the redaction and route-id expectations.

Description

  • Add an operator runbook at docs/runbooks/redacted-route-quality-review.md describing JSONL collection rules, required fields (text, expect, redacted), import and review flows, and safe PASS/FAIL interpretation.
  • Add a small synthetic redacted sample tests/samples/redacted_review_fixture.jsonl that contains redacted:true and expect:"strong" for tooling checks.
  • Extend tests/test_import_review_samples.py to import Path and add two tests that assert the fixture remains redacted and that convert_review_samples accepts it when validating against a set of allowed route_ids.
  • Keep runtime routing behavior unchanged and avoid adding production samples or secrets.

Testing

  • Ran uv run python -m pytest -q and all tests passed (152 passed, 1 warning).
  • Ran uv run python scripts/eval_routes.py --mock-embeddings and the eval run completed successfully (10 eval case(s) passed).

Codex Task

Copy link
Copy Markdown

@sourcery-ai sourcery-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Disaster-Terminator! 👋

Your private repo does not have access to Sourcery.

Please upgrade to continue using Sourcery ✨

@qodo-code-review
Copy link
Copy Markdown

ⓘ You've reached your Qodo monthly free-tier limit. Reviews pause until next month — upgrade your plan to continue now, or link your paid account if you already have one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant