Skip to content

Add operator runbook and redacted fixture guardrails for route review#63

Closed
Disaster-Terminator wants to merge 1 commit intoreview/route-contract-hardening-2026-05-05from
improve-redacted-route-review-runbook-p1c6uk
Closed

Add operator runbook and redacted fixture guardrails for route review#63
Disaster-Terminator wants to merge 1 commit intoreview/route-contract-hardening-2026-05-05from
improve-redacted-route-review-runbook-p1c6uk

Conversation

@Disaster-Terminator
Copy link
Copy Markdown
Owner

Motivation

  • Make the redacted route-review workflow usable and auditable without relying on raw prompts or private logs by providing operator guidance and machine-checkable fixtures.
  • Enforce the contract that review cases must use expect as a product-level route_id (not target_model) so imports and audits compare stable route identifiers.
  • Prevent accidental promotion of unredacted or target-model-based samples into eval banks by adding validation and a simple synthetic fixture for CI checks.

Description

  • Add a concise operator runbook at docs/redacted_route_review_runbook.md describing how to collect redacted samples, required JSONL fields, import with route validation, how to run review_decisions against the decision endpoint, and how to interpret PASS/FAIL safely.
  • Add a tiny synthetic redacted JSONL fixture at tests/samples/redacted_review_samples.synthetic.jsonl to serve as an operator sanity-check sample that contains redacted:true, text, and expect set to a route_id.
  • Extend tests/test_import_review_samples.py with two tests that assert the fixture remains strictly redacted and non-empty and that convert_review_samples succeeds when validating against allowed route ids, and add Path import used by those tests.

Testing

  • Ran the full test suite with uv run python -m pytest -q, which completed successfully with all tests passing (152 passed, 1 warning).
  • Ran the eval smoke command uv run python scripts/eval_routes.py --mock-embeddings, which executed the mock eval cases and reported the expected passes (10 eval case(s) passed).

Codex Task

Copy link
Copy Markdown

@sourcery-ai sourcery-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Disaster-Terminator! 👋

Your private repo does not have access to Sourcery.

Please upgrade to continue using Sourcery ✨

@qodo-code-review
Copy link
Copy Markdown

ⓘ You've reached your Qodo monthly free-tier limit. Reviews pause until next month — upgrade your plan to continue now, or link your paid account if you already have one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant