Skip to content

feat(test): add safety VCR cassettes for OpenAI and Anthropic (Phase T2) - #14

Merged
manas-fortifyroot merged 1 commit into
fr-v0.52.6.xfrom
feat/t2-safety-vcr-cassettes
Mar 31, 2026
Merged

feat(test): add safety VCR cassettes for OpenAI and Anthropic (Phase T2)#14
manas-fortifyroot merged 1 commit into
fr-v0.52.6.xfrom
feat/t2-safety-vcr-cassettes

Conversation

@manas-fortifyroot

Copy link
Copy Markdown

Summary

  • Add safety-specific VCR cassette tests for OpenAI and Anthropic instrumentation packages
  • Tests cover all 3 safety rule types: RegEx (PCI.credit_card, PII.email), List (PII.competitor_org), and UDF (custom_compliance with 2 sub-rules)
  • Tests cover both ALLOW and MASK actions, prompt and completion masking, and streaming
  • 9 tests per provider (7 MASK + 2 ALLOW) = 18 total, with 18 cassette YAML files

Files added

File Provider Description
tests/traces/test_safety_cassettes.py OpenAI 9 safety VCR tests
tests/traces/cassettes/test_safety_cassettes/*.yaml OpenAI 9 cassette recordings
tests/test_safety_cassettes.py Anthropic 9 safety VCR tests
tests/cassettes/test_safety_cassettes/*.yaml Anthropic 9 cassette recordings

Test matrix per provider

# Test Rule Type Action Direction Mode
1 regex_email_prompt RegEx MASK Prompt non-stream
2 regex_credit_card_prompt RegEx MASK Prompt non-stream
3 list_competitor_org_prompt List MASK Prompt non-stream
4 udf_project_codenames_prompt UDF (2 sub-rules) MASK Prompt non-stream
5 completion_email_masked RegEx MASK Completion non-stream
6 combined_rules_prompt All 3 MASK Prompt non-stream
7 streaming_with_pii RegEx+UDF MASK Prompt streaming
8 allow_email_not_masked RegEx ALLOW Prompt non-stream
9 allow_credit_card_not_masked RegEx ALLOW Prompt non-stream

Verification results

Mode OpenAI Anthropic
--fr 56 passed, 0 failed 28 passed, 0 failed
--cassettes 184 passed, 0 failed 82 passed, 0 failed
--all 311 passed, 0 failed 101 passed, 0 failed

Test plan

  • CI fr-tests job passes (Python 3.10–3.13)
  • CI non-fr-tests job passes (no regressions on upstream TL tests)
  • Cassettes replay without API keys (--record-mode=none)
  • No leaked API keys in cassette YAML files

Add safety-specific VCR cassette tests for OpenAI and Anthropic instrumentation
packages. Tests cover all 3 safety rule types (RegEx, List, UDF), both ALLOW and
MASK actions, prompt and completion masking, and streaming.

- 9 tests per provider (7 MASK + 2 ALLOW) = 18 total
- 18 cassette YAML files (9 per provider)
@manas-fortifyroot
manas-fortifyroot merged commit 5897166 into fr-v0.52.6.x Mar 31, 2026
9 checks passed
@manas-fortifyroot
manas-fortifyroot deleted the feat/t2-safety-vcr-cassettes branch March 31, 2026 09:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant