feat: Default Scorers

[monoxgas]

feat: Default Scorers

Key Changes:

• Added a bunch of basic scorers (primarily text focused)
  • bleu
  • character_consistency
  • contains
  • detect_ansi_escapes
  • detect_pii
  • detect_pii_with_presidio
  • detect_refusal
  • detect_sensitive_keywords
  • detect_unsafe_shell_content
  • length_in_range
  • length_ratio
  • length_target
  • readability
  • semantic_similarity
  • sentiment
  • sentiment_with_perspective
  • similarity
• Refactored Scorer class to remove tracer dependency
• Added catch parameter for exception handling on scorers
• Updated scoring APIs to be simpler and more flexible

Changed:

• Scorer constructor API simplified (removed tracer param)
• from_callable method signature updated
• Scorer.call now includes try/catch logic
• Metric.from_many formatting improved

Removed:

• Tracer dependency from Scorer class
• Span tracing from scorer execution

---

Generated Summary:

• Improved readability by reformatting a return statement in the from_many method to span multiple lines.
• Introduced a new catch parameter in the Scorer constructor, allowing exceptions in the scorer function to be caught, leading to a 0 Metric return with error information.
• Updated several methods to incorporate the new catch parameter, affecting how metrics are returned when exceptions occur.
• Added comprehensive documentation for the catch parameter, clarifying its purpose and usage.
• Created a new file scorers.mdx with extensive documentation for various scorer functions including bleu, character_consistency, and others, enhancing developer understanding and usage of scoring functionalities.

These modifications enhance the overall usability and error handling of the scoring system, making it more robust and easier to document.

This summary was generated with ❤️ by rigging

[Copilot]

Pull Request Overview

This PR introduces a comprehensive default scorer system for the dreadnode project, implementing various text-focused scoring functions for evaluating outputs. The changes simplify the Scorer API by removing tracer dependencies and add a new catch parameter for exception handling in scoring functions.

Key changes:

• Added extensive scoring functions for text evaluation (BLEU, similarity, sentiment, PII detection, etc.)
• Simplified Scorer constructor API by removing tracer dependency and adding catch parameter
• Introduced TaskInput class for dynamic input resolution during scoring

Reviewed Changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
dreadnode/util.py	Added warn_at_user_stacklevel import and enhanced clean_str with max_length parameter
dreadnode/tracing/span.py	Updated type annotations for better type safety
dreadnode/task.py	Added TaskInput class, updated Scorer usage, and improved parameter formatting
dreadnode/scorers/	Created comprehensive scorer module with similarity, sentiment, PII detection, and other scoring functions
dreadnode/metric.py	Removed tracer dependency from Scorer, added catch parameter for exception handling
docs/	Added extensive documentation for new scorer functions and TaskInput class