Skip to content

Python: Add experimental decorator to all Evals pieces - #5040

Merged
Eduard van Valkenburg (eavanvalkenburg) merged 2 commits into
mainfrom
copilot/add-experimental-decorator-to-evals
Apr 2, 2026
Merged

Python: Add experimental decorator to all Evals pieces#5040
Eduard van Valkenburg (eavanvalkenburg) merged 2 commits into
mainfrom
copilot/add-experimental-decorator-to-evals

Conversation

Copilot AI commented Apr 1, 2026

Copy link
Copy Markdown
Contributor

Motivation and Context

All public Evals APIs are experimental and should carry the @experimental decorator to emit ExperimentalWarning on use, update docstrings with the stability notice, and set __feature_stage__ metadata — consistent with the existing pattern used for Skills.

Description

  • _feature_stage.py: Added EVALS = "EVALS" to ExperimentalFeature enum.

  • _evaluation.py: Imported ExperimentalFeature / experimental and applied @experimental(feature_id=ExperimentalFeature.EVALS) to all 19 public symbols:

    • Classes: EvalNotPassedError, ConversationSplitter, ConversationSplit, ExpectedToolCall, EvalItem, EvalScoreResult, EvalItemResult, EvalResults, Evaluator, AgentEvalConverter, CheckResult, LocalEvaluator
    • Functions: keyword_check, tool_called_check, tool_calls_present, tool_call_args_match, evaluator, evaluate_agent, evaluate_workflow
  • _foundry_evals.py: Applied @experimental(feature_id=ExperimentalFeature.EVALS) to FoundryEvals, evaluate_traces, and evaluate_foundry_target.

On first use, callers will see:

ExperimentalWarning: [EVALS] LocalEvaluator is experimental and may change or be removed in future versions without notice.

Contribution Checklist

  • The code builds clean without any errors or warnings
  • The PR follows the Contribution Guidelines
  • All unit tests pass, and I have added new tests where possible
  • Is this a breaking change? No.

Copilot AI changed the title [WIP] Add experimental decorator to all Evals pieces Python: Add experimental decorator to all Evals pieces Apr 1, 2026
@TaoChenOSU Tao Chen (TaoChenOSU) added the python Usage: [Issues, PRs], Target: Python label Apr 1, 2026
@TaoChenOSU
Tao Chen (TaoChenOSU) marked this pull request as ready for review April 1, 2026 21:29
Copilot AI review requested due to automatic review settings April 1, 2026 21:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds feature-stage metadata and runtime ExperimentalWarning behavior across the Python Evals API surface so that all public Evals entry points are consistently marked experimental (matching the existing Skills pattern).

Changes:

  • Added EVALS to the ExperimentalFeature enum for feature-id tagging.
  • Decorated core eval types and orchestration functions in _evaluation.py with @experimental(feature_id=ExperimentalFeature.EVALS).
  • Decorated Foundry eval integration entry points (FoundryEvals, evaluate_traces, evaluate_foundry_target) with the same experimental feature id.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
python/packages/core/agent_framework/_feature_stage.py Adds the EVALS experimental feature identifier used by decorators.
python/packages/core/agent_framework/_evaluation.py Marks all public Evals core APIs as experimental via @experimental(...EVALS).
python/packages/foundry/agent_framework_foundry/_foundry_evals.py Marks Foundry Evals public integration APIs as experimental via @experimental(...EVALS).

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Apr 2, 2026
Merged via the queue into main with commit 7607acd Apr 2, 2026
36 checks passed
Ben Thomas (alliscode) pushed a commit to alliscode/agent-framework that referenced this pull request Apr 3, 2026
* Initial plan

* feat(python): add experimental decorator to all Evals pieces

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/99d71249-a5d6-4977-a5b5-6ffe0a3be2bc

Co-authored-by: TaoChenOSU <12570346+TaoChenOSU@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: TaoChenOSU <12570346+TaoChenOSU@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Usage: [Issues, PRs], Target: Python

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Python: Add experimental decorator to all Evals pieces

5 participants