Python: feat(evals): Foundry Adaptive Evals integration (rubric-generation) by alliscode · Pull Request #6101 · microsoft/agent-framework

Ben Thomas (alliscode) · 2026-05-27T00:51:54Z

Motivation and Context

Integrates Foundry Adaptive Evals (rubric-generation) into Agent Framework's Python eval surface, strictly additively on top of the existing FoundryEvals integration (ADR 0023). Adds:

New core typed surfaces — GeneratedEvaluatorRef, RubricDimension, RubricScore, EvalGenerationSource (all @experimental(EVALS)).
FoundryEvals accepts GeneratedEvaluatorRef mixed into the existing evaluators= sequence and emits the correct azure_ai_evaluator testing-criteria.
Per-dimension rubric scores parsed off output_item.results[*].properties.rubric_scores into EvalScoreResult.dimensions.
Assertion helpers on EvalResults — assert_score_at_least, assert_dimension_score_at_least, assert_no_failed_items — for CI gating.
BaseAgent.as_eval_source() / Workflow.as_eval_source() to package the richest source available (instructions, tool defs, context-provider classes, topology) for rubric generation, with conservative privacy defaults.
FoundryEvals.generate_rubric(...) — orchestrates the beta.evaluators.create_generation_job LRO, polls to terminal status, returns a pinned GeneratedEvaluatorRef.
YAML config loader (load_evaluators_from_yaml) + end-to-end sample under python/samples/05-end-to-end/evaluation/foundry_evals/.

The testing-criterion side is unchanged on the wire — we already emit azure_ai_evaluator for builtin.* names, so the new path just supplies a custom evaluator_name + pinned evaluator_version.

The generate_rubric helper gracefully degrades when the installed azure-ai-projects version pre-dates the rubric generation APIs (raises a clear NotImplementedError with install guidance).

Description

6 commits, one per logical phase. Reviewers can step through them in order:

feat(evals): GeneratedEvaluatorRef + RubricDimension/RubricScore types — core only, no Foundry coupling.
feat(foundry-evals): accept GeneratedEvaluatorRef in evaluators= — wires phase 1 into _build_testing_criteria and preserves refs through _filter_tool_evaluators.
feat(evals): parse rubric_scores from output items + assertion helpers — adds EvalScoreResult.dimensions, the three assert_* helpers, and _extract_rubric_scores in _foundry_evals.py.
feat(evals): agent.as_eval_source / workflow.as_eval_source — new EvalGenerationSource core type plus BaseAgent.as_eval_source(...) / Workflow.as_eval_source(...) source-export helpers.
feat(foundry-evals): generate_rubric helper — LRO orchestrator. Imports azure.ai.projects.models.EvaluatorGenerationInputs etc. lazily; raises NotImplementedError with install guidance when unavailable.
feat(foundry-evals): YAML config loader + sample — load_evaluators_from_yaml and evaluate_with_generated_rubric_sample.py.

.NET parity

Not in this PR. Planned as a follow-up 6-commit stack against Microsoft.Agents.AI + Microsoft.Agents.AI.Foundry — unblocked today (the rubric APIs are in Azure.AI.Projects 2.1.0-beta.2, already pinned).

Contribution Checklist

The code builds clean without any errors or warnings
Follows the Coding Guidelines
All unit tests pass (85 core local-eval + 285 foundry; ruff check + format clean; pyright clean on changed files)
I have added unit tests where appropriate
I have updated the documentation accordingly (sample added; ADR follow-up tracked)

Evan Mattson (moonbox3) · 2026-05-27T00:56:47Z

Python Test Coverage Report •

File	Stmts	Miss	Cover	Missing
packages/core/agent_framework
_evaluation.py	722	100	86%	164, 172, 488, 490, 524, 526–534, 536–538, 543, 587, 599, 627, 629–634, 636–638, 642, 791, 794, 873–875, 880, 917–920, 976–977, 980, 986–988, 992, 1025–1027, 1083, 1119, 1131–1133, 1138, 1162–1167, 1260, 1338–1339, 1341–1345, 1351, 1390, 1738, 1740, 1748, 1758, 1762, 1807, 1825–1826, 1904, 1906, 1912, 1920, 1935, 1973, 1979–1983, 2015, 2038, 2046–2047, 2049, 2074–2075, 2080
packages/foundry/agent_framework_foundry
_foundry_evals.py	335	8	97%	471–472, 507–508, 663, 668, 851, 918
TOTAL	37615	4392	88%

Python Unit Test Overview

Tests	Skipped	Failures	Errors	Time
7476	34 💤	0 ❌	0 🔥	1m 44s ⏱️

Copilot

Pull request overview

Adds an additive Python integration for Foundry Adaptive Evals rubric-generation, extending the existing Foundry evals surface with typed rubric references, rubric score parsing, CI-friendly assertions, source export helpers (agent/workflow → generation sources), a YAML config loader, and an end-to-end sample.

Changes:

Introduces new experimental core evaluation types (GeneratedEvaluatorRef, RubricDimension, RubricScore, EvalGenerationSource) plus agent/workflow “export to eval source” helpers.
Extends FoundryEvals to accept generated rubric evaluators, generate new rubrics via Foundry LROs, and parse per-dimension rubric scores into results.
Adds YAML-based evaluator config loading + sample, plus tests covering the new behaviors.

Show a summary per file

File	Description
python/samples/05-end-to-end/evaluation/foundry_evals/evaluators.yaml	Sample YAML config describing a generated rubric evaluator.
python/samples/05-end-to-end/evaluation/foundry_evals/evaluate_with_generated_rubric_sample.py	End-to-end sample that generates a rubric, runs evals, and asserts quality gates.
python/packages/foundry/tests/test_foundry_evals.py	Adds unit tests for generated rubric refs, filtering behavior, rubric score extraction, and rubric-generation orchestration.
python/packages/foundry/tests/test_evals_config.py	New tests for YAML-driven rubric-generation config parsing and source building.
python/packages/foundry/agent_framework_foundry/_foundry_evals.py	Implements generated rubric support in criteria building, rubric score extraction, and `generate_rubric()` LRO polling + conversion to `GeneratedEvaluatorRef`.
python/packages/foundry/agent_framework_foundry/_evals_config.py	New YAML config schema + loader + source builder for rubric generation.
python/packages/foundry/agent_framework_foundry/init.py	Exposes the new evals-config loader/schemas at the foundry package surface.
python/packages/core/tests/core/test_local_eval.py	Adds tests for new rubric assertion helpers and agent/workflow eval-source export helpers.
python/packages/core/agent_framework/_workflows/_workflow.py	Adds `Workflow.as_eval_source()` convenience wrapper.
python/packages/core/agent_framework/_evaluation.py	Adds new rubric-related types, result fields, assertion helpers, and eval-source export helpers.
python/packages/core/agent_framework/_agents.py	Adds `BaseAgent.as_eval_source()` convenience wrapper.
python/packages/core/agent_framework/init.py	Re-exports new evaluation types/helpers at top-level.

Copilot's findings

Files reviewed: 12/12 changed files
Comments generated: 4

github-actions

Automated Code Review

Reviewers: 4 | Confidence: 82% | Result: All clear

Reviewed: Correctness, Security Reliability, Test Coverage, Design Approach

Automated review by alliscode's agents

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…+ accept in evaluators= Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…n helpers Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…elper Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Addresses 4 Copilot review comments on PR microsoft#6101: 1. assert_dimension_score_at_least: drop the (not evaluator or found_any) guard so require_applicable=True correctly raises when the named evaluator produces no entries for the dimension. Adds TestRubricAssertions covering the regression. 2. GeneratedEvaluatorRef docstring: reword to describe actual behaviour (pinning recommended, not required) so it matches the dataclass default and FoundryEvals warning path. 3. _poll_generation_job: switch from asyncio.get_event_loop() to get_running_loop() and bound the per-iteration sleep by remaining time, matching _poll_eval_run. 4. generate_rubric: type category as Literal['quality','safety'] and validate at the entry point with a ValueError; drop the silent 'invalid -> quality' rewrite in _generation_job_to_ref. Adds a regression test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Auto-detect hosted Foundry agents in agent_as_eval_source: when the agent's chat_client exposes a string agent_name (the convention used by RawFoundryAgentChatClient for PromptAgents/HostedAgents), emit a type='agent' EvalGenerationSource so the service fetches instructions and tools from the agent registry instead of relying on the local wrapper (which holds neither for hosted agents). * Add hosted_agent_version kwarg and a new agent_version field on EvalGenerationSource so PromptAgent runs can pin to a specific hosted version for reproducible rubric generation. * Add force_prompt_source escape hatch to bypass auto-detection and always emit a rendered prompt dossier - useful when the local wrapper carries overrides the service-side agent doesnt see. * Fix _to_sdk_source for dataset sources: SDK ctor takes name=/version=, not dataset_name=/dataset_version=. The mismatch would raise TypeError against the real azure-ai-projects 2.3.0a* SDK; only unmocked integration paths were affected. Tests cover: auto-detection happy path, versionless hosted agent, explicit hosted_agent_version forwarding, force_prompt_source override, non-string chat_client attrs (MagicMock test doubles) not mis-detected, agent_version forwarded through _to_sdk_source, and the corrected dataset SDK kwarg names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The published Foundry rubric-evaluator output (Microsoft Learn 'Rubric evaluators' reference) places per-dimension breakdowns under properties.dimension_scores, not properties.rubric_scores. The parser now tries dimension_scores first and falls back to rubric_scores for preview-build compatibility, and tolerates non-list payloads (e.g. MagicMock auto-attrs) by trying the next candidate when parsing yields zero entries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Adds FoundryEvals.create_rubric_evaluator as the agent-framework surface over project_client.beta.evaluators.create_version. This is the manual counterpart to generate_rubric: callers supply RubricDimension instances (authored locally, ported from another framework, or hand-tuned) and we POST a RubricBasedEvaluatorDefinition. The service auto-attaches the non-editable residual dimension (general_quality for quality, general_policy_compliance for safety). Per the Microsoft Learn 'Rubric evaluators' reference, the auto-generation path (create_generation_job) is primarily a portal/UI feature; external SDK clients with rich local agent context are better served by manual create_version. This keeps generate_rubric for users who want to round-trip through a Foundry-registered agent. Validation up front: weight must be in [1,10], ids unique, descriptions non-empty, pass_threshold in [0,1]. The returned GeneratedEvaluatorRef is identical in shape to one obtained from generate_rubric, so downstream evaluators= lists work unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Adds evaluate_with_manual_rubric_sample.py demonstrating the end-to-end dev scenario for FoundryEvals.create_rubric_evaluator: hand-author a list of RubricDimension, register via create_rubric_evaluator, then use the pinned GeneratedEvaluatorRef alongside built-in evaluators in an agent regression run. Also re-exports RubricDimension, GeneratedEvaluatorRef, build_sources, and load_evals_config from agent_framework.foundry (both the lazy runtime shim and the type stub) so the rubric samples can import everything from a single namespace; the auto-generate sample was previously broken because the shim was missing build_sources / load_evals_config. Updates the foundry-evals README with a chooser entry for the two rubric paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Reframes agent-framework as a pure consumer of Foundry rubric evaluators: scoring against rubrics that already exist (authored in the Foundry portal or via the dedicated SDK / REST surface) instead of creating them from the SDK. Removed creation surface area: - FoundryEvals.generate_rubric (auto-generate path) and create_rubric_evaluator (manual path), plus all _GenerationSdkTypes / _ManualRubricSdkTypes / _to_sdk_dimensions / _coalesce_generation_sources / _to_sdk_source / _poll_generation_job / _generation_job_to_ref / _evaluator_version_to_ref / _get_beta_evaluators / _import_*_sdk_types helpers. - EvalGenerationSource (the input source discriminator), RubricDimension (the input dimension type), agent_as_eval_source / workflow_as_eval_source / _detect_hosted_foundry_agent helpers, and the YAML-config loader (_evals_config.py with RubricGenerationSpec / RubricSourceSpec / parse_evals_config / load_evals_config / build_sources). - BaseAgent.as_eval_source / Workflow.as_eval_source plus the _render_agent_dossier / _render_workflow_dossier helpers in core. These existed only to feed the now-removed generation pipeline. - Samples evaluate_with_generated_rubric_sample.py, evaluate_with_manual_rubric_sample.py, and evaluators.yaml. Replaced with a short README section showing how to reference an existing rubric evaluator via GeneratedEvaluatorRef. Kept (consumption surface): - GeneratedEvaluatorRef, slimmed to (name, version, display_name). Still accepted alongside built-in evaluator strings in FoundryEvals(evaluators=[...]). Versionless refs still warn. - RubricScore on EvalScoreResult.dimensions plus EvalResults.assert_dimension_score_at_least for per-dimension CI gates. - _parse_dimension_entries / _extract_rubric_scores output parsing (both canonical dimension_scores and the legacy rubric_scores key). Tests: 160/160 foundry unit tests and 71/71 core local-eval tests pass; pyright is clean across changed files. The pre-existing tests/core/test_telemetry.py::test_detect_hosted_fallback_import_error failure is unrelated and reproduces on the prior commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Adds a runnable end-to-end sample showing how to consume a pre-existing rubric evaluator created in Foundry: reference it with GeneratedEvaluatorRef(name, version), mix it with built-in evaluators in FoundryEvals, and gate CI with assert_dimension_score_at_least on a specific dimension. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mypy infers OutputItemListResponse.sample as dict[str, object] | None while pyright correctly infers the typed Sample model. Cast to Any so both type checkers accept the attribute access pattern, rename the local to avoid shadowing the inner-loop sample binding, and drop the now-stale pyright suppressions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ft.com link The Adaptive Evals authoring docs are not yet published on Microsoft Learn, so the link 404s. Keep the descriptive text without the broken hyperlink; we can re-add it once the docs ship. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions

Automated Code Review

Reviewers: 4 | Confidence: 93% | Result: All clear

Reviewed: Correctness, Security Reliability, Test Coverage, Design Approach

Automated review by alliscode's agents

Per code review feedback (eavanvalkenburg): the test file repeated 'from agent_framework_foundry._foundry_evals import ...' inside 22 test bodies and 'from agent_framework_foundry import GeneratedEvaluatorRef' inside 8 more. Move all of them to the existing top-level imports; the symbols are the same across tests and the local imports were redundant. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Adds the core rubric-evaluator surface that mirrors the Python work in PR microsoft#6101 (commit e45b934). Provider-agnostic types only — no Foundry coupling. Subsequent commits will wire these into FoundryEvals. - RubricScore: per-dimension score record (Id, Score?, Applicable, Weight, Reason). - EvalScoreResult.Dimensions: optional init-only list of RubricScore. Null for non-rubric (built-in) evaluators. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Adds the provider-agnostic surface for referencing a pre-existing rubric evaluator and gating CI on per-item / per-dimension thresholds. Mirrors Python PR microsoft#6101 commits e5830dd (ref type) and 4bc6046 (asserts). - GeneratedEvaluatorRef: name + optional version/display-name, plus a Latest(name) factory for versionless refs (discouraged for CI; consumers should warn at run time). - AgentEvaluationResults.AssertScoreAtLeast: walks DetailedItems[].Scores, optionally filtered by evaluator name, recurses into SubResults. - AgentEvaluationResults.AssertDimensionScoreAtLeast: walks each score's Dimensions list, skips non-applicable dimensions by default, supports requireApplicable to flip that, recurses into SubResults. - AgentEvaluationResults.AssertNoFailedItems: walks DetailedItems for fail/error statuses, recurses into SubResults. All helpers throw InvalidOperationException (matches existing AssertAllPassed). Truncates offender lists to the first 5 with a '+N more' suffix to keep CI output readable, mirroring the Python helpers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Adds a follow-up section to ADR 0023 documenting the adaptive rubric evaluator consumption capability added in this stack: - Scope (consumption only; authoring deferred symmetrically with Python). - Core types: RubricScore, GeneratedEvaluatorRef + Latest factory, EvalScoreResult.Dimensions, AgentEvaluationResults.Assert{Score,DimensionScore,NoFailedItems}AtLeast. - Foundry wiring: FoundryEvaluatorSpec union, azure_ai_evaluator wire format with evaluator_version, dimension_scores parsing with legacy/fallback shapes. - Endpoint footgun call-out. - Sample pointer. Cross-references Python PR microsoft#6101. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Adds the core rubric-evaluator surface that mirrors the Python work in PR microsoft#6101 (commit e45b934). Provider-agnostic types only — no Foundry coupling. Subsequent commits will wire these into FoundryEvals. - RubricScore: per-dimension score record (Id, Score?, Applicable, Weight, Reason). - EvalScoreResult.Dimensions: optional init-only list of RubricScore. Null for non-rubric (built-in) evaluators. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Adds the provider-agnostic surface for referencing a pre-existing rubric evaluator and gating CI on per-item / per-dimension thresholds. Mirrors Python PR microsoft#6101 commits e5830dd (ref type) and 4bc6046 (asserts). - GeneratedEvaluatorRef: name + optional version/display-name, plus a Latest(name) factory for versionless refs (discouraged for CI; consumers should warn at run time). - AgentEvaluationResults.AssertScoreAtLeast: walks DetailedItems[].Scores, optionally filtered by evaluator name, recurses into SubResults. - AgentEvaluationResults.AssertDimensionScoreAtLeast: walks each score's Dimensions list, skips non-applicable dimensions by default, supports requireApplicable to flip that, recurses into SubResults. - AgentEvaluationResults.AssertNoFailedItems: walks DetailedItems for fail/error statuses, recurses into SubResults. All helpers throw InvalidOperationException (matches existing AssertAllPassed). Truncates offender lists to the first 5 with a '+N more' suffix to keep CI output readable, mirroring the Python helpers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* .NET: feat(evals): RubricScore type + EvalScoreResult.Dimensions Adds the core rubric-evaluator surface that mirrors the Python work in PR microsoft#6101 (commit e45b934). Provider-agnostic types only — no Foundry coupling. Subsequent commits will wire these into FoundryEvals. - RubricScore: per-dimension score record (Id, Score?, Applicable, Weight, Reason). - EvalScoreResult.Dimensions: optional init-only list of RubricScore. Null for non-rubric (built-in) evaluators. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * .NET: feat(evals): GeneratedEvaluatorRef + assertion helpers Adds the provider-agnostic surface for referencing a pre-existing rubric evaluator and gating CI on per-item / per-dimension thresholds. Mirrors Python PR microsoft#6101 commits e5830dd (ref type) and 4bc6046 (asserts). - GeneratedEvaluatorRef: name + optional version/display-name, plus a Latest(name) factory for versionless refs (discouraged for CI; consumers should warn at run time). - AgentEvaluationResults.AssertScoreAtLeast: walks DetailedItems[].Scores, optionally filtered by evaluator name, recurses into SubResults. - AgentEvaluationResults.AssertDimensionScoreAtLeast: walks each score's Dimensions list, skips non-applicable dimensions by default, supports requireApplicable to flip that, recurses into SubResults. - AgentEvaluationResults.AssertNoFailedItems: walks DetailedItems for fail/error statuses, recurses into SubResults. All helpers throw InvalidOperationException (matches existing AssertAllPassed). Truncates offender lists to the first 5 with a '+N more' suffix to keep CI output readable, mirroring the Python helpers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * .NET: feat(foundry-evals): accept GeneratedEvaluatorRef in evaluators= Adds FoundryEvaluatorSpec, a readonly-struct union with implicit conversions from both string and GeneratedEvaluatorRef so call sites can mix built-in evaluator names with rubric evaluator references: var evals = new FoundryEvals( projectClient, model, new GeneratedEvaluatorRef("policy-rubric", "3"), FoundryEvals.Relevance, FoundryEvals.Coherence); FoundryEvals constructors (3 overloads), EvaluateTracesAsync, and EvaluateFoundryTargetAsync now take FoundryEvaluatorSpec[]/params instead of string[]/params. Existing call sites using string literals or string[] keep working unchanged via implicit conversion. FoundryEvalConverter.BuildTestingCriteria emits the documented Foundry wire format for rubric refs: { "type": "azure_ai_evaluator", "name": <DisplayName ?? Name>, "evaluator_name": <Name>, "evaluator_version": <Version>, // omitted when null "initialization_parameters": { "deployment_name": <model> }, "data_mapping": { conversation arrays, optional tool_definitions } } WireTestingCriterion gains an optional EvaluatorVersion field. Rubric refs are preserved through FilterToolEvaluators (tool-aware but not tool-required) and ignored by FindMissingGroundTruthEvaluators. A versionless ref emits a Trace.TraceWarning at criterion-build time so CI authors notice the floating version (mirrors the Python warning). Adds 6 new Foundry unit tests (3 BuildTestingCriteria rubric paths, 1 FindMissingGroundTruthEvaluators, 1 FilterToolEvaluators preservation, 1 mixed-order). 369/369 Foundry tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * .NET: feat(foundry-evals): parse rubric dimension_scores into RubricScore Adds FoundryEvals.ParseRubricScores, called per result inside ParseDetailedItem. Each EvalScoreResult now populates Dimensions when the evaluator's sample carries a rubric breakdown. Accepts three shapes for forward compatibility with provider SDK iterations: 1. sample.properties.dimension_scores (canonical Foundry runtime shape) 2. sample.properties.rubric_scores (preview/legacy key) 3. top-level sample.dimension_scores / sample.rubric_scores (defensive fallback) Entries missing 'id', 'weight', or 'applicable' are skipped without invalidating well-formed siblings. Non-applicable dimensions may omit 'score' (parsed as null). Adds 6 unit tests covering canonical and legacy keys, top-level fallback, no-match returns null, malformed-entry skipping, and the non-applicable null-score path. 375/375 Foundry tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * .NET: feat(samples): Evaluation_FoundryRubric end-to-end sample Adds dotnet/samples/05-end-to-end/Evaluation/Evaluation_FoundryRubric mirroring the Python evaluate_with_rubric_sample.py: - Fetches a pre-existing Foundry agent via AgentAdministrationClient (GetAgentAsync for latest, GetAgentVersionAsync when FOUNDRY_AGENT_VERSION is pinned). - References a rubric evaluator by GeneratedEvaluatorRef(name, version); falls back to GeneratedEvaluatorRef.Latest(name) with the documented floating-version warning. - Mixes the rubric with FoundryEvals.Relevance and FoundryEvals.Coherence in a single FoundryEvals run (implicit string-and-ref conversion). - Prints per-dimension breakdowns from EvalScoreResult.Dimensions for each item. - Demonstrates a CI quality gate with AssertDimensionScoreAtLeast("general_quality", 3.0). Documents the FOUNDRY_PROJECT_ENDPOINT footgun (must be project-scoped URL .../api/projects/<project>, not the bare Azure OpenAI endpoint) and the Eval-Definition-vs-Rubric-Evaluator distinction in the README. Ships a .env.example with the FOUNDRY_* variables. Registers the project in agent-framework-dotnet.slnx and cross-links from the sibling Evaluation_Multimodal / Evaluation_ExpectedOutputs READMEs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(foundry-evals): harden FoundryEvals public surface for review Address PR microsoft#6267 review comments on the .NET FoundryEvals integration: - Add source-compat overloads accepting `string[] evaluators` for `FoundryEvals` ctor, `EvaluateTracesAsync`, and `EvaluateFoundryTargetAsync` so existing callers passing string arrays keep compiling unchanged. New overloads forward via a private `ToSpecs` helper that wraps each name through the implicit `string -> FoundryEvaluatorSpec` conversion. - Guard against `default(FoundryEvaluatorSpec)` entries (both `BuiltinName` and `GeneratedRef` null) that would NRE the downstream converter. Adds `FoundryEvaluatorSpec.IsValid` / `EnsureValid` plus an internal `EnsureAllSpecsValid` helper, wired into the main ctor and both static evaluation entry points. - Add 6 unit tests covering the new validation surface. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(sample): set ExitCode=1 when rubric dimension gate trips PR microsoft#6267 review comment: the FoundryRubric sample swallowed the AssertDimensionScoreAtLeast failure, so a CI run that included it as a quality gate would still exit 0 even when the rubric regressed. Set `System.Environment.ExitCode = 1` in the catch so CI fails while still letting the rest of the sample's logging complete cleanly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(foundry-evals): search typed Sample directly for rubric scores PR microsoft#6267 review comment: `_extract_rubric_scores` only searched the `properties` dict when the sample exposed one. When the Azure AI Projects typed SDK returns a Sample object that puts `dimension_scores` / `rubric_scores` directly on the instance (no `properties` wrapper), we missed them and surfaced no per-dimension scores. Add an `else: containers.append(sample)` branch so non-dict typed samples are also inspected for the score keys. Covered by two new tests: one with `dimension_scores` directly on a typed Sample without a `properties` wrapper, and one with the legacy `rubric_scores` key in the same shape. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(evals): cover assert_score_at_least and assert_no_failed_items PR microsoft#6267 review comments: both assertion helpers shipped without unit tests. Add `TestAssertScoreAtLeast` (above threshold, below w/ offenders, evaluator filter, sub_results recursion) and `TestAssertNoFailedItems` (all passing, failed/errored statuses, sub_results recursion) with a shared `_score_results` fixture builder. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(samples): remove dead rubric-evaluator doc link from FoundryRubric sample The Azure AI Foundry rubric evaluator concept doc page has not yet been published, so the link in the sample README and Program.cs comment 404s. Drop the references until the upstream doc is live. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Address PR 6267 review nits Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Ben Thomas <25218250+alliscode@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings May 27, 2026 00:51

Copilot started reviewing on behalf of Ben Thomas (alliscode) May 27, 2026 00:52 View session

Evan Mattson (moonbox3) added the python Usage: [Issues, PRs], Target: Python label May 27, 2026

Copilot AI reviewed May 27, 2026

View reviewed changes

github-actions Bot reviewed May 27, 2026

View reviewed changes

Ben Thomas (alliscode) and others added 6 commits May 27, 2026 08:06

Python: feat(evals): RubricScore type + EvalScoreResult.dimensions

e45b934

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Python: feat(foundry-evals): RubricDimension + GeneratedEvaluatorRef …

e5830dd

…+ accept in evaluators= Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Python: feat(evals): parse rubric_scores from output items + assertio…

4bc6046

…n helpers Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Python: feat(evals): BaseAgent.as_eval_source / Workflow.as_eval_source

38d51d1

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Python: feat(foundry-evals): EvalGenerationSource + generate_rubric h…

a9e4676

…elper Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Python: feat(foundry-evals): YAML config loader + sample

4c7f94f

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Ben Thomas (alliscode) force-pushed the adaptive-evals branch from 08f3b46 to 4c7f94f Compare May 27, 2026 15:23

Ben Thomas (alliscode) and others added 5 commits May 27, 2026 08:48

Evan Mattson (moonbox3) added the documentation Usage: [Issues, PRs], Target: documentation in the code base and learn docs label May 28, 2026

Ben Thomas (alliscode) force-pushed the adaptive-evals branch 2 times, most recently from 9aa152b to 702e8c1 Compare May 28, 2026 17:57

Ben Thomas (alliscode) force-pushed the adaptive-evals branch from 702e8c1 to 907c909 Compare May 28, 2026 18:31

Ben Thomas (alliscode) and others added 2 commits May 29, 2026 08:34

Ben Thomas (alliscode) marked this pull request as ready for review May 29, 2026 16:42

github-actions Bot reviewed May 29, 2026

View reviewed changes

Eduard van Valkenburg (eavanvalkenburg) approved these changes Jun 1, 2026

View reviewed changes

Comment thread python/packages/foundry/tests/test_foundry_evals.py Outdated

Evan Mattson (moonbox3) approved these changes Jun 1, 2026

View reviewed changes

Ben Thomas (alliscode) added this pull request to the merge queue Jun 1, 2026

Merged via the queue into microsoft:main with commit e0d0ad1 Jun 1, 2026
37 checks passed

Navigation Menu

Uh oh!

Python: feat(evals): Foundry Adaptive Evals integration (rubric-generation) - #6101

Python: feat(evals): Foundry Adaptive Evals integration (rubric-generation)#6101
Ben Thomas (alliscode) merged 16 commits into
microsoft:mainfrom
alliscode:adaptive-evals

Ben Thomas (alliscode) commented May 27, 2026

Uh oh!

Evan Mattson (moonbox3) commented May 27, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot left a comment

Uh oh!

github-actions Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Ben Thomas (alliscode) commented May 27, 2026

Motivation and Context

Description

.NET parity

Contribution Checklist

Uh oh!

Evan Mattson (moonbox3) commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Python Unit Test Overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Automated Code Review

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Automated Code Review

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Evan Mattson (moonbox3) commented May 27, 2026 •

edited

Loading