Update CustomerSatisfactionEvaluator to support multi-turn evaluation

[AliMahmoudzadeh]

Update CustomerSatisfactionEvaluator to support multi-turn evaluation

Summary

Add session-level (messages) input path to CustomerSatisfactionEvaluator, mirroring the pattern from TaskCompletionEvaluator PR #4922.

Changes

_customer_satisfaction.py

• Extended ConversationValidator.validate_eval_input to handle flat messages list input (in addition to existing query/response and conversation paths)
• Added preprocessing helpers: _drop_mcp_approval_messages, _normalize_function_call_types, _preprocess_messages, serialize_messages
• Added _MULTI_TURN_PROMPTY_FILE class attr and loaded _multi_turn_flow in __init__
• Added __call__ overload for messages parameter
• Routed _do_eval to new _do_eval_multi_turn when messages is provided
• Extracted shared _parse_prompty_output method (used by both single-turn and multi-turn)
• Fixed query/response presence check from and to or (correctness bug — old code only raised if both were missing)

customer_satisfaction_multi_turn.prompty (new)

• Multi-turn prompt adapted for 1-5 Likert satisfaction scoring across full conversation sessions
• Evaluates helpfulness, completeness, and tone dimensions

spec.yaml

• Added messages to dataMappingSchema
• Changed required to anyOf: [query, response] | [messages]
• Bumped version to 2

Tests

• Added 11 session-level behavioral tests covering: valid input, empty/invalid messages, system messages, tool calls, intermediate responses, string content, MCP approval, flow routing, and non-dict items
• All 55 tests pass (44 existing + 11 new)

[github-actions]

Test Results for assets-test

64 tests   64 ✅  2s ⏱️
 1 suites   0 💤
 1 files     0 ❌

Results for commit 6824f1e.

♻️ This comment has been updated with latest results.

[imatiach-msft]

🔍 5-Model Parallel Code Review — PR #4925

CustomerSatisfactionEvaluator: Multi-turn support
Reviewed by: Claude Opus 4.6, Claude Sonnet 4.6, Claude Sonnet 4, GPT-5.4, Gemini (Goldeneye)
Consensus scoring: X/5 models agreeing on each finding

---

✅ Positive: No prompty template mismatch (unlike PR #4922)

All 5 models confirmed that this PR does NOT have the critical {{conversation}} binding bug found in PR #4922. The prompty input name (conversation), template variable ({{conversation}}), and Python kwarg ({"conversation": conversation_text}) all match correctly. 👍

The and → or bug fix for query/response presence check is also confirmed correct — both query and response are required inputs, so rejecting when either is missing is the right behavior.

---

🔴 Finding 1: Valid output_text final responses are rejected (5/5 models)

File: _customer_satisfaction.py — final assistant content check
Severity: High

The validator only accepts type == "text" as valid text content in the final assistant message. However, the OpenAI Responses API returns assistant content items with type: "output_text". Conversations captured from agents using the Responses API will be incorrectly rejected as "mid-execution."

The existing evaluator code already treats output_text as valid elsewhere, and _normalize_function_call_types does not normalize output_text → text, so this gap is real.

Recommendation: Accept both "text" and "output_text" in the final assistant text check, or add an output_text normalization step to the preprocessing pipeline.

---

🔴 Finding 2: Preprocessing mutates caller's message objects in-place (5/5 models)

File: _customer_satisfaction.py — _normalize_function_call_types
Severity: High

_normalize_function_call_types() directly modifies item["type"] and pop()s fields like function_call_output / openapi_call_output on the caller's original data. If the same messages list is reused across multiple evaluators in a batch/continuous evaluation pipeline (CustomerSatisfaction → TaskCompletion → etc.), the mutation from one evaluator silently corrupts the input for subsequent evaluators.

Recommendation: Deep-copy the messages at the entry point of _preprocess_messages before performing any in-place mutations.

---

🟡 Finding 3: System/developer messages overwritten — only last one kept (4/5 models)

File: _customer_satisfaction.py — serialize_messages()
Severity: Medium

serialize_messages() stores system/developer context in a single system_message variable and overwrites it on each system/developer turn. Only the last such message reaches the prompty, so earlier instructions/context are silently lost.

Additionally, Gemini flagged a crash risk: if a system/developer message has list-based content blocks (e.g., [{"type": "input_text", "text": "..."}]), msg.get("content", "") stores the raw list, which will raise a TypeError during string concatenation in _pretty_format_conversation_history().

Recommendation: Accumulate all system/developer messages, and normalize list-based content to plain text before formatting.

---

ℹ️ Finding 4: messages path skips per-message schema validation (1/5 models — GPT-5.4)

File: _customer_satisfaction.py — ConversationValidator.validate_eval_input
Severity: Low

The new messages branch only checks roles and the final assistant turn. It does not run the existing per-message schema validation (_validate_message_dict / _validate_input_messages_list), so malformed items like {"role": "user"} (missing content) are accepted. serialize_messages() then silently skips bad messages, potentially producing a score for an incomplete conversation.

Recommendation: Consider reusing existing per-message validators for the messages path.

---

Verdict: APPROVE WITH SUGGESTIONS ✅

The core multi-turn flow is well-designed and avoids the critical prompty mismatch bug from PR #4922. The and → or fix is correct. The behavioral tests are comprehensive.

The output_text rejection (Finding 1) is the most impactful issue — it will cause real failures for Responses API users. The in-place mutation (Finding 2) is a correctness concern for batch pipelines. Both are worth addressing before or shortly after merge.

---

📊 Model Agreement Matrix

Finding	Opus 4.6	Sonnet 4.6	Sonnet 4	GPT-5.4	Gemini	Consensus
output_text rejected	✅	✅	✅	✅	✅	5/5
In-place mutation	✅	✅	✅	✅	✅	5/5
System msg overwrite	—	✅	✅	✅	✅	4/5
Schema validation gap	—	—	—	✅	—	1/5
No prompty mismatch ✅	✅	✅	✅	✅	✅	5/5

Review generated by 5-model parallel review — Claude Opus 4.6 · Claude Sonnet 4.6 · Claude Sonnet 4 · GPT-5.4 · Gemini

[github-actions]

Test Results for scripts-test

123 tests   122 ✅  41m 17s ⏱️
  1 suites    0 💤
  1 files      1 ❌

For more details on these failures, see this check.

Results for commit 8eef3a2.

♻️ This comment has been updated with latest results.