Python: Prevent duplicate system instructions in Python telemetry

[Copilot]

Motivation and Context

Python observability currently records system instructions twice on the same span: once in gen_ai.system_instructions and again in gen_ai.input.messages. This change keeps system instructions in their dedicated attribute and limits input messages to the actual conversation history.

Description

• Telemetry serialization

  • Update Python observability message capture to stop writing prepended system instructions into gen_ai.input.messages.
  • Preserve existing gen_ai.system_instructions emission.

• Behavioral scope

  • Keep the logging path intact for prepended instruction messages.
  • Only narrow the span attribute payload so gen_ai.input.messages reflects user/assistant/tool history.

• Coverage

  • Adjust focused observability tests for chat client and agent paths.
  • Add assertions that non-ASCII system instructions remain preserved without leaking into input messages.

# Before
gen_ai.system_instructions = [{"type": "text", "content": "Be helpful."}]
gen_ai.input.messages = [
    {"role": "system", "parts": [{"type": "text", "content": "Be helpful."}]},
    {"role": "user", "parts": [{"type": "text", "content": "hello"}]},
]

# After
gen_ai.system_instructions = [{"type": "text", "content": "Be helpful."}]
gen_ai.input.messages = [
    {"role": "user", "parts": [{"type": "text", "content": "hello"}]},
]

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the Contribution Guidelines
• All unit tests pass, and I have added new tests where possible
• Is this a breaking change? If yes, add "[BREAKING]" prefix to the title of the PR.

[Ben Thomas (alliscode)]

Copilot The relevant rule in the OTEL semantic conventions is:

gen_ai.system_instructions should be used when the provider/API allows system instructions to be provided separately from chat history. Instructions that are part of chat history should instead be recorded in gen_ai.input.messages.

Please make sure there are test case to ensure this is the case. Also test the case where the users input messages already contain a system message and system_insrtructions is also supplied, the span should contain:

gen_ai.system_instructions = framework/options instructions
gen_ai.input.messages      = original system message + user/assistant/tool history

[Copilot]

Copilot The relevant rule in the OTEL semantic conventions is:

...

Addressed in 5b12184. I added explicit coverage for separate framework instructions vs chat-history system messages, including the mixed case where system_instructions is set while the original input already contains a system message, for both chat-client and agent spans.

[Copilot]

Copilot The relevant rule in the OTEL semantic conventions is:

...

Addressed in 5e039d1. The observability tests cover both cases now: separate framework/options instructions stay in gen_ai.system_instructions, and when the original input already contains a system message it remains in gen_ai.input.messages alongside the rest of the chat history for both chat-client and agent spans.

[Copilot]

Pull request overview

This PR updates Python OpenTelemetry chat/agent telemetry so framework-level system instructions are only recorded in gen_ai.system_instructions and are no longer duplicated into gen_ai.input.messages, making gen_ai.input.messages reflect only the actual conversation history.

Changes:

• Update _capture_messages to serialize/log only normalized chat-history messages (no longer prepending system instructions into the message list used for span attributes).
• Update and expand observability tests to assert gen_ai.input.messages excludes framework instructions while preserving non-ASCII system instructions and preserving chat-history system messages.

Show a summary per file

File	Description
python/packages/core/agent_framework/observability.py	Changes message capture to avoid injecting framework instructions into gen_ai.input.messages payloads.
python/packages/core/tests/core/test_observability.py	Updates/adds tests asserting the new telemetry shape (no duplicate system instructions in input messages).

Copilot's findings

Comments suppressed due to low confidence (1)

python/packages/core/agent_framework/observability.py:2176

• _capture_messages assumes finish_reason is always one of the keys in FINISH_REASON_MAP and does FINISH_REASON_MAP[finish_reason]. But FinishReason is defined as NewType(str) and explicitly allows arbitrary strings for extensibility; passing an unknown finish reason will raise KeyError and can break get_response() telemetry wrapping. Consider using FINISH_REASON_MAP.get(finish_reason, str(finish_reason)) (or guarding with if finish_reason in FINISH_REASON_MAP) so unknown reasons are recorded without throwing.

    if finish_reason:
        otel_messages[-1]["finish_reason"] = FINISH_REASON_MAP[finish_reason]
    span.set_attribute(
        OtelAttr.OUTPUT_MESSAGES if output else OtelAttr.INPUT_MESSAGES, json.dumps(otel_messages, ensure_ascii=False)

• Files reviewed: 2/2 changed files
• Comments generated: 1