Skip to content

feat(text): support native multimodal message parts#283

Merged
Kamilbenkirane merged 1 commit into
mainfrom
multimodal-text-messages
May 22, 2026
Merged

feat(text): support native multimodal message parts#283
Kamilbenkirane merged 1 commit into
mainfrom
multimodal-text-messages

Conversation

@Kamilbenkirane

@Kamilbenkirane Kamilbenkirane commented May 22, 2026

Copy link
Copy Markdown
Member

Summary

Make text chat messages natively multimodal instead of forcing media through top-level analyze helpers or provider-specific request patches.

This PR introduces a narrow, ordered message-content model:

  • MessageContent = str | list[MessagePart]
  • MessagePart = TextPart | ImagePart | AudioPart | VideoPart | DocumentPart with a type discriminator
  • media values stay as Celeste Artifact objects inside their corresponding part wrappers
  • ToolResult is now a separate message-list item with ToolResultContent, not a Message subclass

The intended app shape is now:

Message(
    role=Role.USER,
    content=[
        TextPart(text="Describe this image"),
        ImagePart(image=ImageArtifact(path="image.png")),
    ],
)

Why

Celeste text clients already expose chat-style generate(messages=...), but message content could not reliably represent native media parts. Mixed raw content like ['text', ImageArtifact(...)] could validate through loose fallbacks, serialize as anonymous JSON, and round-trip without preserving semantic type.

This PR makes Celeste-AI the semantic owner of multimodal chat input:

  • apps and Celeste9 should pass Celeste Message + MessagePart values
  • provider serializers should map those parts to native provider shapes
  • fallback analyze tools can remain app-level behavior for models that do not support native media

Public API changes

  • Adds top-level exports for MessageContent, MessagePart, TextPart, ImagePart, AudioPart, VideoPart, DocumentPart, and ToolResultContent.
  • Removes the broad Content alias from public message typing.
  • Narrows Message.content to plain text or an ordered list of message parts.
  • Keeps media storage in existing ImageArtifact, AudioArtifact, VideoArtifact, and DocumentArtifact values.
  • Keeps ToolResult.content structured through ToolResultContent; structured tool payloads no longer travel as normal Message.content.

Breaking changes

  • Raw artifacts are no longer valid Message.content.
  • Mixed raw lists such as ["text", ImageArtifact(...)] are no longer valid Message.content.
  • Structured Pydantic payloads are no longer valid normal Message.content; use ToolResult.content for structured tool results.
  • Code importing or depending on the removed Content alias must migrate to MessageContent, TextContent, media content aliases, or ToolResultContent depending on intent.
  • Code relying on ToolResult as a Message subclass must treat it as its own message-list item.

Code changes

  • Adds src/celeste/messages.py as the canonical helper layer for:
    • message-part normalization
    • media type collection from messages and top-level media kwargs
    • request-message construction
    • text serialization for text-only fields
    • tool-result object serialization
    • provider part support checks
  • Updates text clients and namespaces so messages accepts list[Message | ToolResult].
  • Updates OpenResponses and Chat Completions text protocol adapters to serialize message parts into protocol-native request payloads.
  • Updates Anthropic, Google, Cohere, and xAI text serializers to use the shared message helpers and fail clearly on unsupported media parts.
  • Updates telemetry content events so multimodal message parts serialize consistently.
  • Updates templates so generated modality/provider code follows the same message/tool-result patterns.

Skill/reference updates

Adds the celeste-python agent skill to the repository with current SDK guidance for future agents. The skill documents:

  • native multimodal MessagePart usage
  • the new src/celeste/messages.py helper seam
  • protocol/base URL support as text-only today
  • root exports versus submodule public types
  • focused verification routing
  • anti-patterns such as app-local message-part duplicates or raw artifacts in Message.content

.agents/ is normally ignored, so these files were intentionally force-added for this PR.

Tests

Adds or updates unit coverage for:

  • discriminated multimodal message content and JSON round-trips
  • rejection of raw artifacts, mixed raw lists, and structured payloads in normal messages
  • provider request building for OpenResponses, Chat Completions, Anthropic, Google, Cohere, and xAI
  • media support validation for media contained inside messages
  • tool-result serialization after separating ToolResult from Message
  • telemetry content events for multimodal messages
  • template coverage for generated tool mapper files

Validation

  • uv run python -m compileall -q src/celeste
  • uv run ruff check on touched SDK and test files
  • uv run pytest tests/unit_tests/test_text_multimodal_message_content.py tests/unit_tests/test_text_multimodal_message_request_building.py tests/unit_tests/test_text_tool_results.py tests/unit_tests/test_tool_outputs.py tests/unit_tests/test_text_media_support_validation.py tests/unit_tests/test_telemetry_content_events.py tests/unit_tests/test_io.py -q (58 passed)
  • uv run mypy src/celeste
  • Commit hook passed: Ruff, Ruff format, mypy, test mypy, Bandit
  • Push hook passed: Ruff, Ruff format, mypy, test mypy, Bandit, coverage tests

Not included

  • No backward-compatible Content alias shim.
  • No app-local or Celeste9-local multimodal message model.
  • No provider-specific fallback path that silently drops unsupported media parts.

@Kamilbenkirane Kamilbenkirane marked this pull request as ready for review May 22, 2026 15:34
@claude

claude Bot commented May 22, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

@Kamilbenkirane Kamilbenkirane merged commit 1972e89 into main May 22, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant