Skip to content

fix(core): accept string items in message content arrays#1035

Merged
christso merged 1 commit intomainfrom
fix/1034-mixed-content-blocks
Apr 10, 2026
Merged

fix(core): accept string items in message content arrays#1035
christso merged 1 commit intomainfrom
fix/1034-mixed-content-blocks

Conversation

@christso
Copy link
Copy Markdown
Collaborator

Closes #1034

Summary

agentv validate accepted message content arrays mixing inline strings with structured blocks like {type: file, value: ...}, but agentv eval run skipped the same test as "incomplete". The validator and the runtime loader disagreed on what a valid TestMessage looked like.

This PR makes the loader, prompt builder, and isTestMessage guard agree with the validator: plain string items inside a content array are treated as inline text segments alongside structured blocks.

Changes

  • packages/core/src/evaluation/types.ts
    • Widen TestMessageContent to string | JsonObject | (string | JsonObject)[].
    • isTestMessage accepts content arrays whose items are strings or JSON objects.
  • packages/core/src/evaluation/input-message-utils.ts
    • extractContentSegments converts string items into {type:'text',value:...} segments (drops empty strings).
  • packages/core/src/evaluation/loaders/message-processor.ts
    • Both processMessages and processExpectedMessages convert string items to text segments. processMessages also surfaces them through textParts for prompt building.
  • Tests
    • shorthand-expansion.test.ts: lock in that mixed string + structured-block content arrays survive expandInputShorthand.
    • message-processor.test.ts: cover the mixed-content shape end to end (resulting segments + textParts).

Red / Green UAT

Reproduction file:

description: red repro with inline string block in content array
tests:
  - id: red-inline-string-block
    criteria: Returns a short answer
    input:
      - role: user
        content:
          - |-
            Use the local file.
          - type: file
            value: /README.md

Red (origin/main):

$ agentv validate red-inline-string-block.eval.yaml
✓ red-inline-string-block.eval.yaml

$ agentv eval run red-inline-string-block.eval.yaml --test-id red-inline-string-block --dry-run
Error: Skipping incomplete test: red-inline-string-block. Missing required fields: id, input, and at least one of criteria/expected_output/assertions
Error: No tests matched the provided filters.

Green (this branch):

$ bun apps/cli/src/cli.ts eval run .tmp/agentv-repro/red-inline-string-block.eval.yaml --test-id red-inline-string-block --dry-run
...
Top performing tests:
  1. red-inline-string-block: 0.000
Results written to: .agentv/results/runs/.../index.jsonl

The generated input.md shows the inline string and the resolved file content interleaved correctly:

@[user]:
Use the local file.

<file path="README.md">
# AgentV
...
</file>

Test plan

🤖 Generated with Claude Code

isTestMessage previously rejected message content arrays whose items were
not all JSON objects, so an eval whose content mixed an inline string and
a {type: file, ...} block validated cleanly but was skipped at runtime as
"incomplete". The validator already accepted this shape.

Make the loader and prompt builder agree with the validator by treating
plain string items inside a content array as inline text segments. The
existing structured-block handling for type: text / file / image is
unchanged.

- Widen TestMessageContent to allow (string | JsonObject)[] arrays.
- Update isTestMessage to accept mixed string/object content arrays.
- Convert string items to {type:text,value:...} segments in
  extractContentSegments, processMessages, and processExpectedMessages.
- Add unit tests covering the mixed-content shape end to end.

Closes #1034

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@cloudflare-workers-and-pages
Copy link
Copy Markdown

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: 7ac3567
Status: ✅  Deploy successful!
Preview URL: https://c5675fe8.agentv.pages.dev
Branch Preview URL: https://fix-1034-mixed-content-block.agentv.pages.dev

View logs

@christso christso marked this pull request as ready for review April 10, 2026 06:47
@christso christso merged commit 99c698f into main Apr 10, 2026
4 checks passed
@christso christso deleted the fix/1034-mixed-content-blocks branch April 10, 2026 07:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

agentv eval run skips valid eval when message content array mixes string and file blocks

1 participant