fix(core): accept string items in message content arrays#1035
Merged
Conversation
isTestMessage previously rejected message content arrays whose items were
not all JSON objects, so an eval whose content mixed an inline string and
a {type: file, ...} block validated cleanly but was skipped at runtime as
"incomplete". The validator already accepted this shape.
Make the loader and prompt builder agree with the validator by treating
plain string items inside a content array as inline text segments. The
existing structured-block handling for type: text / file / image is
unchanged.
- Widen TestMessageContent to allow (string | JsonObject)[] arrays.
- Update isTestMessage to accept mixed string/object content arrays.
- Convert string items to {type:text,value:...} segments in
extractContentSegments, processMessages, and processExpectedMessages.
- Add unit tests covering the mixed-content shape end to end.
Closes #1034
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deploying agentv with
|
| Latest commit: |
7ac3567
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://c5675fe8.agentv.pages.dev |
| Branch Preview URL: | https://fix-1034-mixed-content-block.agentv.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1034
Summary
agentv validateaccepted messagecontentarrays mixing inline strings with structured blocks like{type: file, value: ...}, butagentv eval runskipped the same test as "incomplete". The validator and the runtime loader disagreed on what a validTestMessagelooked like.This PR makes the loader, prompt builder, and
isTestMessageguard agree with the validator: plain string items inside a content array are treated as inline text segments alongside structured blocks.Changes
packages/core/src/evaluation/types.tsTestMessageContenttostring | JsonObject | (string | JsonObject)[].isTestMessageaccepts content arrays whose items are strings or JSON objects.packages/core/src/evaluation/input-message-utils.tsextractContentSegmentsconverts string items into{type:'text',value:...}segments (drops empty strings).packages/core/src/evaluation/loaders/message-processor.tsprocessMessagesandprocessExpectedMessagesconvert string items to text segments.processMessagesalso surfaces them throughtextPartsfor prompt building.shorthand-expansion.test.ts: lock in that mixed string + structured-block content arrays surviveexpandInputShorthand.message-processor.test.ts: cover the mixed-content shape end to end (resulting segments +textParts).Red / Green UAT
Reproduction file:
Red (origin/main):
Green (this branch):
The generated
input.mdshows the inline string and the resolved file content interleaved correctly:Test plan
bun run typecheckbun run test(1976 tests pass)bunx prek run --all-files --hook-stage pre-push(Build / Typecheck / Lint / Test / Validate eval YAML files all green)🤖 Generated with Claude Code