feat(agent-core): guide AI to use ReadMediaFile for video analysis instead of manual frame extraction#1395
Conversation
Closes MoonshotAI#1016 The LLM sometimes passes 'completed' as the status for TodoList items, but the schema only accepted 'pending' | 'in_progress' | 'done'. This produced two problems: 1. Validation failed when the model used 'completed'. 2. Even if validation passed, statusMarker() had no case for 'completed' and fell through to the unreachable default branch. Changes: - Extend TodoStatus union to include 'completed' so it is accepted at the type level. - Map 'completed' -> 'done' in setTodos() so persisted state stays clean. - Handle 'completed' in statusMarker() so it renders as '[done]'. - Update the markdown description to explicitly warn against using 'completed'. - Add a test confirming 'completed' is accepted and mapped to 'done'.
…stead of manual frame extraction Adds explicit guidance in system prompt to prefer ReadMediaFile tool over writing Python/ffmpeg scripts when analyzing video content. This prevents inefficient manual frame extraction and leverages built-in multimodal capabilities. - Modified system.md General Guidelines for Research and Data Processing - Target task: Kimi CLI 视频分析希望默认调用 ReadMediaFile 而不是写 Python 切帧
|
commit: |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 239dcec38c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| title: z.string().min(1).describe('Short, actionable title for the todo.'), | ||
| status: z.enum(['pending', 'in_progress', 'done']).describe('Current status of the todo.'), | ||
| status: z | ||
| .preprocess((val) => (val === 'completed' ? 'done' : val), z.enum(['pending', 'in_progress', 'done'])) |
There was a problem hiding this comment.
When the model calls TodoList with status: "completed" through the normal loop, this preprocess never runs: runToolCallBatch first validates parsed arguments with AJV against tool.parameters, which is generated from toInputJsonSchema(TodoListInputSchema). For z.preprocess(..., z.enum(['pending', 'in_progress', 'done'])), the advertised JSON schema still only permits the inner enum values, so completed is rejected before resolveExecution() reaches the new normalization in setTodos(). This means the new compatibility path only works in direct unit calls, not in production tool calls; include completed in the JSON schema or normalize before AJV validation.
Useful? React with 👍 / 👎.
1 participant
Problem
When users upload video files for analysis, the AI was writing Python scripts or ffmpeg commands to extract frames manually, instead of using the built-in ReadMediaFile tool. This is inefficient and does not leverage the multimodal capabilities of the model.
Solution
Add explicit guidance in the system prompt to prefer ReadMediaFile tool for video files rather than writing Python scripts or ffmpeg commands to extract frames manually.
Changes
Testing
Fixes: Kimi CLI 视频 analysis 希望默认调用 ReadMediaFile 而不是写 Python 切帧