feat(agent-core): guide AI to use ReadMediaFile for video analysis instead of manual frame extraction#2183
feat(agent-core): guide AI to use ReadMediaFile for video analysis instead of manual frame extraction#2183bj456736 wants to merge 1 commit into
Conversation
…stead of manual frame extraction - Update system.md to explicitly prefer ReadMediaFile over Python/ffmpeg for video analysis - Update read-media.md tool description to reinforce video analysis guidance Fixes P0 task: Kimi CLI 视频分析希望默认调用 ReadMediaFile 而不是写 Python 切帧
🦋 Changeset detectedLatest commit: aa2d189 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aa2d189e50
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| - If automatic compression cannot safely produce an image within model limits, the tool returns an error and does not send the original image. Follow the error: use Bash or an available image-processing tool to create a smaller copy, then read that copy. Do not retry the unchanged file. | ||
| - The system will notify you when there is anything wrong when reading the file. | ||
| - This tool is a tool that you typically want to use in parallel. Always read multiple files in one response when possible. | ||
| - For video files, this is the preferred way to analyze video content. Do not write Python scripts or ffmpeg commands to extract frames manually — use this tool to read the video directly. |
There was a problem hiding this comment.
For image-only models such as the catalogued OpenAI and Anthropic models, this unconditional directive is included in the tool description because ReadMediaFileTool is registered when either image_in or video_in is enabled. The generated description then contradicts itself by appending “Video files are not supported,” and attempting the instructed call returns an error telling the user to switch models. Prohibiting Python/ffmpeg frame extraction therefore removes the viable fallback for video analysis on these models; add this advice only when video_in is true and retain extraction guidance otherwise.
Useful? React with 👍 / 👎.
commit: |
1 participant
Summary
Adds explicit guidance in both the system prompt and the ReadMediaFile tool description to prefer the
ReadMediaFiletool over writing Python/ffmpeg scripts when analyzing video content. This prevents inefficient manual frame extraction and leverages built-in multimodal capabilities.Changes
packages/agent-core/src/profile/default/system.mdReadMediaFiletool to read it directly rather than writing Python scripts or ffmpeg commands to extract frames manually.packages/agent-core/src/tools/builtin/file/read-media.mdMotivation
Currently, when users upload video files, the AI often defaults to writing Python scripts with OpenCV or ffmpeg commands to extract frames manually. This approach is:
ReadMediaFiletool already provides native multimodal video reading capabilitiesBy explicitly guiding the AI to prefer
ReadMediaFilefor video files at both the system prompt level and the tool description level, we improve user experience and reduce unnecessary code generation.Testing
Related