feat(agent): prefer ReadMediaFile for video analysis in system prompt#2208
feat(agent): prefer ReadMediaFile for video analysis in system prompt#2208bj456736 wants to merge 10 commits into
Conversation
When vis-server binds to 0.0.0.0 or :: (all interfaces), the startup banner and CLI output now display the actual LAN IP addresses that other devices on the same network can use to connect. This enables lan-range remote control from phones, tablets, or other machines. Changes: - Add isAllInterfaces() and getLocalNetworkAddresses() to config.ts - Add lanUrls field to StartedVisServer in start.ts - Update formatStartupBanner() to list LAN URLs when applicable - Update vis CLI handler to print LAN access URLs - Add unit tests for isAllInterfaces and getLocalNetworkAddresses
- Extend UrlFetchResult with new 'image' kind and imageUrl field - LocalFetchURLProvider detects image content-types, reads body as arrayBuffer, converts to base64 data URL via existing image-format-policy helpers - FetchURLTool returns image as ContentPart[] (text + image_url) for LLM consumption - Covers both agent-core and agent-core-v2 - Added tests for image conversion, MIME normalization, and oversized rejection
Updated the system prompt to explicitly guide the AI to use ReadMediaFile directly for video analysis, rather than writing Python scripts to extract frames. This leverages the built-in video input capability of supported models and avoids unnecessary overhead.
🦋 Changeset detectedLatest commit: 4ca9b1c The changes in this PR will be included in the next version bump. Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4ca9b1c319
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| - Make plans before doing deep or wide research, to ensure you are always on track. | ||
| - Search on the Internet if possible, with carefully-designed search queries to improve efficiency and accuracy. | ||
| - Use proper tools or shell commands or Python packages to process or generate images, videos, PDFs, docs, spreadsheets, presentations, or other multimedia files. Detect if there are already such tools in the environment. If you have to install third-party tools/packages, you MUST ensure that they are installed in a virtual/isolated environment. | ||
| - Use proper tools or shell commands or Python packages to process or generate images, videos, PDFs, docs, spreadsheets, presentations, or other multimedia files. For video analysis, prefer `ReadMediaFile` directly over writing Python scripts to extract frames. Detect if there are already such tools in the environment. If you have to install third-party tools/packages, you MUST ensure that they are installed in a virtual/isolated environment. |
There was a problem hiding this comment.
This changes only the agent-core-v2 template, but the normal kimi and kimi -p paths still use the SDK-backed v1 harness (apps/kimi-code/src/cli/prompt-session.ts:4-10), which loads packages/agent-core/src/profile/default/system.md through packages/agent-core/src/profile/default.ts:6-16; that template remains unchanged. Consequently, the new video instruction applies only to experimental v2 and web sessions, while the default CLI—the primary user-visible path described by this changeset—continues preferring the old behavior. Mirror the instruction into the legacy profile template as well.
AGENTS.md reference: AGENTS.md:L17-L17
Useful? React with 👍 / 👎.
| const normalizedMime = normalizeImageMime(contentType); | ||
| if (MODEL_ACCEPTED_IMAGE_MIMES.has(normalizedMime)) { |
There was a problem hiding this comment.
When an origin labels AVIF, BMP, or arbitrary bytes as image/png, this branch trusts the header and rebuilds the payload as a PNG data URL even though the bytes are authoritative. Providers then reject the malformed or unsupported image, and because the tool result is persisted, subsequent requests can keep failing. Use the existing magic-byte detection/resolveEffectiveImageMime policy before accepting and canonicalizing the image; the mirrored legacy provider has the same defect.
Useful? React with 👍 / 👎.
| const buffer = await response.arrayBuffer(); | ||
| if (buffer.byteLength > this.maxBytes) { | ||
| throw new Error( | ||
| `Response body too large: ${String(buffer.byteLength)} bytes exceeds maxBytes (${String(this.maxBytes)}).`, |
There was a problem hiding this comment.
For a valid supported image between the media delivery ceiling and DEFAULT_MAX_BYTES—for example, a 5 MiB PNG—this check permits the original bytes to be embedded unchanged because maxBytes defaults to 10 MiB. The media path elsewhere enforces a 3.75 MiB hard per-image ceiling and normally compresses reads to a much smaller byte and edge budget; bypassing those limits can make the next provider request fail and leave the oversized part in session history. Run fetched images through the same compression and maximum-edge policy before constructing the data URL in both engine implementations.
Useful? React with 👍 / 👎.
| port: info.port, | ||
| host, | ||
| url: `http://${hostForUrl(host)}:${info.port}/`, | ||
| lanUrls: isAllInterfaces(host) ? getLocalNetworkAddresses(info.port) : undefined, |
There was a problem hiding this comment.
When binding to 0.0.0.0 or ::, resolveVisAuthToken requires a token, but lanUrls is generated using only the port. A browser on another device has neither a token parameter nor the server's existing local-storage value, so every /api request receives 401 and the newly advertised URL is not usable as printed. Include the token in a fragment/query understood by apps/vis/web/src/api.ts, or print explicit authentication instructions alongside each LAN URL.
Useful? React with 👍 / 👎.
| if (!iface.internal && iface.family === 'IPv4') { | ||
| addresses.push(`http://${iface.address}:${port}/`); |
There was a problem hiding this comment.
For --host :: on an IPv6-only host or LAN, this filter discards every reachable address, so lanUrls is empty even though the server is listening and the changeset explicitly promises support for ::. Include non-internal IPv6 interfaces and format them with brackets in URL authorities; IPv4 URLs can still be included when dual-stack connectivity is available.
Useful? React with 👍 / 👎.
| const addresses = getLocalNetworkAddresses(3001); | ||
| expect(addresses.length).toBeGreaterThan(0); |
There was a problem hiding this comment.
This test fails on valid IPv6-only, loopback-only, or network-isolated CI environments because os.networkInterfaces() is allowed to contain no non-internal IPv4 address. Mock the network-interface result and assert deterministic filtering/formatting rather than requiring the machine running Vitest to expose a particular adapter.
Useful? React with 👍 / 👎.
1 participant
Summary
Updated the system prompt to explicitly guide the AI to use
ReadMediaFiledirectly for video analysis, rather than writing Python scripts to extract frames.Problem
When users asked Kimi CLI to analyze video files, the AI would often default to writing Python scripts to extract frames (e.g. using OpenCV or ffmpeg), even though the model supports native video input via the
ReadMediaFiletool. This created unnecessary overhead and degraded the user experience.Solution
Added a specific instruction in
system.mdunder the "General Guidelines for Research and Data Processing" section:This leverages the built-in video input capability of supported models and aligns with the existing
ReadMediaFiletool behavior (which already supports video files whenvideo_incapability is present).Testing
Changeset
Patch-level change to
@moonshot-ai/kimi-code— user-visible behavior improvement.