feat(intelligence): 接上端到端实时模型，让一轮语音真的能听见回应 - #201
Merged
Wintercom merged 5 commits intoAug 12, 2026
Conversation
Audio goes in and speech comes back out, with tool calls in between. The adapter keeps the vendor's event names and its base64 inside itself: what the dialogue layer receives is already decoded and named in its own terms. The seam is declared twice on purpose. Infrastructure must not depend on a product layer, so the adapter states the shape it reports through as its own Observer while the dialogue layer states the same shape as TurnObserver. Nothing imports the other, which also means nothing would notice one side drifting -- so a test assigns across both declarations and mypy rejects it the moment they stop matching. Two behaviours are worth naming. Turn boundaries stay with our own protocol (turn_detection is off), because voice.stream.start and end already draw them and a model deciding as well would fight them. And a turn is one response unless a tool runs: answering the tool asks the model to respond again, and that second response is the one carrying the audio, so the pump counts them instead of stopping at the first response.done.
The transport had a stand-in behind it that reported the same fixed result to every stream. This puts the real model there: audio goes to it, and what comes back reaches the client through three of the outlets the boundary already has -- the user's words, the reply's wording as it forms, and the speech itself. The wording is forwarded as the model produces it rather than held for the audio's opening message. Measured against the real model it finishes about 600 ms before the first audio chunk exists, and voice.tts.start cannot be sent until there is audio to announce, so binding them costs the client that gap. It is settled with a final update after the audio ends, not before, so a client can use it to stop showing that an answer is still coming. Composition fails closed on the stand-in rather than on missing credentials: a deployment that configures the model is a real one and should start, while falling back outside development would leave a server calling commands applied that were never carried out.
… the turn Found by running it: voice.tts.end is what ends a turn, so a client that stops reading there -- the reasonable reading -- never saw the update marking the wording final, and would go on showing an answer as still arriving. Measured on the real model the wording is final around 800 ms before the audio ends, so the turn's end is merely the earliest moment this side can be certain, not the moment it becomes true. The wiring test that covers the stand-in guard was inheriting whatever credentials the developer's .env held, which made it mean one thing in CI and another locally. It now pins them, and a new case covers the path this change opens: a deployment that configures the model builds without injecting a sink.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
2 tasks
![fennoai[bot]](https://avatars.githubusercontent.com/in/1924301?s=60&v=4){kind=small}
There was a problem hiding this comment.
Found three issues that will affect the realtime path:
- The session is not configured to emit input transcription events, so the adapter never reports the user transcript it expects.
- Session setup failures leak the already-open WebSocket.
- Audio-stream/pump failures can leave the pump task running in the background.
python3 -m compileall -q backend/src backend/tests passed. The repository test runner could not be run because neither pytest nor uv is installed in the workspace.
…pull request Only the facts a maintainer would break by not knowing them are kept, one line each: that our protocol owns turn boundaries, that a tool makes a turn two responses, and that the wording has to settle before the audio closes. The last of those is guarded by a test, which is a stronger place for it than prose.
Two ways a turn could end while still holding something open, both found in review of this branch: A session whose `configure()` failed left its WebSocket open. The caller never receives the session, so nobody else can close it. The factory now closes the transport itself before re-raising, and `configure()` moved inside the open timeout -- it had none before. An exception while sending audio left the pump task sitting in `recv()`. Only `CancelledError` was cancelling it, so any other failure -- and the send path is where transport failures show up -- orphaned the task. Cancellation now happens on every exceptional exit. One test each: a transport that refuses to send, and a chunk iterator that raises mid-stream.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
关联 Issue
#197 —— 完成其中 PR 1（接上模型，能听见回应）。
改动
把端到端实时模型接到 #196 的边界后面。这是第一次能真的听见助手说话——在此之前边界后面挂的是
FakeAgent，对每条流回同一条固定结果。intelligence/realtime/ports.pyTurnObserver/RealtimeSession/RealtimeSessionFactoryinfrastructure/external/realtime/qwen_audio.pyintelligence/realtime/agent.pyRealtimeAgent：一轮音频进，转写 / 回复文字 / 语音出infrastructure/settings.py.env.examplealiyun_audio_*配置（与 #189 的aliyun_asr_*平行）main.pyFakeAgenttests/…/test_qwen_audio.pytests/…/test_realtime_agent.py用到边界五个出口里的三个：
deliver_transcript、deliver_reply_text、deliver_audio。新增目录与级联方案（#189 / #195 的
conversation/、external/{asr,llm}/）不重叠。真机实测
说「你好，帮我记一下明天下午三点在203开会」，自
voice.stream.end起算：voice.asr.completedvoice.dialogue.reply（到 1330 ms 共 27 条，逐步累积）voice.tts.start+ 首帧音频voice.tts.end，301,540 字节 / 6.3 秒语音文字比首帧音频早 764 ms —— #196 把回复文字做成独立出口的收益，这里量出来了。
模型说：「好的呀，我帮你记下了：明天下午 3 点在 203 开会。需要我到时候提醒你吗？」
验证
bash backend/scripts/check.sh全绿：229 通过 17 跳过，覆盖率 91.69%，mypy strict 56 文件connect是构造器接缝，测试传假 transporttest_app_wiring.py一处：守卫测试原先继承开发者.env里的密钥，在 CI 和本机含义不同，现已显式钉住本轮不含（见 #197）
Function Call 查日程、
ask_user追问、会话按conversation_id复用、写操作。failed()目前只记日志——模型失败时客户端收不到任何东西，会一直等到超时。这是 #197 里记录在案、本轮按「假设成功」推进的已知缺口，单独一个 PR 补。