[Bug]: DeepSeek streaming tool parsers drop text that precedes a tool call in the same delta

[Yigtwxx]

System Info

• CPU architecture: N/A (reproducible with a pure-Python snippet, no GPU required)
• GPU: N/A
• TensorRT-LLM branch: main
• TensorRT-LLM commit: 3d3d7c9192
• OS: N/A

The report concerns tensorrt_llm/serve/tool_parser/ and affects streaming
/v1/chat/completions responses for deepseek_v3, deepseek_v31, deepseek_v32 and
deepseek_v4.

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

When a streaming increment contains ordinary text followed by a tool call, the text is
dropped. detect_and_parse on the same string returns it as normal_text.

The guard at the top of parse_streaming_increment only emits the buffer when there is no
tool call in it. Once the start token is present the whole buffer, prefix included, goes
into the tool-parsing branch, which returns normal_text="" and then advances the buffer
past the matched call (for example deepseekv32_parser.py:268,
deepseekv31_parser.py:182), so the prefix is discarded rather than emitted.

from tensorrt_llm.serve.openai_protocol import ChatCompletionToolsParam, FunctionDefinition
from tensorrt_llm.serve.tool_parser.tool_parser_factory import ToolParserFactory

tools = [ChatCompletionToolsParam(
    type="function",
    function=FunctionDefinition(
        name="get_weather", description="Get the weather",
        parameters={"type": "object",
                    "properties": {"city": {"type": "string"}},
                    "required": ["city"]}))]

text = ('Normal text <｜DSML｜function_calls> <｜DSML｜invoke name="get_weather"> '
        '<｜DSML｜parameter name="city" string="true">NYC</｜DSML｜parameter> '
        '</｜DSML｜invoke> </｜DSML｜function_calls>')

print(ToolParserFactory.create_tool_parser("deepseek_v32")
      .parse_streaming_increment(text, tools))
print(ToolParserFactory.create_tool_parser("deepseek_v32")
      .detect_and_parse(text, tools))

All four parsers behave the same way on their own tool-call format:

parser	streaming normal_text	streaming calls	non-streaming normal_text	non-streaming calls
deepseek_v3	''	1	'Normal text'	1
deepseek_v31	''	1	'Normal text'	1
deepseek_v32	''	2	'Normal text'	1
deepseek_v4	''	2	'Normal text'	1

The call counts differ because the streaming path emits the name and the arguments as two
separate ToolCallItems, which is the documented streaming contract, so that column is
not itself a defect. The normal_text column is.

Expected behavior

Concatenating the normal_text of every streamed increment reproduces the text that
detect_and_parse returns for the whole response. Text that precedes a tool call is
visible content and must be streamed before the call is emitted.

actual behavior

The text preceding the tool call is dropped. Nothing raises, and the tool call itself is
emitted correctly, so the only symptom is that the streamed response is missing the
sentence that came before the call.

additional notes

Whether this is reachable depends on the model emitting content and the start token inside
one delta. That happens whenever more than one token is decoded per iteration, which is
the case with speculative decoding and with stream_interval above 1, and it is also what
happens when the detokenizer flushes several tokens at once.

This is separate from #17572, which is about text withheld by the partial-token guard and
then discarded. That one never reaches the tool-parsing branch; this one is caused by the
tool-parsing branch itself, and the fix belongs in a different part of the same method. It
came up in review of PR #17573, where I left it out of scope for the same reason.

The four deepseek_* classes in tests/unittest/llmapi/apps/test_tool_parsers.py have no
streaming case that combines ordinary text with a tool call, which is why this is not
covered today. TestGlm47ToolParserSglangSuite has cases of that shape for GLM-4.7 and
could serve as the model for the DeepSeek ones.

I am happy to open a PR for this once #17573 lands, since both touch
parse_streaming_increment in the same three files.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.