[https://nvbugs/6428092][fix] Forward use_host_stop_criteria alongside host and py_result_diffs in the PP… by trtllm-agent · Pull Request #16133 · NVIDIA/TensorRT-LLM

trtllm-agent · 2026-07-08T19:12:49Z

Summary

Root cause: In PP mode, non-last ranks defaulted use_host_stop_criteria=False and re-indexed the empty finish_reasons_list produced when the last rank took the fast host stop-criteria path.
Fix: Forward use_host_stop_criteria alongside host and py_result_diffs in the PP ring send/recv, using getattr/hasattr to stay safe against sampler flavors without the field.
Automated fix generated by repair-bot

Test plan

Verify fix on the same GPU type as the original failure
Check for regressions in related tests

Links

Bug: https://nvbugs/6428092

Summary by CodeRabbit

Bug Fixes
- Propagated use_host_stop_criteria through the PP ring SAMPLE_STATE payload.
- Preserved consistent stopping behavior across pipeline stages.
- Maintained compatibility with sampler variants that do not define the flag.
- Removed the waiver for the GPU2 multi-GPU PyTorch test.

Dev Engineer Review

The PP send and receive paths use compatible payload shapes.
getattr and hasattr provide compatibility with sampler variants without use_host_stop_criteria.
The default value is False when the field is absent.
The change is limited to the required pipeline state and waiver entry.
No public declarations or configuration values changed.

QA Engineer Review

No test code changed.
Removed the waiver entry for unittest/llmapi/test_llm_multi_gpu_pytorch.py -m "gpu2" from tests/integration/test_lists/waives.txt.
No test-db/ or qa/ entries were modified.
Verdict: needs follow-up because CBTS coverage data is unavailable.

coderabbitai · 2026-07-08T19:16:00Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b61d7a26-d617-44f4-afa3-a69447e0c263

📥 Commits

Reviewing files that changed from the base of the PR and between 71f025e and 526914b.

📒 Files selected for processing (2)

tensorrt_llm/_torch/pyexecutor/py_executor.py
tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

tests/integration/test_lists/waives.txt

🚧 Files skipped from review as they are similar to previous changes (1)

tensorrt_llm/_torch/pyexecutor/py_executor.py

Walkthrough

The pipeline-parallel sample_state broadcast now sends and receives the use_host_stop_criteria flag. The change also removes a waiver for a multi-GPU PyTorch integration test.

Changes

Sample State Broadcast Update

Layer / File(s)	Summary
Propagate use_host_stop_criteria flag `tensorrt_llm/_torch/pyexecutor/py_executor.py`	The send path includes the flag with a False default. The receive path conditionally assigns it to sample_state.
Remove multi-GPU test waiver `tests/integration/test_lists/waives.txt`	The waiver entry for the GPU2 multi-GPU PyTorch test is removed.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 52691

This change forwards the stop-criteria state needed for pipeline-parallel execution while preserving compatibility with sampler variants; no actionable merge-blocking risk remains after normal checks and review.

Possibly related PRs

NVIDIA/TensorRT-LLM#16132: Updates related pipeline-parallel propagation of use_host_stop_criteria.
NVIDIA/TensorRT-LLM#16134: Changes the same sample-state broadcast and removes a waiver.
NVIDIA/TensorRT-LLM#16136: Matches the same broadcast and waiver updates.

Suggested reviewers: lori-ren, reasonsolo, tabrizian

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the bug fix and matches the required NVBugs and fix format.
Description check	✅ Passed	The description explains the root cause and fix, identifies the bug, and lists relevant test coverage, but omits the repository checklist.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

achartier · 2026-07-10T19:50:10Z

+            # Propagate use_host_stop_criteria: the last rank's fast host-stop
+            # path leaves host.finish_reasons=None, so non-last ranks must
+            # know to skip the finish_reasons indexing in update_requests.
+            if hasattr(sample_state, "use_host_stop_criteria"):


No need for that check

achartier · 2026-07-10T19:50:19Z

                self.send_handles[microbatch_id] = self.dist.isend_object(
-                    (sample_state.host, py_result_diffs),
+                    (sample_state.host, py_result_diffs,
+                     getattr(sample_state, "use_host_stop_criteria", False)),


No need for that check

brnguyen2

use_host_stop_criteria doesn't exist in this codebase. grep -rn use_host_stop_criteria tensorrt_llm tests returns only the five lines this PR adds — SampleStateTorch / SampleStateTensorsHostTorch (sampler/sampler.py:1111-1130) have no such field. So getattr(..., False) always sends False, hasattr(...) is always False, and the receive-side assignment never runs. The change is a no-op that only widens the wire tuple, and the waiver removal in waives.txt is unsupported by it.

Also, finish_reasons_host is only None when requests is empty (sampler.py:4137), and update_requests returns early in that case — so the stated root cause needs re-checking against the actual failure.

brnguyen2 · 2026-08-11T12:13:11Z

+            # Propagate use_host_stop_criteria: the last rank's fast host-stop
+            # path leaves host.finish_reasons=None, so non-last ranks must
+            # know to skip the finish_reasons indexing in update_requests.
+            if hasattr(sample_state, "use_host_stop_criteria"):


hasattr(sample_state, "use_host_stop_criteria") is always False: no sampler state class in the tree defines that attribute, and nothing assigns it. This branch never executes, and the matching getattr(..., False) on line 3115 always sends False. Before revising, confirm the field name against the real sampler state and re-verify the failing test actually reproduces and then passes.

test_tinyllama_logits_processor_2gpu[1-2] (pp_size=2) crashed with IndexError in finish_if_reason because the last PP rank produced a SampleStateTorch with use_host_stop_criteria=True (and host.finish_reasons=None), while only sample_state.host was sent through _ring_broadcast_sample_state. Non-last ranks kept the constructor default False and tried to index into an empty finish_reasons list from process_draft_tokens. Include the flag in the PP send/recv payload and restore it on the receiver. Use getattr/hasattr to stay compatible with sampler flavors that lack this field. Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>

Signed-off-by: handongl <handongl@nvidia.com> Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>

coderabbitai · 2026-08-15T22:28:15Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

trtllm-agent requested a review from a team as a code owner July 8, 2026 19:12

trtllm-agent requested a review from achartier July 8, 2026 19:12

trtllm-agent assigned mingyangHao Jul 8, 2026

github-actions Bot assigned trtllm-agent Jul 8, 2026

trtllm-agent force-pushed the repair-bot-bug6428092 branch from ecfbedf to a6873af Compare July 8, 2026 20:43

achartier reviewed Jul 10, 2026

View reviewed changes

achartier mentioned this pull request Jul 10, 2026

[https://nvbugs/6426886][fix] Forward use_host_stop_criteria alongside sample_state.host and py_result_diffs… #16149

Closed

2 tasks

trtllm-agent force-pushed the repair-bot-bug6428092 branch from a6873af to ab04caa Compare July 10, 2026 21:05

trtllm-agent force-pushed the repair-bot-bug6428092 branch from ab04caa to 7eed8b4 Compare July 24, 2026 07:32

trtllm-agent requested review from a team as code owners July 24, 2026 07:32

trtllm-agent requested review from QiJune, Tabrizian, lori-ren, mikeiovine, reasonsolo, schetlur-nv, yingguo-trt, yiqingy0 and yuanjingx87 July 24, 2026 07:32

yingguo-trt approved these changes Jul 28, 2026

View reviewed changes

brnguyen2 requested changes Aug 11, 2026

View reviewed changes

trtllm-agent added 2 commits August 15, 2026 15:21

[nvbugs/6428092][chore] Remove stale waiver after fix

526914b

Signed-off-by: handongl <handongl@nvidia.com> Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>

trtllm-agent force-pushed the repair-bot-bug6428092 branch from 7eed8b4 to 526914b Compare August 15, 2026 22:28

Navigation Menu

Uh oh!

[https://nvbugs/6428092][fix] Forward use_host_stop_criteria alongside host and py_result_diffs in the PP… - #16133

[https://nvbugs/6428092][fix] Forward use_host_stop_criteria alongside host and py_result_diffs in the PP…#16133
trtllm-agent wants to merge 2 commits into
NVIDIA:mainfrom
tensorrt-cicd:repair-bot-bug6428092

trtllm-agent commented Jul 8, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jul 8, 2026 •

edited

Loading

Uh oh!

achartier Jul 10, 2026

Uh oh!

achartier Jul 10, 2026

Uh oh!

brnguyen2 left a comment

Uh oh!

brnguyen2 Aug 11, 2026

Uh oh!

coderabbitai Bot commented Aug 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

trtllm-agent commented Jul 8, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Links

Summary by CodeRabbit

Dev Engineer Review

QA Engineer Review

Uh oh!

coderabbitai Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

achartier Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

achartier Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

brnguyen2 left a comment

Choose a reason for hiding this comment

Uh oh!

brnguyen2 Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot commented Aug 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

trtllm-agent commented Jul 8, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 8, 2026 •

edited

Loading