Skip to content

[https://nvbugs/6487038][fix] Stop single-rank disagg errors from crashing all gen ranks - #16834

Merged
Shixiaowei02 merged 3 commits into
NVIDIA:mainfrom
Shixiaowei02:user/xiaoweis/fix-disagg-async-transfer-throttle
Aug 13, 2026
Merged

[https://nvbugs/6487038][fix] Stop single-rank disagg errors from crashing all gen ranks#16834
Shixiaowei02 merged 3 commits into
NVIDIA:mainfrom
Shixiaowei02:user/xiaoweis/fix-disagg-async-transfer-throttle

Conversation

@Shixiaowei02

@Shixiaowei02 Shixiaowei02 commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Description

Three separate changes, one per commit so each can be bisected and reverted on its own. They
are in one PR so the disagg CI cost is paid once rather than three times.

1. [fix] Do not crash every ADP rank on a disagg transfer errorpy_executor.py

_pad_attention_dp_dummy_request() asserted expected_num_active_requests >= len(active_requests). Requests whose KV transfer errored can linger in active_requests for a
tick before cleanup drains them, so a single-rank disagg error tripped the assert and took down
the generation loop on every attention-DP rank. The overshoot is transient and
self-correcting, and the padding keys on the schedulable count rather than the raw length, so
it now warns once and continues.

2. [fix] Surface the engine event-loop error from start_threadworker.py

A ManagedThread that already ran cannot be restarted, so start_thread fell through to
thread.start() and raised RuntimeError: threads can only be started once out of submit(),
hiding the engine failure that actually killed the thread. It now reports the stashed
_event_loop_error, wrapped as RequestError(str(err)) from errsubmit() calls start()
every time, and re-raising the same object would append a frame to its __traceback__ on each
call. This also matches how base_worker.py reports its other submit-path failures. The
post-shutdown path (stop() set stop_event, no error stashed) returns quietly.

3. [chore] Tune gb300 disagg perf-sanity transfer configs — three YAMLs

A behavior change on three CI perf cases, so calling it out explicitly:

  • all three migrate the cache transceiver from CPP to PYTHON;
  • the two 8k1k con4096 cases get kv_cache_bounce_size_mb: 2048 and
    TRTLLM_KV_TRANSFER_NUM_THREADS=4;
  • all three now set kv_transfer_timeout_ms: 600000. The 8k1k cases previously relied on the
    60 s default while switching transceiver, which is the exact bound NVBug 6487038 reports being
    exceeded; transfer time varies with the environment, so all three are bounded the same way.

The 128k8k case deliberately gets neither the bounce buffer nor the extra transfer threads, and
says so inline, because measurement says they hurt this shape.

Test Coverage

Unit tests, all CPU-only:

  • tests/unittest/executor/test_event_loop_error_broadcast.py::TestStartThreadAfterExit (new):
    engine error → RequestError with __cause__ intact and no restart; repeated calls → distinct
    exception objects, guarding the __traceback__ accumulation; _event_loop_error is None
    quiet return; fresh thread → still starts.
  • tests/unittest/_torch/executor/test_py_executor.py: the overshoot test now pins the outcome
    (add_dummy_calls == [], active_requests unchanged) instead of only asserting no raise, and
    test_pad_dummy_added_when_overshoot_has_no_schedulable_requests covers the branch that
    matters — overshoot with zero schedulable requests must still add the pad dummy.

A/B for the 128k8k transfer config, GB300, 3 nodes / 12 GPUs, disagg-e2e, one run per arm:

metric without bounce+threads with bounce+threads delta
total token throughput (tok/s) 13690.73 12603.99 −7.9%
output token throughput (tok/s) 805.34 741.41 −7.9%
benchmark duration (s) 7812.20 8485.79 +8.6%
mean TTFT (ms) 2114892.99 2304223.47 +9.0%
mean E2EL (ms) 2182876.21 2373675.33 +8.7%
total runtime (s) 8821 9513 +7.8%
requests 768/768 768/768 both pass

One sample per arm on a shared cluster, so the exact percentages are soft — but every metric
moves the same way, and the slower arm finishes in 9513 s against the case's 180 min budget,
which is why the tuning is not applied there.

Dev Engineer Review

  • Replaced the transient active-request assertion with a one-time warning.
  • Updated worker-thread handling to surface event-loop errors as RequestError and avoid invalid restarts.
  • Updated three GB300 NIXL configurations to use the Python transceiver.
  • Added KV-transfer threads, bounce buffers, and 600000 ms timeouts to the 8k1k configurations.
  • No public API changes were identified.
  • The admission-control bypass removal is not reflected in the listed file changes and requires verification.

QA Engineer Review

  • Added test_pad_dummy_tolerates_active_request_overshoot().
  • Added test_pad_dummy_added_when_overshoot_has_no_schedulable_requests().
  • Added test_surfaces_engine_error_as_request_error().
  • Added test_repeated_calls_do_not_accumulate_traceback().
  • Added test_post_shutdown_exit_returns_quietly().
  • Added test_fresh_thread_is_started().
  • Added test_does_not_start_when_enqueueing_is_disabled().
  • No corresponding test-db/ or qa/ entries were provided for these tests.
  • CI and manual QA coverage are not confirmed.
  • Verdict: needs follow-up.

@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

The change tolerates transient active-request overshoot, handles completed response threads and event-loop errors, adds regression tests, and updates disaggregated NIXL transfer settings.

Changes

Executor safeguards and transfer configuration

Layer / File(s) Summary
Active-request padding tolerance
tensorrt_llm/_torch/pyexecutor/py_executor.py, tests/unittest/_torch/executor/test_py_executor.py
Overshoot logs a warning instead of raising. Tests cover schedulable and non-schedulable request states.
Response-thread lifecycle guards
tensorrt_llm/executor/worker.py, tests/unittest/executor/test_event_loop_error_broadcast.py
start_thread avoids invalid restarts and converts captured engine event-loop errors to RequestError. Tests cover repeated calls, shutdown, fresh threads, and disabled request enqueueing.
Disaggregated transfer configuration
tests/scripts/perf-sanity/disaggregated/*.yaml
NIXL configurations use Python runtimes and set KV-transfer threads, bounce buffers, timeouts, and related comments.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested reviewers: bo-nv, cascade812, chuangz0

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 6.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the NVBug, fix type, and primary disaggregated execution failure addressed by the pull request.
Description check ✅ Passed The description clearly explains the three changes and provides detailed unit-test and performance coverage; only the checklist section is omitted.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@Shixiaowei02
Shixiaowei02 force-pushed the user/xiaoweis/fix-disagg-async-transfer-throttle branch 4 times, most recently from adad095 to d290731 Compare July 27, 2026 05:13
@Shixiaowei02

Copy link
Copy Markdown
Collaborator Author

/bot run --post-merge --stage-list "GB200-16_GPUs-4_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE2-GPU8-Post-Merge, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge, GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #61811 [ run ] triggered by Bot. Commit: d290731 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #61811 Bot args parsing error: CI requested by --post-merge or an approval-gated stage selector (*, *Post-Merge*, or *PerfSanity*) requires the ci: post-merge approved PR label. Ask a member of NVIDIA/trt-llm-ci-approvers to apply it.

Link to invocation

@Shixiaowei02
Shixiaowei02 requested a review from litaotju July 27, 2026 05:49
@Shixiaowei02
Shixiaowei02 force-pushed the user/xiaoweis/fix-disagg-async-transfer-throttle branch from 05f16c5 to 9727efc Compare July 27, 2026 05:58
@Shixiaowei02

Copy link
Copy Markdown
Collaborator Author

/bot run --post-merge --stage-list "GB200-16_GPUs-4_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE2-GPU8-Post-Merge, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge, GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #61826 [ run ] triggered by Bot. Commit: 9727efc Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #61826 Bot args parsing error: CI requested by --post-merge or an approval-gated stage selector (*, *Post-Merge*, or *PerfSanity*) requires the ci: post-merge approved PR label. Ask a member of NVIDIA/trt-llm-ci-approvers to apply it.

Link to invocation

@niukuo niukuo added the ci: post-merge approved Approved by TRT-LLM CI approvers for broad post-merge CI requests label Jul 27, 2026
@github-actions github-actions Bot removed the ci: post-merge approved Approved by TRT-LLM CI approvers for broad post-merge CI requests label Jul 27, 2026
@github-actions

Copy link
Copy Markdown

Removed the "ci: post-merge approved" label because @niukuo could not be verified as an active member of NVIDIA/trt-llm-ci-approvers. Ask a member of that team to apply it.

@litaotju litaotju added the ci: post-merge approved Approved by TRT-LLM CI approvers for broad post-merge CI requests label Jul 27, 2026
@Shixiaowei02
Shixiaowei02 force-pushed the user/xiaoweis/fix-disagg-async-transfer-throttle branch from 9727efc to e14fc1b Compare July 27, 2026 08:37
@Shixiaowei02

Copy link
Copy Markdown
Collaborator Author

/bot run --add-multi-gpu-test --disable-fail-fast --extra-stage "GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-1, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-3, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-4, GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1, GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2, GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-1, GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-2, GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-3"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65189 [ run ] triggered by Bot. Commit: a312365 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65189 [ run ] completed with state FAILURE. Commit: a312365
/LLM/main/L0_MergeRequest_PR pipeline #52978 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@Shixiaowei02

Copy link
Copy Markdown
Collaborator Author

/bot run --add-multi-gpu-test --disable-fail-fast --extra-stage "GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-1, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-3, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-4, GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1, GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2, GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-1, GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-2, GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-3"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65514 [ run ] triggered by Bot. Commit: a312365 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65514 [ run ] completed with state FAILURE. Commit: a312365
/LLM/main/L0_MergeRequest_PR pipeline #53251 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Ask a member of NVIDIA/trt-llm-ci-approvers to add the label, then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

@Shixiaowei02

Copy link
Copy Markdown
Collaborator Author

/bot run --add-multi-gpu-test --disable-fail-fast --extra-stage "GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-1, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-3, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-4, GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1, GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2, GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-1, GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-2, GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-3"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65538 [ run ] triggered by Bot. Commit: a312365 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65538 [ run ] completed with state FAILURE. Commit: a312365
/LLM/main/L0_MergeRequest_PR pipeline #53276 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@Shixiaowei02
Shixiaowei02 enabled auto-merge (squash) August 12, 2026 12:31
@Shixiaowei02
Shixiaowei02 disabled auto-merge August 12, 2026 12:34
@Shixiaowei02

Copy link
Copy Markdown
Collaborator Author

/bot run --add-multi-gpu-test --disable-fail-fast --extra-stage "GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-1, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-3, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-4, GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1, GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2, GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-1, GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-2, GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-3"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65573 [ skip ] triggered by Bot. Commit: a312365 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65574 [ run ] triggered by Bot. Commit: a312365 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65573 [ skip ] completed with state ABORTED. Commit: a312365

Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65574 [ run ] completed with state SUCCESS. Commit: a312365
/LLM/main/L0_MergeRequest_PR pipeline #53307 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@Shixiaowei02

Copy link
Copy Markdown
Collaborator Author

/bot skip --comment "Infra-only failures on GB300 capacity, no regression from this PR."

@Shixiaowei02
Shixiaowei02 enabled auto-merge (squash) August 13, 2026 06:33
@Shixiaowei02
Shixiaowei02 disabled auto-merge August 13, 2026 06:35
@Shixiaowei02

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --extra-stage "GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-1, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-3, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-4, GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1, GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2, GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-1, GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-2, GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-3"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65812 [ skip ] triggered by Bot. Commit: a312365 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65814 [ run ] triggered by Bot. Commit: a312365 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65812 [ skip ] completed with state ABORTED. Commit: a312365

Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65814 [ run ] completed with state FAILURE. Commit: a312365
/LLM/main/L0_MergeRequest_PR pipeline #53516 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@Shixiaowei02

Copy link
Copy Markdown
Collaborator Author

/bot skip --comment "Infra-only failures on GB300 capacity, no regression from this PR."

@Shixiaowei02
Shixiaowei02 enabled auto-merge (squash) August 13, 2026 08:45
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65853 [ skip ] triggered by Bot. Commit: a312365 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65853 [ skip ] completed with state SUCCESS. Commit: a312365
Skipping testing for commit a312365

Link to invocation

@Shixiaowei02
Shixiaowei02 merged commit 925148a into NVIDIA:main Aug 13, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci: full pre-merge approved ci: post-merge approved Approved by TRT-LLM CI approvers for broad post-merge CI requests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants