[https://nvbugs/6590664][fix] Reap idle single-rank CTX transfers

[chienchunhung]

Summary

• Reap completed context-side transfers when a single-rank CTX Python executor is idle in synchronous-transfer mode.
• Release the KV blocks pinned by completed sender sessions so CTX can continue admitting requests.
• Remove the waiver for the exact GB200 GPT-OSS disaggregated gen_only perf-sanity test.

Root cause

The affected test uses CTX TP1 and GEN TP4 with TRTLLM_DISABLE_KV_CACHE_TRANSFER_OVERLAP=1.

In synchronous-transfer mode, _check_disagg_transfer_progress_when_idle() returned at the multi-rank safety guard before polling check_context_transfer_status(). GEN-side KV receives could complete, but the corresponding CTX sender sessions were not reaped. Consequently, AsyncTransferManager.end_transfer() was not reached, completed transfers kept their KV blocks pinned, and CTX eventually stopped admitting requests with its KV pool at 99.9% utilization.

The apparent memory pressure and stalled GEN queue were downstream symptoms of missing CTX transfer cleanup—not a device OOM or a GEN model-forward failure.

Fix

When all of the following are true, poll CTX transfer status with timeout 0 before returning:

• the executor is idle;
• synchronous GEN transfer is configured;
• the executor has one rank; and
• at least one transfer is in flight.

This is the minimal liveness fix needed by the affected CTX TP1 topology. It is extracted from the broader work in #16402 without including that PR's queue-wakeup, timeout, cancellation, request-broadcast, C++, or client-lifecycle changes.

Safety boundary

The existing guard for multi-rank synchronous transfer remains intact.

A synchronous GEN receive is blocking and rank-local: one model-parallel rank may still be in request_and_receive_sync() while another reaches the idle-progress path. Context-status progress may enter model-parallel collectives, so polling it from only the idle rank can deadlock.

Asynchronous GEN transfer already uses the rank-coordinated idle-progress path added by #17107. The remaining generalization—multi-rank CTX with synchronous GEN transfer—requires explicitly coordinated polling such as the broader queue/broadcast machinery in #16402. It is intentionally out of scope because this test uses CTX TP1.

Targeted regression

Test

perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_gpt-oss-120b-fp4_8k1k_con1024_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL]

CI stage

GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2

The test remains registered in tests/integration/test_lists/test-db/l0_gb200_multi_nodes_perf_sanity_ctx1_node1_gpu1_gen1_node1_gpu4.yml; this PR removes only its waiver from waives.txt.

Validation

• Added a focused unit regression proving that idle, single-rank CTX reaps synchronous transfers without entering distributed collectives.
• Preserved the existing multi-rank regression proving synchronous GEN does not enter idle-progress collectives.
• Repository pre-commit checks passed on all modified files.
• Targeted CI passed on rebased head 53bbe3cecf6e9f10574327865f0ab88ab2d5fe44:
  • PR_Github #66052 and L0_MergeRequest_PR #53735 completed successfully.
  • The CI report recorded 1 passed, 0 failed, and 0 skipped tests.
  • OpenSearch recorded the exact GPT-OSS selector above as PASSED—not waived or skipped—in 410.013 seconds on GB200 Slurm job 6131689.
  • This directly validates that the NVBUG 6590664 reproducer completes successfully with the fix. Because the case has historically been intermittent, additional full-CI results provide further confidence against recurrence.

Related bugs: NVBUG 6590664, parent NVBUG 6581075.

[chienchunhung]

/bot run --disable-reuse-test --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2"

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 74ea5e36-fc97-40f5-925a-2a65d3391e42

📥 Commits

Reviewing files that changed from the base of the PR and between ac6e8ef and 53bbe3c.

📒 Files selected for processing (3)

• tensorrt_llm/_torch/pyexecutor/py_executor.py
• tests/integration/test_lists/waives.txt
• tests/unittest/_torch/executor/test_py_executor.py

🚧 Files skipped from review as they are similar to previous changes (3)

• tests/integration/test_lists/waives.txt
• tensorrt_llm/_torch/pyexecutor/py_executor.py
• tests/unittest/_torch/executor/test_py_executor.py

---

Walkthrough

The change adds nonblocking context-transfer reaping during single-rank synchronous idle progress, adds regression coverage, and removes the related GB200 GPT-OSS 120B performance-test waiver.

Changes

Disaggregated transfer execution

Layer / File(s)	Summary
Reap idle context transfers tensorrt_llm/_torch/pyexecutor/py_executor.py, tests/unittest/_torch/executor/test_py_executor.py	The single-rank idle path checks and reaps completed context transfers without blocking. The regression test verifies context-transfer polling and excludes distributed collectives and generation-transfer polling.
Remove obsolete performance waiver tests/integration/test_lists/waives.txt	The skip entry for the GB200 GPT-OSS 120B disaggregated generation performance test was removed.

Estimated code review effort: 2 (Simple) | ~10 minutes

Mergeability Score: ⚪ Minimal · up to 53bbe

The PR narrowly restores transfer cleanup for the targeted single-rank configuration and removes its waiver; no actionable merge-blocking risk remains beyond normal checks and review.

Possibly related PRs

• NVIDIA/TensorRT-LLM#17535: Related disaggregated KV-cache transfer and PyExecutor transfer-progress changes.
• NVIDIA/TensorRT-LLM#17609: Related GB200 disaggregated performance-sanity coverage for synchronous context-transfer behavior.

Suggested reviewers: tongyuantongyu, pcastonguay, nv-xtf

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title follows the required NVBugs and type format and clearly describes the main fix.
Description check	✅ Passed	The description explains the root cause, fix, safety boundary, regression test, validation, and related bugs in sufficient detail.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[tensorrt-cicd]

PR_Github #65631 [ run ] triggered by Bot. Commit: a785da6 Link to invocation

[tensorrt-cicd]

PR_Github #65631 [ run ] completed with state FAILURE. Commit: a785da6
/LLM/main/L0_MergeRequest_PR pipeline #53353 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-reuse-test --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2"

[tensorrt-cicd]

PR_Github #65652 [ run ] triggered by Bot. Commit: a785da6 Link to invocation

[tensorrt-cicd]

PR_Github #65652 [ run ] completed with state FAILURE. Commit: a785da6
/LLM/main/L0_MergeRequest_PR pipeline #53378 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-reuse-test --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2"

[tensorrt-cicd]

PR_Github #65737 [ run ] triggered by Bot. Commit: a785da6 Link to invocation

[tensorrt-cicd]

PR_Github #65737 [ run ] completed with state FAILURE. Commit: a785da6
/LLM/main/L0_MergeRequest_PR pipeline #53453 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-reuse-test --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2"

[chienchunhung]

/bot run --disable-reuse-test --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2"

[tensorrt-cicd]

PR_Github #66052 [ run ] triggered by Bot. Commit: 53bbe3c Link to invocation

[tensorrt-cicd]

PR_Github #66052 [ run ] completed with state SUCCESS. Commit: 53bbe3c
/LLM/main/L0_MergeRequest_PR pipeline #53735 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

[chienchunhung]

/bot run --disable-reuse-test

[tensorrt-cicd]

PR_Github #66312 [ run ] triggered by Bot. Commit: 53bbe3c Link to invocation

[tensorrt-cicd]

PR_Github #66312 [ run ] completed with state SUCCESS. Commit: 53bbe3c
/LLM/main/L0_MergeRequest_PR pipeline #53967 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #66365 [ run ] triggered by Bot. Commit: 53bbe3c Link to invocation

[tensorrt-cicd]

PR_Github #66365 [ run ] completed with state FAILURE. Commit: 53bbe3c
/LLM/main/L0_MergeRequest_PR pipeline #54008 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation