[NVBUG-6327718][test] Unwaive test_disaggregated_videomme[nemotron_na… by aswinvisva · Pull Request #17031 · NVIDIA/TensorRT-LLM

aswinvisva · 2026-07-29T22:47:54Z

…no_v3_omni_fp8] on B200 and H20

The root cause of the pre-fix failure — TypeError: isinstance() arg 2 must be a type, a tuple of types, or a union at tensorrt_llm/executor/proxy.py when the test infrastructure's session-reuse cache monkey-patches MpiPoolSession in the proxy module namespace with a factory function — is already fixed on main. The proxy code was refactored to identify pool-backed sessions by excluding (MpiCommSession, RemoteMpiCommSessionClient) instead of isinstance(x, MpiPoolSession).

Remove the two waivers linked to NVBug 6327718 so CI can verify the fix on both platforms.

Dev Engineer Review

Removed the two nvbug: 6327718 waivers for the targeted test on B200 and H20.
Updated the FP8 test to use the default worker count.
Added repetitions rep20 through rep39 and periodic thread-stack dumps.
Updated the related QA and test-db entries consistently.
Waiver formatting and test-list entries appear correctly scoped.

QA Engineer Review

Modified tests/integration/test_lists/qa/llm_function_core.txt, tests/integration/test_lists/test-db/l0_b200.yml, and tests/integration/test_lists/test-db/l0_h100.yml.
Updated test_disaggregated_videomme to accept the repetition parameter.
Added rep20 through rep39 coverage in the B200 and H100 test lists.
The modified test is covered in test-db/ for CI and qa/ for manual QA.
Verdict: needs follow-up pending CBTS coverage data.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

aswinvisva · 2026-07-29T22:49:04Z

/bot run --stage-list "DGX_B200-PyTorch-4"

coderabbitai · 2026-07-29T22:49:22Z

Walkthrough

The VideoMME E/PD test now runs repetitions 20 through 39 and records periodic thread stacks during execution. B200 and H100 test lists select the repetitions, QA entries use rep20, and related waivers are updated.

Changes

VideoMME repetition coverage

Layer / File(s)	Summary
Parameterize repeated VideoMME execution `tests/integration/defs/accuracy/test_epd_disagg_multimodal.py`	The test accepts repetitions `rep20` through `rep39`, uses the default worker count for Nano Omni FP8, and cancels periodic thread-stack dump timers in a `finally` block.
Update test selections and waivers `tests/integration/test_lists/qa/llm_function_core.txt`, `tests/integration/test_lists/test-db/l0_b200.yml`, `tests/integration/test_lists/test-db/l0_h100.yml`, `tests/integration/test_lists/waives.txt`	Test lists select the new repetitions. B200 and H20 VideoMME waivers are removed, and two other B200 waivers are added.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: qijune, tburt-nv, xinhe-nv

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the NVBug, test change, and affected VideoMME variant.
Description check	✅ Passed	The description clearly explains the prior failure and waiver removal, but it does not provide explicit test coverage details or complete the checklist.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

tensorrt-cicd · 2026-07-29T23:03:35Z

PR_Github #62622 [ run ] triggered by Bot. Commit: 08015b1 Link to invocation

tensorrt-cicd · 2026-07-30T00:50:58Z

PR_Github #62622 [ run ] completed with state SUCCESS. Commit: 08015b1
/LLM/main/L0_MergeRequest_PR pipeline #50762 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

BowenFu · 2026-08-01T07:22:28Z

The two lines removed are full:-scoped, i.e. QA/post-merge lists. The green /bot run --stage-list "DGX_B200-PyTorch-4" here executes l0_b200.yml, which lists only the qwen3vl_2b_instruct and nemotron_nano_v3_omni_nvfp4 variants — not the fp8 one being unwaived — so this PR's CI is zero coverage for the change.

Two other gaps against the description: nvbugs/6327718 is still P0 Dev - Open - To fix, and its recorded signature is Fatal Python error: Aborted in executor/ipc.py / inputs/multimodal_data.py, not the isinstance() TypeError; the repair-bot cannot-reproduce comment is for the nvfp4 variant. And the fix you cite (e15883702b, #16444) merged 2026-07-16, while these two waivers were added on 07-19/20 by #16593 / #16602 — after it.

Could you attach a passing run of the fp8 variant on B200 and H20 (host + commit + pytest command), the way repair-bot did for nvfp4? For what it's worth the fp8 variant already runs unwaived pre-merge via l0_h100.yml:166, so the removal is probably right — it just isn't demonstrated on the two platforms being unwaived.

brnguyen2

This looks like a no-op. [tests/integration/test_lists/waives.txt:16](https://github.com/NVIDIA/TensorRT-LLM/pull/17031/files#diff-621bd2af82a3b97c7a5948368d36c14582ffdf73361deb7f42d2ff395b22167eR16) still waives the same node ID unconditionally (accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_fp8], nvbugs/6478692), so the test stays skipped on every platform and CI can't confirm the fix. Also, the fp8 param only appears in test-db/l0_h100.yml:164 and qa/llm_function_core.txt:86 — l0_b200.yml lists only the qwen3vl/nvfp4 variants — so the B200 entry was already vestigial. Worth confirming the 6327718 signature matches what you believe is fixed, and updating that bug alongside the removal.

aswinvisva · 2026-08-10T16:32:17Z

/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast

tensorrt-cicd · 2026-08-10T16:38:46Z

PR_Github #65073 [ run ] triggered by Bot. Commit: eefdfc4 Link to invocation

tensorrt-cicd · 2026-08-10T18:54:23Z

PR_Github #65073 [ run ] completed with state FAILURE. Commit: eefdfc4
/LLM/main/L0_MergeRequest_PR pipeline #52878 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

aswinvisva · 2026-08-10T19:52:24Z

/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast

tensorrt-cicd · 2026-08-10T19:58:17Z

PR_Github #65111 [ run ] triggered by Bot. Commit: eefdfc4 Link to invocation

tensorrt-cicd · 2026-08-10T21:15:55Z

PR_Github #65111 [ run ] completed with state SUCCESS. Commit: eefdfc4
/LLM/main/L0_MergeRequest_PR pipeline #52909 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

aswinvisva · 2026-08-11T16:21:41Z

/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast

aswinvisva · 2026-08-11T16:45:22Z

/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast

aswinvisva · 2026-08-11T16:47:31Z

/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast

tensorrt-cicd · 2026-08-13T02:14:51Z

PR_Github #65704 [ run ] completed with state SUCCESS. Commit: 78a01ab
/LLM/main/L0_MergeRequest_PR pipeline #53421 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

Prior run (78a01ab, rep360-459) completed 28 videomme reps clean, 0 crashes. Statistically consistent with historical ~3% base rate (P(0 in 28) = 43%). Retry with a fresh rep range to keep sampling. Signed-off-by: Aswin Visva <31215515+aswinvisva@users.noreply.github.com>

aswinvisva · 2026-08-13T15:56:59Z

/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast

tensorrt-cicd · 2026-08-13T16:02:23Z

PR_Github #65959 [ run ] triggered by Bot. Commit: cb5c38e Link to invocation

Prior runs at 100 reps only yielded ~11 videomme reps per shard due to pytest-split with 9 groups distributing across shards, while only 2 shards (DGX_H100-PyTorch-1 and DGX_B200-PyTorch-4) run per /bot run request. 500 reps in the parametrize means ~55 videomme reps per shard = ~110 total per CI run, giving ~96% probability of catching the crash at the historical ~3% failure rate. Signed-off-by: Aswin Visva <31215515+aswinvisva@users.noreply.github.com>

aswinvisva · 2026-08-13T17:32:30Z

/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast

tensorrt-cicd · 2026-08-13T17:39:42Z

PR_Github #65995 [ run ] triggered by Bot. Commit: 1f89f35 Link to invocation

tensorrt-cicd · 2026-08-13T17:44:12Z

PR_Github #65959 [ run ] completed with state ABORTED. Commit: cb5c38e

Link to invocation

tensorrt-cicd · 2026-08-13T22:20:39Z

PR_Github #65995 [ run ] completed with state SUCCESS. Commit: 1f89f35
/LLM/main/L0_MergeRequest_PR pipeline #53681 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

aswinvisva · 2026-08-13T22:59:17Z

/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast

Prior run against commit 1f89f35 (500-rep parametrize) completed 140 videomme reps clean, 0 SIGSEGV/failures. Empty commit forces a fresh CI run so we can re-sample and either confirm crash rate stayed low OR finally reproduce a stack. Signed-off-by: Aswin Visva <31215515+aswinvisva@users.noreply.github.com>

aswinvisva · 2026-08-13T23:02:39Z

/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast

tensorrt-cicd · 2026-08-13T23:07:52Z

PR_Github #66075 [ run ] triggered by Bot. Commit: 0f09e73 Link to invocation

tensorrt-cicd · 2026-08-13T23:09:40Z

PR_Github #66077 [ run ] triggered by Bot. Commit: 0f09e73 Link to invocation

tensorrt-cicd · 2026-08-13T23:15:54Z

PR_Github/17031-0f09e73 #66075 was force-killed by a newer pipeline run.
L0 job information not available (job may not have been triggered yet).

Link to superseding invocation

tensorrt-cicd · 2026-08-14T03:52:01Z

PR_Github #66077 [ run ] completed with state FAILURE. Commit: 0f09e73
/LLM/main/L0_MergeRequest_PR pipeline #53756 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

… capture Layers existing in-tree debug knobs on top of the current faulthandler setup so any rep934-style deadlock produces enough evidence to root-cause without re-running: - sitecustomize.py bootstrap propagates faulthandler + persistent stderr into every MPI worker, so silent worker crashes leave a real file (fixes --capture=fd swallowing). - TRTLLM_WORKER_PRINT_STACKS_PERIOD=60 dumps all Python threads inside each MPI worker every 60s. - TLLM_LLMAPI_ZMQ_DEBUG=1 logs proxy socket send/recv. - TLLM_TRACE_EXECUTOR_LOOP=1 traces per-step executor progress. - TORCH_NCCL_DUMP_ON_TIMEOUT / trace-buffer / debug-info-temp-file dumps NCCL communicator state on collective timeout. - Best-effort py-spy install for native-stack dumps; failure is silent. Signed-off-by: Aswin Visva <31215515+aswinvisva@users.noreply.github.com>

aswinvisva · 2026-08-14T17:02:26Z

/bot run --stage-list "DGX_B200-PyTorch-4" --disable-fail-fast

tensorrt-cicd · 2026-08-14T17:09:43Z

PR_Github #66320 [ run ] triggered by Bot. Commit: 8953e97 Link to invocation

aswinvisva · 2026-08-14T17:13:21Z

/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast

TLLM_LLMAPI_ZMQ_DEBUG and TLLM_TRACE_EXECUTOR_LOOP inflate the Jenkins console 10-100x per rep; across ~500 videomme reps that risks truncating the tail where a crash actually lands. The framework's built-in --periodic-hang-traceback already dumps main-thread stacks on hang, so the remaining unique value of this patch is: - worker-side faulthandler (SIGSEGV/SIGABRT stacks in MPI workers) - persistent stderr (survives --capture=fd on worker death) - TORCH_NCCL_DUMP_ON_TIMEOUT (quiet on success) None of those need the trace env vars. Signed-off-by: Aswin Visva <31215515+aswinvisva@users.noreply.github.com>

aswinvisva · 2026-08-14T17:19:11Z

/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast

tensorrt-cicd · 2026-08-14T17:19:24Z

PR_Github #66323 [ run ] triggered by Bot. Commit: cc96930 Link to invocation

tensorrt-cicd · 2026-08-14T17:24:06Z

PR_Github #66320 [ run ] completed with state ABORTED. Commit: 8953e97

Link to invocation

tensorrt-cicd · 2026-08-14T17:25:34Z

PR_Github #66327 [ run ] triggered by Bot. Commit: cc96930 Link to invocation

tensorrt-cicd · 2026-08-14T17:30:12Z

PR_Github #66323 [ run ] completed with state ABORTED. Commit: cc96930

Link to invocation

tensorrt-cicd · 2026-08-14T17:30:17Z

PR_Github #66327 [ run ] completed with state FAILURE. Commit: cc96930

Link to invocation

aswinvisva · 2026-08-14T19:58:53Z

/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast

tensorrt-cicd · 2026-08-14T20:05:03Z

PR_Github #66364 [ run ] triggered by Bot. Commit: cc96930 Link to invocation

tensorrt-cicd · 2026-08-15T00:46:21Z

PR_Github #66364 [ run ] completed with state FAILURE. Commit: cc96930
/LLM/main/L0_MergeRequest_PR pipeline #54007 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

aswinvisva requested review from a team as code owners July 29, 2026 22:47

aswinvisva requested review from QiJune, ZhanruiSunCh, brnguyen2, crazydemo, tburt-nv and xinhe-nv July 29, 2026 22:47

aswinvisva marked this pull request as draft July 29, 2026 22:48

github-actions Bot assigned aswinvisva Jul 29, 2026

yingguo-trt approved these changes Aug 3, 2026

View reviewed changes

brnguyen2 reviewed Aug 5, 2026

View reviewed changes

aswinvisva force-pushed the avisva/unwaive-nvbug-6327718 branch from 08015b1 to eefdfc4 Compare August 10, 2026 16:25

aswinvisva force-pushed the avisva/unwaive-nvbug-6327718 branch from eefdfc4 to 9264ea2 Compare August 11, 2026 16:18

aswinvisva force-pushed the avisva/unwaive-nvbug-6327718 branch from 9264ea2 to 6dc8867 Compare August 11, 2026 16:47

Uh oh!

Conversation

aswinvisva commented Jul 29, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev Engineer Review

QA Engineer Review

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

aswinvisva commented Jul 29, 2026

Uh oh!

coderabbitai Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 30, 2026

Uh oh!

BowenFu commented Aug 1, 2026

Uh oh!

brnguyen2 left a comment

Choose a reason for hiding this comment

Uh oh!

aswinvisva commented Aug 10, 2026

Uh oh!

tensorrt-cicd commented Aug 10, 2026

Uh oh!

tensorrt-cicd commented Aug 10, 2026

Uh oh!

aswinvisva commented Aug 10, 2026

Uh oh!

tensorrt-cicd commented Aug 10, 2026

Uh oh!

tensorrt-cicd commented Aug 10, 2026

Uh oh!

aswinvisva commented Aug 11, 2026

Uh oh!

aswinvisva commented Aug 11, 2026

Uh oh!

aswinvisva commented Aug 11, 2026

Uh oh!

tensorrt-cicd commented Aug 13, 2026

Uh oh!

aswinvisva commented Aug 13, 2026

Uh oh!

tensorrt-cicd commented Aug 13, 2026

Uh oh!

aswinvisva commented Aug 13, 2026

Uh oh!

tensorrt-cicd commented Aug 13, 2026

Uh oh!

tensorrt-cicd commented Aug 13, 2026

Uh oh!

tensorrt-cicd commented Aug 13, 2026

Uh oh!

aswinvisva commented Aug 13, 2026

Uh oh!

aswinvisva commented Aug 13, 2026

Uh oh!

tensorrt-cicd commented Aug 13, 2026

Uh oh!

tensorrt-cicd commented Aug 13, 2026

Uh oh!

tensorrt-cicd commented Aug 13, 2026

Uh oh!

tensorrt-cicd commented Aug 14, 2026

Uh oh!

aswinvisva commented Aug 14, 2026

Uh oh!

tensorrt-cicd commented Aug 14, 2026

Uh oh!

aswinvisva commented Aug 14, 2026

Uh oh!

aswinvisva commented Aug 14, 2026

Uh oh!

tensorrt-cicd commented Aug 14, 2026

aswinvisva commented Jul 29, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 29, 2026 •

edited

Loading