Skip to content

[https://nvbugs/6535779][fix] Remove stale Qwen3.5 and DeepSeekV32 waivers - #17649

Open
VALLIS-NERIA wants to merge 2 commits into
NVIDIA:mainfrom
VALLIS-NERIA:agent/unwaive-nvbug6535779
Open

[https://nvbugs/6535779][fix] Remove stale Qwen3.5 and DeepSeekV32 waivers#17649
VALLIS-NERIA wants to merge 2 commits into
NVIDIA:mainfrom
VALLIS-NERIA:agent/unwaive-nvbug6535779

Conversation

@VALLIS-NERIA

@VALLIS-NERIA VALLIS-NERIA commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

What

Why

NVBugs 6535779 and 6483369 are fixed, so retaining these entries hides valid integration coverage. The NVBug 6483369 waiver removal is moved from #17438; its MoE unit-test changes are intentionally discarded. This fresh change supersedes #17611 because that branch now conflicts with the current main waiver list.

Validation

  • Commit-time pre-commit suite for the moved waiver deletion: all hooks passed, including duplicate-waiver and 2,073-entry AST validation.
  • Exact B300 DeepSeekV32 case from [https://nvbugs/6483369][fix] Keep the engineer-confirmed one-line stale-waiver removal, and add a… #17438: MMLU 87.99, GSM8K 96.25; 1 passed.
  • python3 -m pre_commit run --files tests/integration/test_lists/waives.txt
  • Full B200 x4 E2E:
    • TestQwen3_5_397B_A17B::test_nvfp4_mtp3_gdn_replay_tep4
    • TP4/EP4, MTP3, cached GDN replay, V2 cache manager, full GSM8K (1,319 samples)
    • accuracy: 95.53 average (91.797 threshold)
    • 1 passed in 809.08 s; executor evaluation time 74.219 s
    • logs confirmed MambaHybridCacheManagerV2, KVCacheV2Scheduler, cached replay, and replay state updates enabled

Replay manager performance sanity check

Qwen3.5-35B-A3B-FP8 on one B200, MTP3 cached replay, fixed 128-request synthetic workload (ISL 2048 / OSL 512, concurrency 64), two reversed-order A/B pairs:

  • V1 mean: 11.345 req/s, 5,808.7 output tok/s
  • V2 mean: 12.697 req/s, 6,501.1 output tok/s
  • V2: +11.9% throughput, -10.7% total latency, -8.7% average request latency
  • Both pairwise comparisons favored V2 (+10.5% and +13.4%)

This is a directional executor-level comparison; GPU clocks were not locked, and the second pair used an otherwise idle GPU on a shared host.

Dev Engineer Review

  • Removed four stale waiver entries from tests/integration/test_lists/waives.txt.
  • Removed three Qwen3.5-397B-A17B NVFP4 waivers for fixed NVBug 6535779.
  • Removed one B300 DeepSeekV32 chunked-prefill latency waiver for fixed NVBug 6483369.
  • Restored CI coverage for all four cases.
  • Added no new waivers.
  • Made no code, API, or performance changes.
  • Preserved unrelated MoE unit-test changes outside this PR scope.
  • CI failed and requires follow-up before merge.

QA Engineer Review

  • Modified only tests/integration/test_lists/waives.txt.
  • Modified no test-db/ or qa/ files.
  • Removed three Qwen3.5-397B-A17B waiver entries and one B300 DeepSeekV32 waiver entry.
  • Added no test-list entries.
  • CBTS coverage data is unavailable.
  • Verdict: needs follow-up.

Copy link
Copy Markdown
Collaborator Author

/bot run --extra-stage "DGX_B200-4_GPUs-PyTorch-Post-Merge-1, DGX_B200-4_GPUs-PyTorch-Post-Merge-2, DGX_B200-4_GPUs-PyTorch-Post-Merge-3, DGX_B200-4_GPUs-PyTorch-Post-Merge-4"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #66118 [ run ] triggered by Bot. Commit: d3358f0 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #66118 [ run ] completed with state FAILURE. Commit: d3358f0
/LLM/main/L0_MergeRequest_PR pipeline #53793 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@VALLIS-NERIA
VALLIS-NERIA marked this pull request as ready for review August 14, 2026 14:55
@coderabbitai

coderabbitai Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7e26695d-fb70-488d-90ae-565dfc15d461

📥 Commits

Reviewing files that changed from the base of the PR and between 5972811 and 3784d5a.

📒 Files selected for processing (1)
  • tests/integration/test_lists/waives.txt
🚧 Files skipped from review as they are similar to previous changes (1)
  • tests/integration/test_lists/waives.txt

Walkthrough

The integration test waive list removes three Qwen3.5 397B NVFP4 skip entries and one B300 DeepSeekV32 chunked-prefill skip entry.

Changes

Integration test waive-list cleanup

Layer / File(s) Summary
Remove obsolete waive entries
tests/integration/test_lists/waives.txt
The list removes three Qwen3.5 397B entries and one B300 DeepSeekV32 multi-GPU chunked-prefill latency entry.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: ⚪ Minimal · up to 3784d

This change restores coverage by removing four stale test waivers, and no actionable merge-blocking risk remains after normal checks and review.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title uses the required NVBugs URL and [fix] format and accurately identifies removal of stale Qwen3.5 and DeepSeekV32 waivers.
Description check ✅ Passed The description clearly covers the issue, solution, scope, and validation with relevant test results.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@VALLIS-NERIA VALLIS-NERIA changed the title [https://nvbugs/6535779][fix] Remove stale Qwen3.5 waivers [https://nvbugs/6535779][fix] Remove stale Qwen3.5 and DeepSeekV32 waivers Aug 14, 2026
Signed-off-by: Xiwen Yu <13230610+VALLIS-NERIA@users.noreply.github.com>
Signed-off-by: Xiwen Yu <13230610+VALLIS-NERIA@users.noreply.github.com>
@VALLIS-NERIA
VALLIS-NERIA force-pushed the agent/unwaive-nvbug6535779 branch from 5972811 to 3784d5a Compare August 14, 2026 15:20
@VALLIS-NERIA
VALLIS-NERIA requested review from brnguyen2 and trtllm-agent and removed request for brnguyen2 August 15, 2026 12:49
@VALLIS-NERIA

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #66465 [ run ] triggered by Bot. Commit: 3784d5a Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #66465 [ run ] completed with state FAILURE. Commit: 3784d5a
/LLM/main/L0_MergeRequest_PR pipeline #54107 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Ask a member of NVIDIA/trt-llm-ci-approvers to add the label, then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants