Skip to content

[https://nvbugs/6490036][test] Isolate part1 FP8 weight-update tests - #17503

Merged
shuyixiong merged 1 commit into
NVIDIA:mainfrom
shikicloud:isolate-part1-fp8-tests
Aug 12, 2026
Merged

[https://nvbugs/6490036][test] Isolate part1 FP8 weight-update tests#17503
shuyixiong merged 1 commit into
NVIDIA:mainfrom
shikicloud:isolate-part1-fp8-tests

Conversation

@shikicloud

@shikicloud shikicloud commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Dev Engineer Review

  • Updated tests/integration/test_lists/test-db/l0_dgx_b200.yml with explicit Qwen3 8B and Qwen3 30B-A3B FP8 partial weight-update test IDs.
  • Removed the shared part1 waiver from tests/integration/test_lists/waives.txt.
  • The change has limited scope and does not modify product code or test implementation.
  • Test-list formatting and waiver scope require confirmation against the focused diff.

QA Engineer Review

  • Modified tests/integration/test_lists/test-db/l0_dgx_b200.yml.
  • Modified tests/integration/test_lists/waives.txt.
  • Added separate CI entries for the Qwen3 8B and Qwen3 30B-A3B FP8 partial weight-update tests.
  • Removed the shared part1 waiver.
  • Verdict: needs follow-up because CBTS coverage data is unavailable.

Description

PR #16700 waived the Ray orchestrator part0 and part1 FP8 weight-update tests under NVBug 6490036 after intermittent failures in the same CI stage.

PR #16102 restored the part0 coverage by replacing the marker-based entry with its two explicit parametrizations. Both isolated part0 cases passed in pre-merge CI.

This follow-up applies the same isolation to the remaining part1 tests:

Qwen3 8B → Qwen3 8B FP8 partial weight update

Qwen3 30B-A3B → Qwen3 30B-A3B FP8 partial weight update

This PR replaces the shared -m "part1" test-list entry with the two explicit test IDs and removes the corresponding NVBug 6490036 waiver. Listing them separately allows each parametrization to run as an independent CI test-list entry and to be diagnosed or waived independently.

No product code or test implementation is changed. This restores the remaining waived coverage and tests the hypothesis that separate invocations prevent Ray/CUDA IPC state from leaking between parametrized cases.

Test Coverage

Pre-merge CI: CI65453.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

@coderabbitai

coderabbitai Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

The pre-merge configuration now selects explicit multi-GPU FP8 partial weight-update tests for Qwen3 8B and Qwen3 30B A3B. The waiver no longer skips the part1 marker.

Changes

FP8 weight-update test coverage

Layer / File(s) Summary
Explicit multi-GPU test selection
tests/integration/test_lists/test-db/l0_dgx_b200.yml, tests/integration/test_lists/waives.txt
The Ray test list adds explicit Qwen3 FP8 partial weight-update tests. The waiver removes the part1 skip entry.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Possibly related PRs

Suggested reviewers: bowenfu

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the NVBug, test scope, and primary change: isolating the part1 FP8 weight-update tests.
Description check ✅ Passed The description explains the problem, solution, affected tests, rationale, test coverage, and checklist status.
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@shikicloud

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65303 [ run ] triggered by Bot. Commit: ce8ffec Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65303 [ run ] completed with state FAILURE. Commit: ce8ffec
/LLM/main/L0_MergeRequest_PR pipeline #53079 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Ask a member of NVIDIA/trt-llm-ci-approvers to add the label, then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

@shikicloud

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65311 [ run ] triggered by Bot. Commit: ce8ffec Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65311 [ run ] completed with state SUCCESS. Commit: ce8ffec
/LLM/main/L0_MergeRequest_PR pipeline #53085 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@brnguyen2 brnguyen2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked -m "part1" against the source: it marks only test_llm_partial_update_weights_fp8, and its two parametrize IDs match the new entries exactly, so coverage is unchanged. Same pattern as #16102. Fine to merge once pre-merge CI confirms both isolated cases pass.

@shikicloud

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65442 [ run ] triggered by Bot. Commit: ce8ffec Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65442 [ run ] completed with state FAILURE. Commit: ce8ffec

Link to invocation

@shikicloud

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65453 [ run ] triggered by Bot. Commit: ce8ffec Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65453 [ run ] completed with state SUCCESS. Commit: ce8ffec
/LLM/main/L0_MergeRequest_PR pipeline #53202 completed with status: 'SUCCESS'

CI Report

Link to invocation

@shuyixiong
shuyixiong merged commit ae1465e into NVIDIA:main Aug 12, 2026
20 of 21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants