Skip to content

[None][test] Waive 3 DGX_B200 main-side CI flakes (kv-cache eviction, qwen3 fp8 lora, gemma4 e2e) - #17663

Merged
brnguyen2 merged 1 commit into
NVIDIA:mainfrom
brnguyen2:user/brnguyen/waive-b200-flakes
Aug 13, 2026
Merged

[None][test] Waive 3 DGX_B200 main-side CI flakes (kv-cache eviction, qwen3 fp8 lora, gemma4 e2e)#17663
brnguyen2 merged 1 commit into
NVIDIA:mainfrom
brnguyen2:user/brnguyen/waive-b200-flakes

Conversation

@brnguyen2

@brnguyen2 brnguyen2 commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Description

Three unit/integration tests flake on main CI independent of any change under test — they fail intermittently across hundreds of unrelated MRs and many distinct users, all with flake-typical signatures (hang/timeout, terminate-unexpectedly, assertion). Under fail-fast each flaky red gates the whole pipeline, so they block unrelated PRs. Waiving them in waives.txt, each with a filed tracking bug.

Test Flake (14d on main) Signature Bug
kv_cache/test_kv_cache_v2_scheduler.py::TestKVCacheV2Llama::test_chunked_prefill_eviction_block_reuse 1.9% (17/874), 331 MRs, 10 users terminate / assert / CUDA OOM https://nvbugs/6607481
unittest/_torch/modules/tests_lora_modules/test_qwen3_sanity.py::TestQwen3LoRA::test_qwen3_fp8_lora 0.5% (11/2376), 334 MRs, 6 users hang → 2400s timeout https://nvbugs/6607487
unittest/_torch/modeling/test_gemma4_e2e_dummy.py::test_e2e_text_31b_dummy 0.8% (13/1711), 331 MRs, 4 users terminate unexpectedly https://nvbugs/6607482

Evidence from the whole-of-main stability report (trtllm-infra), 14-day window. These surfaced on the CI of #17414; that PR cherry-picks these same three waives to unblock and will rebase them away once this lands on main.

Test Coverage

Test-list-only change (tests/integration/test_lists/waives.txt). No code paths changed; the three tests are skipped in CI until their tracking bugs are resolved.

PR Checklist

  • Please check this after reviewing the above items as appropriate for this PR.

Dev Engineer Review

  • Added three DGX_B200 integration tests to tests/integration/test_lists/waives.txt.
  • Each entry includes its tracking NVBugs ticket:
    • KV-cache eviction block reuse — nvbugs/6607481
    • Qwen3 FP8 LoRA sanity — nvbugs/6607487
    • Gemma4 31B dummy E2E — nvbugs/6607482
  • The change is limited to the waiver list.
  • No code paths, APIs, or exported entities changed.
  • Verify the waiver entries against the repository format and confirm that no duplicate test or bug entries exist.

QA Engineer Review

  • Only tests/integration/test_lists/waives.txt changed.
  • Three waiver entries were added.
  • No test-db/ or qa/ files were modified.
  • Verdict: needs follow-up until CBTS coverage data is available.

These three tests flake on main CI independent of any change, failing
across hundreds of unrelated MRs and many distinct users, with
flake-typical signatures (hang/timeout, terminate-unexpectedly,
assertion). Under fail-fast each flaky red gates the whole pipeline.
Waive with tracking bugs:

- kv_cache/test_kv_cache_v2_scheduler.py::TestKVCacheV2Llama::test_chunked_prefill_eviction_block_reuse (nvbugs/6607481)
- unittest/_torch/modules/tests_lora_modules/test_qwen3_sanity.py::TestQwen3LoRA::test_qwen3_fp8_lora (nvbugs/6607487)
- unittest/_torch/modeling/test_gemma4_e2e_dummy.py::test_e2e_text_31b_dummy (nvbugs/6607482)

Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>
@coderabbitai

coderabbitai Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 10046974-fa9b-48c0-b1c7-4d9b08b630fe

📥 Commits

Reviewing files that changed from the base of the PR and between 7b1bb1a and 30898d6.

📒 Files selected for processing (1)
  • tests/integration/test_lists/waives.txt

Walkthrough

The waiver list adds three skipped integration test cases. Each entry references an NVBugs ticket.

Changes

Integration test waivers

Layer / File(s) Summary
Add NVBugs-linked waiver entries
tests/integration/test_lists/waives.txt
Adds waivers for the KV-cache chunked-prefill eviction, Gemma 4 31B dummy end-to-end text, and Qwen3 FP8 LoRA sanity tests.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Mergeability Score: ⚪ Minimal · up to 30898

This change only waives three documented flaky CI tests and does not alter product code or runtime behavior. No actionable merge-blocking risk remains after normal review and checks.

Possibly related PRs

Suggested reviewers: schetlur-nv

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the test waiver and names the three affected DGX_B200 CI flakes.
Description check ✅ Passed The description explains the issue, solution, affected tests, tracking bugs, test coverage, and checklist status.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@brnguyen2

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #66072 [ run ] triggered by Bot. Commit: 30898d6 Link to invocation

@brnguyen2

Copy link
Copy Markdown
Collaborator Author

/bot skip --comment "waive only PR"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #66082 [ skip ] triggered by Bot. Commit: 30898d6 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #66072 [ run ] completed with state ABORTED. Commit: 30898d6

Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #66082 [ skip ] completed with state SUCCESS. Commit: 30898d6
Skipping testing for commit 30898d6

Link to invocation

@brnguyen2
brnguyen2 merged commit 9ce3062 into NVIDIA:main Aug 13, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants