[https://nvbugs/6600098][test] Waive flaky TestKVCacheV2Llama scheduler tests on DGX_B200

[brnguyen2]

Description

SKIP-waive four chronically-flaky TestKVCacheV2Llama KV-cache-v2 scheduler
integration tests in the pre-merge L0 DGX_B200 stage. They fail
intermittently independent of the PR under test, so they redden unrelated PRs
and — via fail-fast — cost a full GPU-matrix rerun each.

Flake rates over the last 30 days, measured across all of main CI (not one
PR) via the infra stability report:

Test	flake rate	distinct MRs	distinct failing users
test_overlap_scheduler[overlap]	5.3% (104/1954)	593	13
test_token_budget_limited	5.2% (102/1978)	597	11
test_overlap_scheduler[non_overlap]	4.9% (94/1913)	592	10
test_eviction[cuda_graph]	2.1% (39/1845)	589	8

Failing users span the team; all failures land on DGX_B200 with a mix of
Test terminated unexpectedly / CUDA OOM / assertion — a shared B200-side
instability, not a single bug. Two siblings in the same class are already waived
(test_chunked_prefill → nvbugs/6428002, test_eviction_with_block_reuse →
nvbugs/6462303).

The waived tests keep running post-merge, so regression coverage is retained
— this only takes them off the pre-merge critical path until the underlying
instability is fixed. test_eviction[no_cuda_graph] (1.7%) is left pre-merge for now.

Tracked under umbrella bug nvbugs/6600098 (links the two existing P0s).

Test Coverage

Test-list change only. pre-commit passed the waives.txt duplicate check and the
AST check that every waived node-id exists in the test DB.

PR Checklist

• Commit messages follow [NVBUG][type] description, DCO signed-off.
• Tests remain covered post-merge; only pre-merge gating changes.

Dev Engineer Review

• Added four waiver entries for TestKVCacheV2Llama under nvbugs/6600098.
• Waived:
  • test_overlap_scheduler[overlap]
  • test_token_budget_limited
  • test_overlap_scheduler[non_overlap]
  • test_eviction[cuda_graph]
• The change is limited to tests/integration/test_lists/waives.txt.
• The waiver entries preserve post-merge execution.
• test_eviction[no_cuda_graph] remains in pre-merge testing.
• The waiver format and bug reference are consistent with the file.
• No public API, performance, or error-handling changes were made.
• No duplicate waiver or unintended scope change was reported.

QA Engineer Review

• No test-db/ or qa/ files were modified.
• The change adds four pre-merge waivers and does not modify test code.
• The listed tests remain covered by post-merge execution.
• Verdict: sufficient.

[brnguyen2]

/bot skip --comment "waive only"

[coderabbitai]

Walkthrough

The integration test waiver list adds skip entries for five KV-cache v2 scheduler tests linked to NV bug 6600098.

Changes

KV-cache v2 scheduler waivers

Layer / File(s)	Summary
Add scheduler test waivers tests/integration/test_lists/waives.txt	Adds skip records for CUDA-graph eviction, non-overlap and overlap scheduling, and token-budget-limited execution.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

• NVIDIA/TensorRT-LLM#17444: Updates the same waiver list for different tests and bugs.
• NVIDIA/TensorRT-LLM#17542: Adds waiver entries for different NV bug test failures.

Suggested reviewers: bowenfu, qijune

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description check	✅ Passed	The description explains the four waivers, their cause, scope, retained post-merge coverage, test validation, and checklist status.
Title check	✅ Passed	The title clearly identifies the NVBugs ticket, test-related change, flaky scheduler tests, and DGX_B200 scope.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/integration/test_lists/waives.txt`:
- Line 286: Update the waiver entries for TestKVCacheV2Llama eviction tests in
waives.txt to use stage-aware scoping, or move them into a pre-merge-only waiver
file, so they do not suppress DGX_B200 post-merge execution. Apply the same
correction to all four newly added entries.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 6603b2d4-124b-427c-a162-75f5c0fc729c

📥 Commits

Reviewing files that changed from the base of the PR and between 3612c80 and 3e9df88.

📒 Files selected for processing (1)

• tests/integration/test_lists/waives.txt

[brnguyen2]

/bot run

[tensorrt-cicd]

PR_Github #65669 [ skip ] triggered by Bot. Commit: 3e9df88 Link to invocation

[tensorrt-cicd]

PR_Github #65672 [ run ] triggered by Bot. Commit: 3e9df88 Link to invocation

[tensorrt-cicd]

PR_Github #65669 [ skip ] completed with state ABORTED. Commit: 3e9df88

Link to invocation

[tensorrt-cicd]

PR_Github #65672 [ run ] completed with state FAILURE. Commit: 3e9df88
/LLM/main/L0_MergeRequest_PR pipeline #53392 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[brnguyen2]

/bot run

[tensorrt-cicd]

PR_Github #65682 [ run ] triggered by Bot. Commit: 3e9df88 Link to invocation

[brnguyen2]

/bot skip --comment "waive only"

[tensorrt-cicd]

PR_Github #65690 [ skip ] triggered by Bot. Commit: 3e9df88 Link to invocation

[tensorrt-cicd]

PR_Github #65682 [ run ] completed with state ABORTED. Commit: 3e9df88

Link to invocation

[tensorrt-cicd]

PR_Github #65690 [ skip ] completed with state SUCCESS. Commit: 3e9df88
Skipping testing for commit 3e9df88

Link to invocation