Skip to content

[https://nvbugs/6606123][test] Waive AutoDeploy shim SWA-eviction + mamba-cache unit tests (setup OOM) - #17653

Merged
brnguyen2 merged 1 commit into
NVIDIA:mainfrom
brnguyen2:user/brnguyen/waive-autodeploy-shim-oom
Aug 13, 2026
Merged

[https://nvbugs/6606123][test] Waive AutoDeploy shim SWA-eviction + mamba-cache unit tests (setup OOM)#17653
brnguyen2 merged 1 commit into
NVIDIA:mainfrom
brnguyen2:user/brnguyen/waive-autodeploy-shim-oom

Conversation

@brnguyen2

@brnguyen2 brnguyen2 commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Description

The test_unittests_v2[unittest/auto_deploy/singlegpu/shim] wrapper flakes on H100_PCIe-AutoDeploy-1 and A30-AutoDeploy-1 — ~2.8% over the last 14 days (44/1547 runs), across 321 distinct PRs and 25 distinct users, so it is main-side, not any one PR.

The failures are not spread across the directory (114 tests / 5 files). They concentrate in a small set of memory-heavy tests that OOM on setup (cudaMalloc ... out of memory in kv_cache_manager::WindowBlockManager; the mamba test hits torch CUDA out of memory) while allocating their KV / window cache — intermittent because it depends on how much GPU memory earlier tests in the shard leave free.

This waives only those tests, so the other ~109 shim tests keep their coverage (rather than a directory-level waive):

  • unittest/auto_deploy/singlegpu/shim/test_ad_executor_swa_eviction.py (4 of its 9 tests fail; whole file waived)
  • unittest/auto_deploy/singlegpu/shim/test_cached_sequence_interface.py::test_initialize_resources_mixed_creates_mamba_hybrid_cache_manager

Tracked in nvbugs/6606123 (assigned to the AutoDeploy owner).

Test Coverage

Test-list-only change (no source/build change). Local pre-commit passed, including Validate test list entries exist in source files (AST) and the waives.txt duplicate/format checks.

PR Checklist

  • PR title follows the required format
  • PR description explains the change
  • Commits are signed off (DCO)

Dev Engineer Review

  • Added two AutoDeploy singlegpu shim test waivers in tests/integration/test_lists/waives.txt.
  • Waived test_ad_executor_swa_eviction.py and test_initialize_resources_mixed_creates_mamba_hybrid_cache_manager.
  • Associated both entries with nvbugs/6606123.
  • The change modifies only test-list entries. No API, performance, or production-code changes are included.
  • The waiver scope is limited to the reported intermittent GPU out-of-memory failures.

QA Engineer Review

  • Modified tests/integration/test_lists/waives.txt.
  • Added two waiver entries. No test-db or QA entries were added or removed.
  • The waived tests remain tracked by nvbugs/6606123.
  • Verdict: needs follow-up. CBTS coverage data is unavailable.

…amba-cache unit tests (setup OOM)

The `test_unittests_v2[unittest/auto_deploy/singlegpu/shim]` wrapper flakes
on H100_PCIe-AutoDeploy-1 and A30-AutoDeploy-1 (~2.8% over 14 days, 321 MRs,
25 users). The failures are concentrated in a small set of memory-heavy tests
that OOM on setup (cudaMalloc, kv_cache_manager::WindowBlockManager) while
allocating their KV/window cache — not spread across the directory. Waive only
those tests so the other ~109 shim tests keep running, rather than the whole
directory. Tracked in nvbugs/6606123.

Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>

@allisonlim-nv allisonlim-nv left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@brnguyen2

Copy link
Copy Markdown
Collaborator Author

/bot skip --comment "waives-only PR"

@brnguyen2
brnguyen2 marked this pull request as ready for review August 13, 2026 17:48
@brnguyen2
brnguyen2 enabled auto-merge (squash) August 13, 2026 17:49
@coderabbitai

coderabbitai Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 4a492892-1c38-4d05-981f-adc165e1e630

📥 Commits

Reviewing files that changed from the base of the PR and between aa70f57 and 3b1961b.

📒 Files selected for processing (1)
  • tests/integration/test_lists/waives.txt

Walkthrough

The integration waiver list adds two NVBug-linked skip entries for auto-deploy singlegpu shim tests.

Changes

Auto-deploy test waivers

Layer / File(s) Summary
Add NVBug-linked test waivers
tests/integration/test_lists/waives.txt
Adds skip entries for the SWA eviction executor and mixed-cache initialization tests under NVBug 6606123.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Mergeability Score: ⚪ Minimal · up to 3b196

This PR only waives two identified setup-OOMing tests while leaving the rest of the shim coverage unchanged; no production or build behavior changes are introduced, and no actionable merge-blocking risk remains beyond normal checks and review.

Possibly related PRs

Suggested reviewers: bowenfu

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title follows the required NVBugs and type format and clearly identifies the waived AutoDeploy shim tests and setup OOM issue.
Description check ✅ Passed The description explains the failure pattern, affected tests, waiver scope, tracking issue, and test coverage.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #66004 [ skip ] triggered by Bot. Commit: 3b1961b Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #66004 [ skip ] completed with state SUCCESS. Commit: 3b1961b
Skipping testing for commit 3b1961b

Link to invocation

@brnguyen2
brnguyen2 merged commit 86dbc1c into NVIDIA:main Aug 13, 2026
16 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants