[https://nvbugs/6606123][test] Waive AutoDeploy shim SWA-eviction + mamba-cache unit tests (setup OOM) - #17653
Conversation
…amba-cache unit tests (setup OOM) The `test_unittests_v2[unittest/auto_deploy/singlegpu/shim]` wrapper flakes on H100_PCIe-AutoDeploy-1 and A30-AutoDeploy-1 (~2.8% over 14 days, 321 MRs, 25 users). The failures are concentrated in a small set of memory-heavy tests that OOM on setup (cudaMalloc, kv_cache_manager::WindowBlockManager) while allocating their KV/window cache — not spread across the directory. Waive only those tests so the other ~109 shim tests keep running, rather than the whole directory. Tracked in nvbugs/6606123. Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>
|
/bot skip --comment "waives-only PR" |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
WalkthroughThe integration waiver list adds two NVBug-linked skip entries for auto-deploy singlegpu shim tests. ChangesAuto-deploy test waivers
Estimated code review effort: 1 (Trivial) | ~2 minutes Mergeability Score: ⚪ Minimal · up to This PR only waives two identified setup-OOMing tests while leaving the rest of the shim coverage unchanged; no production or build behavior changes are introduced, and no actionable merge-blocking risk remains beyond normal checks and review. Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
PR_Github #66004 [ skip ] triggered by Bot. Commit: |
|
PR_Github #66004 [ skip ] completed with state |
Description
The
test_unittests_v2[unittest/auto_deploy/singlegpu/shim]wrapper flakes on H100_PCIe-AutoDeploy-1 and A30-AutoDeploy-1 — ~2.8% over the last 14 days (44/1547 runs), across 321 distinct PRs and 25 distinct users, so it is main-side, not any one PR.The failures are not spread across the directory (114 tests / 5 files). They concentrate in a small set of memory-heavy tests that OOM on setup (
cudaMalloc ... out of memoryinkv_cache_manager::WindowBlockManager; the mamba test hitstorch CUDA out of memory) while allocating their KV / window cache — intermittent because it depends on how much GPU memory earlier tests in the shard leave free.This waives only those tests, so the other ~109 shim tests keep their coverage (rather than a directory-level waive):
unittest/auto_deploy/singlegpu/shim/test_ad_executor_swa_eviction.py(4 of its 9 tests fail; whole file waived)unittest/auto_deploy/singlegpu/shim/test_cached_sequence_interface.py::test_initialize_resources_mixed_creates_mamba_hybrid_cache_managerTracked in nvbugs/6606123 (assigned to the AutoDeploy owner).
Test Coverage
Test-list-only change (no source/build change). Local pre-commit passed, including
Validate test list entries exist in source files (AST)and the waives.txt duplicate/format checks.PR Checklist
Dev Engineer Review
tests/integration/test_lists/waives.txt.test_ad_executor_swa_eviction.pyandtest_initialize_resources_mixed_creates_mamba_hybrid_cache_manager.nvbugs/6606123.QA Engineer Review
tests/integration/test_lists/waives.txt.nvbugs/6606123.