Skip to content

[TRTLLM-15115][test] Prune Starcoder2-3B functional and unit tests - #17674

Open
xinhe-nv wants to merge 3 commits into
NVIDIA:mainfrom
xinhe-nv:trtllm-15115
Open

[TRTLLM-15115][test] Prune Starcoder2-3B functional and unit tests#17674
xinhe-nv wants to merge 3 commits into
NVIDIA:mainfrom
xinhe-nv:trtllm-15115

Conversation

@xinhe-nv

@xinhe-nv xinhe-nv commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Remove out-of-scope Starcoder2-3B coverage per Test Scope Review 2026.7, including QA/CI list entries, encode/cli harnesses, and 3B-specific unit tests while leaving StarCoder2-7B/15B coverage intact.

Dev Engineer Review

  • Removed StarCoder2-3B accuracy references, harness coverage, fixtures, performance configuration, and unit-test coverage.
  • Preserved StarCoder2-7B and StarCoder2-15B coverage.
  • Updated QA, A100, and test-duration entries consistently.
  • No production-code or API changes were introduced.
  • Configuration and test-list changes have valid scope and no apparent format or duplication issues.

QA Engineer Review

  • Removed TestStarcoder2_3B from tests/integration/defs/accuracy/test_cli_flow.py.
  • Removed StarCoder2-3B scenarios from tests/unittest/_torch/modeling/test_modeling_starcoder2.py.
  • Removed StarCoder2-3B tokenizer coverage from tests/unittest/llmapi/test_llm.py.
  • Removed the related accuracy entries from:
    • tests/integration/test_lists/qa/llm_function_core.txt
    • tests/integration/test_lists/test-db/l0_a100.yml
  • The removed test coverage is reflected in the QA and CI lists. StarCoder2-7B and StarCoder2-15B coverage remains.
  • Verdict: needs follow-up because two A100 CI runs failed. The subsequent A100X run passed with fail-fast disabled.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

@xinhe-nv

Copy link
Copy Markdown
Collaborator Author

/bot run --stage-list "a100"

@coderabbitai

coderabbitai Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 0059d5db-7bb9-4153-8006-84365b666592

📥 Commits

Reviewing files that changed from the base of the PR and between 662299d and 18dc0c7.

📒 Files selected for processing (1)
  • tests/unittest/llmapi/test_llm.py
💤 Files with no reviewable changes (1)
  • tests/unittest/llmapi/test_llm.py

Walkthrough

The PR removes StarCoder2-3B from accuracy references, integration schedules, model fixtures, performance configurations, timing data, attention settings, modeling tests, and tokenizer test parameters.

Changes

StarCoder2-3B coverage removal

Layer / File(s) Summary
Integration model coverage
tests/integration/defs/accuracy/test_llm_api_pytorch_encode.py, tests/integration/defs/conftest.py, tests/integration/defs/perf/allowed_configs.py
The decoder model parameter, model fixtures, and performance allowlist entry no longer include starcoder2-3b.
Integration references and schedules
tests/integration/defs/accuracy/references/gsm8k.yaml, tests/integration/defs/accuracy/test_cli_flow.py, tests/integration/test_lists/qa/llm_function_core.txt, tests/integration/test_lists/test-db/l0_a100.yml, tests/integration/defs/.test_durations
StarCoder2-3B accuracy references, scheduled decoder encode tests, and recorded test timing are removed.
Unit model coverage
tests/unittest/_torch/attention/model_attn_config.py, tests/unittest/_torch/modeling/test_modeling_starcoder2.py, tests/unittest/llmapi/test_llm.py
The StarCoder2-3B attention configuration, modeling scenarios, related imports, and tokenizer parameter are removed. StarCoder2-7B and StarCoder2-15B coverage remains.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 18dc0

This test-coverage pruning change is merge-ready after normal checks and review; no actionable merge-blocking risk remains.

Suggested reviewers: asfiyab-nvidia, brnguyen2, yechank-nvidia, stanleysun639, zhaoyuanh-nvidia

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the removal of StarCoder2-3B functional and unit test coverage.
Description check ✅ Passed The description explains the scope and intent, but the Test Coverage section is empty while the checklist is completed.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #66167 [ run ] triggered by Bot. Commit: 3e3cce7 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #66167 [ run ] completed with state FAILURE. Commit: 3e3cce7
/LLM/main/L0_MergeRequest_PR pipeline #53841 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@xinhe-nv

Copy link
Copy Markdown
Collaborator Author

/bot run --gpu-type "A100X"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #66202 [ run ] triggered by Bot. Commit: 3e3cce7 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #66202 [ run ] completed with state FAILURE. Commit: 3e3cce7
/LLM/main/L0_MergeRequest_PR pipeline #53871 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@xinhe-nv

Copy link
Copy Markdown
Collaborator Author

/bot run --gpu-type "A100X" --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #66296 [ run ] triggered by Bot. Commit: db9af45 Link to invocation

Remove out-of-scope Starcoder2-3B coverage per Test Scope Review 2026.7,
including QA/CI list entries, encode/cli harnesses, and 3B-specific unit
tests while leaving StarCoder2-7B/15B coverage intact.

Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #66296 [ run ] completed with state SUCCESS. Commit: db9af45
/LLM/main/L0_MergeRequest_PR pipeline #53955 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants