[TRTLLM-15078][test] Prune non-Llama-3.1-8B Llama tests by xinhe-nv · Pull Request #17392 · NVIDIA/TensorRT-LLM

xinhe-nv · 2026-08-07T02:25:22Z

Dev Engineer Review

Pruned non-Llama-3.1-8B model references, accuracy data, performance configurations, and integration tests.
Replaced selected TinyLlama usage with Llama-3.1-8B-Instruct.
Removed Ray, MPI, LoRA, KV-cache, LLM API, and disaggregated-serving coverage.
Added Blackwell SM120 Mixtral E2E coverage and Qwen3 Eagle3 coverage.
Validate removed functional coverage, model paths, YAML files, test references, waiver entries, and test_llm.py golden outputs.
No production runtime changes were identified.

QA Engineer Review

Test code changed across accuracy, disaggregated serving, E2E, KV-cache, LLM API, Ray, MPI, LoRA, sampler, and AutoDeploy areas.

Removed or modified tests include accuracy classes, disaggregated-serving functions, Ray and MPI examples, KV-cache tests, LLM API tests, LoRA helpers, sampler tests, and model-path fixtures. Several retained tests now use Llama-3.1-8B-Instruct.

Affected manifests:

tests/integration/test_lists/qa/llm_function_core.txt
tests/integration/test_lists/qa/llm_spark_core.txt
tests/integration/test_lists/qa/llm_spark_func.yml
tests/integration/test_lists/qa/llm_spark_perf.yml
tests/integration/test_lists/test-db/l0_a10.yml
tests/integration/test_lists/test-db/l0_a100.yml
tests/integration/test_lists/test-db/l0_b200.yml
tests/integration/test_lists/test-db/l0_dgx_b200.yml
tests/integration/test_lists/test-db/l0_dgx_h100.yml
tests/integration/test_lists/test-db/l0_dgx_h200.yml
tests/integration/test_lists/test-db/l0_gb200_multi_gpus.yml
tests/integration/test_lists/test-db/l0_h100.yml
tests/integration/test_lists/test-db/l0_l40s.yml
tests/integration/test_lists/test-db/l0_sanity_check.yml
tests/integration/test_lists/waives.txt

CBTS coverage data is unavailable. Verdict: needs follow-up.

Description

The background is https://docs.google.com/spreadsheets/d/17qLRr5oQESqn1rT9LrIUnpjvhH8wcC59273YEPT9k44/edit?gid=177162358#gid=177162358

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

xinhe-nv · 2026-08-07T03:39:26Z

@coderabbitai review the change, we need to retire llama tests, if the test is must have, please show me in a table.

coderabbitai · 2026-08-07T03:43:04Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

Walkthrough

The PR removes obsolete model accuracy, performance, disaggregated-serving, Ray, LoRA, and API test coverage. It updates integration manifests and retargets selected unit tests from TinyLlama to Llama 3.1 8B Instruct.

Changes

Test matrix cleanup

Layer / File(s)	Summary
Model references and accuracy coverage `tests/integration/defs/accuracy/`, `tests/integration/defs/conftest.py`, `tests/integration/defs/perf/`, `tests/test_common/llm_data.py`	Removed obsolete accuracy records, model mappings, fixtures, and performance configurations.
Disaggregated and Ray coverage `tests/integration/defs/disaggregated/*`, `tests/integration/defs/examples/test_ray.py`	Removed TinyLlama disaggregated tests, Ray flows, model helpers, and configurations.
Integration definitions and manifests `tests/integration/defs/test_e2e.py`, `tests/integration/test_lists/`, `tests/scripts/perf-sanity/`	Removed obsolete cases and retained or added current Qwen, DeepSeek, GPT-OSS, and decoder-encoding coverage.
Unit-test model migration `tests/unittest/_torch/**`, `tests/unittest/llmapi/test_async_llm.py`, `tests/unittest/llmapi/test_executor.py`, `tests/unittest/llmapi/test_grpc.py`	Changed selected tests to use Llama 3.1 8B Instruct.
LLM API and LoRA cleanup `tests/unittest/llmapi/apps/*`, `tests/unittest/llmapi/lora_test_utils.py`, `tests/unittest/llmapi/test_llm_pytorch.py`	Removed obsolete API and LoRA harnesses. Updated the remaining GQA LoRA test dimensions and output checks.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#14660: Removes deprecated model coverage and shared model-path mappings.
NVIDIA/TensorRT-LLM#17444: Removes obsolete entries from the same waiver file.
NVIDIA/TensorRT-LLM#17487: Reorganizes Ray-related modules and imports used by these tests.

Suggested reviewers: qijune, pcastonguay, joyang-nv

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description provides only a background link and does not explain the changes, rationale, or relevant test coverage.	Add a concise description of the changes and rationale, and list the relevant tests or validation performed.
Docstring Coverage	⚠️ Warning	Docstring coverage is 32.61% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the main change: pruning tests for models other than Llama-3.1-8B.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

⚔️ Resolve merge conflicts 💡

Resolve merge conflict in branch prune-llama

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

tensorrt-cicd · 2026-08-13T15:49:47Z

PR_Github #65953 [ run ] completed with state FAILURE. Commit: cb578f6
/LLM/main/L0_MergeRequest_PR pipeline #53643 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xinhe-nv · 2026-08-13T16:37:40Z

/bot run

tensorrt-cicd · 2026-08-13T16:44:04Z

PR_Github #65976 [ run ] triggered by Bot. Commit: ae2b5d4 Link to invocation

tensorrt-cicd · 2026-08-13T17:28:19Z

PR_Github #65976 [ run ] completed with state SUCCESS. Commit: ae2b5d4
/LLM/main/L0_MergeRequest_PR pipeline #53664 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

Signed-off-by: YihuiLu512 <269394165+YihuiLu512@users.noreply.github.com>

temperature is typed as float but set to None in the greedy-sampling branch of sampler_strategy.py. Suppress the assignment error at the module level in pyproject.toml pending an annotation fix upstream. Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

xinhe-nv · 2026-08-14T02:28:59Z

/bot run

tensorrt-cicd · 2026-08-14T02:34:28Z

PR_Github #66126 [ run ] triggered by Bot. Commit: f879867 Link to invocation

Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

tensorrt-cicd · 2026-08-14T03:00:03Z

PR_Github #66126 [ run ] completed with state FAILURE. Commit: f879867
/LLM/main/L0_MergeRequest_PR pipeline #53800 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xinhe-nv · 2026-08-14T03:12:02Z

/bot run

tensorrt-cicd · 2026-08-14T03:20:33Z

PR_Github #66146 [ run ] triggered by Bot. Commit: 7abfa52 Link to invocation

tensorrt-cicd · 2026-08-14T03:52:55Z

PR_Github #66146 [ run ] completed with state FAILURE. Commit: 7abfa52
/LLM/main/L0_MergeRequest_PR pipeline #53822 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>

…list entries Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

BowenFu · 2026-08-14T05:00:25Z

    "redundant-cast",
+    # temperature is typed as float but set to None in the greedy case (sampler_strategy.py).
+    "assignment",
 ]


Could you remove this sampler-wide assignment suppression and its preceding comment from this test-pruning PR? It weakens future checks across the module; keep the focused type fix in the owning sampler change.

xinhe-nv · 2026-08-14T06:06:01Z

/bot run

tensorrt-cicd · 2026-08-14T06:12:57Z

PR_Github #66197 [ run ] triggered by Bot. Commit: 098a4ec Link to invocation

tensorrt-cicd · 2026-08-14T07:03:29Z

PR_Github #66197 [ run ] completed with state FAILURE. Commit: 098a4ec
/LLM/main/L0_MergeRequest_PR pipeline #53869 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…move deleted test file refs Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

xinhe-nv · 2026-08-14T07:12:37Z

/bot run

tensorrt-cicd · 2026-08-14T07:25:40Z

PR_Github #66213 [ run ] triggered by Bot. Commit: bd4082d Link to invocation

tensorrt-cicd · 2026-08-14T09:12:34Z

PR_Github #66213 [ run ] completed with state FAILURE. Commit: bd4082d
/LLM/main/L0_MergeRequest_PR pipeline #53886 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

github-actions Bot assigned xinhe-nv Aug 7, 2026

xinhe-nv requested review from QiJune, chenfeiz0326, fredricz-20070104, joyang-nv, kaiyux and litaotju and removed request for fredricz-20070104 August 7, 2026 02:26

xinhe-nv force-pushed the prune-llama branch from 9715f40 to a254aa0 Compare August 7, 2026 02:55

xinhe-nv marked this pull request as ready for review August 7, 2026 03:37

xinhe-nv requested review from a team as code owners August 7, 2026 03:37

xinhe-nv requested review from EmmaQiaoCh, StanleySun639, chienchunhung, fredricz-20070104, hyukn, mlefeb01, pcastonguay and ruodil August 7, 2026 03:37

xinhe-nv force-pushed the prune-llama branch from 0624279 to ae2b5d4 Compare August 13, 2026 16:33

xinhe-nv and others added 4 commits August 14, 2026 10:28

prune none-llama3.8-8b models

0e655f0

Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

prune none-llama3.8-8b models

395af1c

Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

[https://nvbugs/6422337][fix] Unwaive testcase (NVIDIA#17496)

b34b8f2

Signed-off-by: YihuiLu512 <269394165+YihuiLu512@users.noreply.github.com>

xinhe-nv force-pushed the prune-llama branch from ae2b5d4 to f879867 Compare August 14, 2026 02:28

fix issue

76d258c

Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

Merge branch 'main' into prune-llama

7abfa52

xinhe-nv added 3 commits August 14, 2026 12:12

Merge branch 'main' into prune-llama

757e8d3

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>

[TRTLLM-15078][fix] Remove stale type: ignore comment and stale test …

bcb0400

…list entries Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

[None][test] Restore test_llm_inference_async_ray to l0_h100

098a4ec

Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

BowenFu reviewed Aug 14, 2026

View reviewed changes

xinhe-nv added 2 commits August 14, 2026 15:08

[None][test] Fix invalid test list entries: strip stale params and re…

c820aa1

…move deleted test file refs Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

Merge branch 'main' into prune-llama

bd4082d

Uh oh!

Conversation

xinhe-nv commented Aug 7, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev Engineer Review

QA Engineer Review

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

xinhe-nv commented Aug 7, 2026

Uh oh!

coderabbitai Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

❌ Failed checks (2 warnings)

Uh oh!

tensorrt-cicd commented Aug 13, 2026

Uh oh!

xinhe-nv commented Aug 13, 2026

Uh oh!

tensorrt-cicd commented Aug 13, 2026

Uh oh!

tensorrt-cicd commented Aug 13, 2026

Uh oh!

xinhe-nv commented Aug 14, 2026

Uh oh!

tensorrt-cicd commented Aug 14, 2026

Uh oh!

tensorrt-cicd commented Aug 14, 2026

Uh oh!

xinhe-nv commented Aug 14, 2026

Uh oh!

tensorrt-cicd commented Aug 14, 2026

Uh oh!

tensorrt-cicd commented Aug 14, 2026

Uh oh!

BowenFu Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

xinhe-nv commented Aug 14, 2026

Uh oh!

tensorrt-cicd commented Aug 14, 2026

Uh oh!

tensorrt-cicd commented Aug 14, 2026

Uh oh!

xinhe-nv commented Aug 14, 2026

Uh oh!

tensorrt-cicd commented Aug 14, 2026

Uh oh!

tensorrt-cicd commented Aug 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

xinhe-nv commented Aug 7, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Aug 7, 2026 •

edited

Loading