Skip to content

[#17723][feat] Parse multiple compressed-tensors config groups - #17725

Draft
mihai-chiorean wants to merge 2 commits into
NVIDIA:mainfrom
mihai-chiorean:feat/17723-compressed-tensors-groups
Draft

[#17723][feat] Parse multiple compressed-tensors config groups#17725
mihai-chiorean wants to merge 2 commits into
NVIDIA:mainfrom
mihai-chiorean:feat/17723-compressed-tensors-groups

Conversation

@mihai-chiorean

@mihai-chiorean mihai-chiorean commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Summary

Parse every compressed-tensors config_groups entry into a MIXED_PRECISION global config plus an explicit per-module quant_config_dict. Preserve target ordering, reject unmatched targets, and keep each module configuration independent. Resolve module names from local safetensors files or a Hugging Face Hub index.

This is required by unsloth/Qwen3.8-27B-NVFP4, which mixes packed NVFP4 dense MLPs with rowwise FP8 attention, GDN, later MLP, and lm_head modules.

Related to #17723.

Details

  • Preserve homogeneous single-group behavior.
  • Parse nvfp4-pack-quantized as native W4A4 NVFP4; activation scales are part of that format and must not be discarded by relabeling it W4A16.
  • Apply producer ignore entries while materializing the authoritative multi-group map without copying them into recursive global exclusions, which could shadow explicitly quantized children.
  • Preserve only modules_to_not_convert as a recursive exclusion in the multi-group path.
  • Read sharded indexes and single safetensors headers from local paths or Hub IDs; fail before model construction if the per-module map cannot be resolved.

Testing

  • pre-commit run --files on all changed files
  • Local-index, Hub-index, and ModelConfig.load_hf_quant_config wiring tests
  • Parser and model-config tests included in a 200-pass integrated Spark run
  • Full Qwen3.8-27B NVFP4 SM121 smoke reached native trtllm::nvfp4_gemm and generated coherent output

Dependencies

This parser is generic and can merge independently. Qwen3.8 end-to-end loading additionally needs the packed-name, dense-MLP path, and FP8 lm_head PRs linked from #17723.

Interaction with #17551

#17551 merges producer ignore into recursive exclusions for homogeneous configs. If it lands first, this PR must retain an explicit MIXED_PRECISION carve-out: the per-module map is authoritative for multi-group configs, and recursively re-injecting ignore can shadow a quantized child.

Signed-off-by: Mihai Chiorean <mihai.v.chiorean@gmail.com>
Signed-off-by: Mihai Chiorean <mihai.v.chiorean@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant