[#17723][feat] Parse multiple compressed-tensors config groups - #17725
Draft
mihai-chiorean wants to merge 2 commits into
Draft
[#17723][feat] Parse multiple compressed-tensors config groups#17725mihai-chiorean wants to merge 2 commits into
mihai-chiorean wants to merge 2 commits into
Conversation
Signed-off-by: Mihai Chiorean <mihai.v.chiorean@gmail.com>
This was referenced Aug 14, 2026
Signed-off-by: Mihai Chiorean <mihai.v.chiorean@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Parse every compressed-tensors
config_groupsentry into aMIXED_PRECISIONglobal config plus an explicit per-modulequant_config_dict. Preserve target ordering, reject unmatched targets, and keep each module configuration independent. Resolve module names from local safetensors files or a Hugging Face Hub index.This is required by
unsloth/Qwen3.8-27B-NVFP4, which mixes packed NVFP4 dense MLPs with rowwise FP8 attention, GDN, later MLP, andlm_headmodules.Related to #17723.
Details
nvfp4-pack-quantizedas native W4A4NVFP4; activation scales are part of that format and must not be discarded by relabeling it W4A16.ignoreentries while materializing the authoritative multi-group map without copying them into recursive global exclusions, which could shadow explicitly quantized children.modules_to_not_convertas a recursive exclusion in the multi-group path.Testing
pre-commit run --fileson all changed filesModelConfig.load_hf_quant_configwiring teststrtllm::nvfp4_gemmand generated coherent outputDependencies
This parser is generic and can merge independently. Qwen3.8 end-to-end loading additionally needs the packed-name, dense-MLP path, and FP8
lm_headPRs linked from #17723.Interaction with #17551
#17551 merges producer
ignoreinto recursive exclusions for homogeneous configs. If it lands first, this PR must retain an explicitMIXED_PRECISIONcarve-out: the per-module map is authoritative for multi-group configs, and recursively re-injectingignorecan shadow a quantized child.