[#17723][fix] Preserve rowwise FP8 Qwen lm_head weights - #17728
Draft
mihai-chiorean wants to merge 2 commits into
Draft
[#17723][fix] Preserve rowwise FP8 Qwen lm_head weights#17728mihai-chiorean wants to merge 2 commits into
mihai-chiorean wants to merge 2 commits into
Conversation
Signed-off-by: Mihai Chiorean <mihai.v.chiorean@gmail.com>
3 tasks
Signed-off-by: Mihai Chiorean <mihai.v.chiorean@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Keep an explicit
FP8_PER_CHANNEL_PER_TOKENQwen3.5/3.8lm_headentry quantized instead of forcing the head to bf16. Reject unsupported FP8 layouts where fallback would cast the FP8 weight and discard its scale.Related to #17723.
Motivation
unsloth/Qwen3.8-27B-NVFP4storeslm_head.weightas E4M3 plus a rowwise scale. The previous Qwen normalization always excludedlm_headfrom quantization; the generic loader then assigned the E4M3 tensor to a bf16 parameter and ignored its scale, corrupting logits.Safety
The quantized path is enabled only for the rowwise FP8 format demonstrated by the checkpoint. Tied embeddings, lm_head TP under attention-DP, missing/non-divisible vocabulary metadata, and unsupported FP8 formats raise instead of silently losing scale information. Other quantized algorithms retain their existing fallback behavior. Existing W4A16 NVFP4 behavior is preserved.
Ordinary attention-DP without lm_head TP is allowed because
LMHeadis TP-less in that mode; this matchesDecoderModelForCausalLM._resolve_lm_head_quant_config.Testing
pre-commit run --fileson all changed filesLMHeadasFP8_PER_CHANNEL_PER_TOKENwith weight shape(248320, 5120)and E4M3 dtypeDependencies
Qwen3.8 receives the explicit rowwise-FP8 head entry from #17725.