Skip to content

[#17723][fix] Preserve rowwise FP8 Qwen lm_head weights - #17728

Draft
mihai-chiorean wants to merge 2 commits into
NVIDIA:mainfrom
mihai-chiorean:fix/17723-qwen38-fp8-lm-head
Draft

[#17723][fix] Preserve rowwise FP8 Qwen lm_head weights#17728
mihai-chiorean wants to merge 2 commits into
NVIDIA:mainfrom
mihai-chiorean:fix/17723-qwen38-fp8-lm-head

Conversation

@mihai-chiorean

@mihai-chiorean mihai-chiorean commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Summary

Keep an explicit FP8_PER_CHANNEL_PER_TOKEN Qwen3.5/3.8 lm_head entry quantized instead of forcing the head to bf16. Reject unsupported FP8 layouts where fallback would cast the FP8 weight and discard its scale.

Related to #17723.

Motivation

unsloth/Qwen3.8-27B-NVFP4 stores lm_head.weight as E4M3 plus a rowwise scale. The previous Qwen normalization always excluded lm_head from quantization; the generic loader then assigned the E4M3 tensor to a bf16 parameter and ignored its scale, corrupting logits.

Safety

The quantized path is enabled only for the rowwise FP8 format demonstrated by the checkpoint. Tied embeddings, lm_head TP under attention-DP, missing/non-divisible vocabulary metadata, and unsupported FP8 formats raise instead of silently losing scale information. Other quantized algorithms retain their existing fallback behavior. Existing W4A16 NVFP4 behavior is preserved.

Ordinary attention-DP without lm_head TP is allowed because LMHead is TP-less in that mode; this matches DecoderModelForCausalLM._resolve_lm_head_quant_config.

Testing

  • pre-commit run --files on all changed files
  • Focused head tests included in a 200-pass integrated Spark run
  • Real SM121 load constructed LMHead as FP8_PER_CHANNEL_PER_TOKEN with weight shape (248320, 5120) and E4M3 dtype
  • Full model generated coherent output both without MTP and with the checkpoint MTP layer

Dependencies

Qwen3.8 receives the explicit rowwise-FP8 head entry from #17725.

Signed-off-by: Mihai Chiorean <mihai.v.chiorean@gmail.com>
Signed-off-by: Mihai Chiorean <mihai.v.chiorean@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant