hy3: Metal decode path + shared gate/up fusion#523
Open
audreyt wants to merge 16 commits into
Open
Conversation
M1 - GGUF production tooling:
- gguf-tools/glm_template/build_glm_template.py: header-only GLM template
builder. Emits the GLM-5.2 layout (98 layers incl. dens/MLP, attention standard
MLA with separate kv_b/o_proj, optional IndexShare disabled for first port),
glm.* metadata (qk_nope/rope, v_head, kv_lora, dense_ff, router_scoring,
index_*), DeepSeek-reused keys for dimensions ds4 already reads, GLM tokenizer
(BPE 154820 base + 36 added tokens).
- gguf-tools/glm-quantize.c: DeepSeek4-quantize.c cloned; adapted for GLM:
- F8_E4M3 + F32 weight_scale_inv dequant (DeepSeek used e8m0 .scale); partial-
block tolerant for the kv_a_proj 576x6144 edge case.
- Expert path uses GLM per-expert gate_proj/up_proj/down_proj (FP8+scale_inv);
no FP4 path.
- GLM tensor map: attn_kv_b / attn_output (new, GLM is standard MLA, no
factored output_a/b); ffn_gate/up/down (dense MLP); exp_probs_b.bias
(e_score_correction_bias) named to match ds4's binder.
- MTP (layer 78) deferred: standard model loaded as mtp.0.* via --mtp.
- ggt-tools/Makefile: build glm-quantize alongside deepseek4-quantize.
M3 - Loader support in ds4.c:
- Raise DS4_MAX_LAYER 61->80, EX4_VOCAB->154880, EX4_LORA_Q->2048,
EX4_EXPERT_USED->8, ADD DS4_MAX_DENSE_FF=12288, INDEXER_TOP_K->2048.
- Add DS4_VARIANT_GLM=2 + DS4_SHAPE_GLM (78L, 6144 embd, 64 heads, qk=256,
v=256, kv_lora=512, 256 experts@top-8, 3 dense, 2048 dense-ff, sigmoid/noaux,
8em6 rope base, no HC/SWA/compression/indexer).
- Extend ds4_shape with GLM-only fields: n_qk_nope/n_qk_rope/n_v_head_dim/
n_kv_lora/n_dense_layers/n_dense_ff/router_scoring/index_topk_freq.
- config_validate_model: arch-tag dispatch (glm_moe_dsa) to a new
config_validate_model_glm() that reads glm.* keys; legacy DeepSeek path
keeps strict rejection.
- Expected compress ratio: 0 for GLM (no SWA).
- Weights struct: add GLM-only ffn_gate/up/down + attn_kv_b + attn_output
pointers; weights_layer_has_required/weights_have_output_head gated on
DS4_IS_GLM(); weights_bind_output / weights_bind_layer / weights_validate_layout
branch on variant to bind GLM's standard MLA + (dense | MoE-FFN) layout with the
ds4 convention that d0=in, d1=out (matching DeepSeek tensor_nbytes semantics).
- model_open / parse_tensors gain an allow_short_file flag gated on inspect_only
so --inspect on the header-only template doesn't trigger the tensor-outside-GGUF
guard; real loads keep the safety check.
Verified: --inspect on the GLM template prints 78 layers / 256 experts / top-8
with q8_0/q2_k/iq2_xxs/f16/f32 tensors; DeepSeek model-free tests still pass
(test_q4k_dot, ds4-agent_test, ds4-eval --self-test-extractors).
Add a self-contained, correct CPU forward path used as the Gate-1 parity
oracle for the Metal graph port. GLM has no hyper-connection, so the top-
level decode driver (forward_token_raw_swa_cpu_decode_scratch) dispatches
to a plain-[n_embd] residual path: forward_glm_cpu_decode_scratch -> per-
layer layer_forward_glm_decode_one -> glm_mla_attention_one + (dense-MLP |
top-8 MoE + shared) + output_logits_one_decode_scratch_glm.
- K4786: kv_cache_init now allocates DS4_N_KV_LORA + DS4_N_QK_ROPE (576)
per row for GLM (was DS4_N_HEAD_DIM=256) — the cache stores the full
kv_a latent + rotated rope tail.
- glm_mla_attention_one:
* K_nope per-row = first 192 of kv_b(raw_kv[r][:512]) — each cached row's
own kv_b expansion (the linear map to n_head*448). K_rope per-row =
raw_kv[r][512:576] (that row's already-rotated 64-dim tail).
* Score = q_h · K over qk_nope+qk_rope=256, single softmax across the
n_raw+1 candidate rows, weighted sum of V_h = kv_b(row[:512])[h*448+192:+256].
* No inverse-rope on the output (GLM is standard MLA; V_h is unroted).
* Cost is O(seq*kv_b) per layer — fine for the CPU oracle; the GPU path
(M5) is the optimized production target.
- glm_dense_ffn_one: standard SwiGLU MLP for layers 0..n_dense_layers-1
using ffn_gate/ffn_up/ffn_down (12288-wide intermediate).
- glm_moe_one + glm_moe_router: top-8 router scoring sigmoid + e_score_corr
ection_bias for selection only; weights = sigmoid(logits)*scale/sum, runs
the IQ2_XXS/Q2_K expert matvecs and reuses the Q8_0 shared-FFN helper.
Forward declarations for both *-decode_scratch functions added so the
DeepSeek output_logits_one_decode_scratch (defined later in the file) is
visible to the GLM dispatch branch.
Verified: build clean (no warnings); the model-free subset (Q4_K dot,
extractors, agent test) still passes; --inspect still loads the GLM template.
- layer_forward_glm_decode_one: q_a and q_a_n must be sized with DS4_N_LORA_Q=2048 (the GLM Q lora rank), not kv_lora=512, which only applies to the kv_a latent. Fixes an OOB write through matvec_q8_0 (attn_q_a's output dim is 2048). - prefill_layer_major_cpu: add a GLM branch that runs the decode path token-by-token over the prompt on a plain [n_embd] residual, so the CPU oracle path covers M4's full-prompt reference too. The GPU-resident prefill graph (M5) is the optimized production path. Verified: build clean, DeepSeek model-free tests still pass (Q4_K dot, extractors, agent test).
Make Hy3 Metal runnable on M5 Max: plain-residual graph sizing when n_hc==0, LLaMA-style split-half RoPE for Q/K, GQA attention kernel, and a CLI guard that refuses oversized Metal diagnostics so a 159 GiB GGUF cannot wedge the machine during CPU/GPU compare paths.
Hy3 routes 8 experts. The Metal slots8 Q4 kernels already existed, but the C-side and batch entry still required top-6, so Q4 streaming fell back to full expert maps. Allow DS4_MAX_EXPERT_USED (8) and require the slots8 pipelines on that path so M5 Max SSD streaming can stay selected-only.
Default short-prompt prefill batches many tokens. Q4 selected-slots only existed for n_tokens==1, so multi-token SSD streaming would wrap full multi-GiB expert tensors and wedge 128 GiB machines. When SSD streaming + Q4_K + top-6/8 selected slots are available, loop the one-token slots path over batch rows. Also refuse full-expert batch wraps under streaming unless explicitly opted in, and print all 8 selected expert IDs in the profile line. Verified on M5 Max with hy3-80layer-q4k: --ssd-streaming --prefill-chunk 8 --ctx 256 -n 2, path=q4/q4 mode=stream-cache n_expert=8, ~8 GiB RSS.
Route Hy3 decode through residual single-token tensors instead of bouncing each layer through batch buffers. Fuse shared expert gate/up/SwiGLU on n_tokens==1 (same helper as Flash decode). Dense layer 0 uses batch MoE scratch sized for DS4_N_DENSE_FF. Kill switch: DS4_METAL_DISABLE_HY3_SINGLE_TOKEN_DECODE=1 restores the legacy batch bounce path. Smoke-only validation on M5 Max (~12 t/s at ctx 2048); no CPU/reference numeric equivalence yet.
|
Great to see a Metal path for Hy3. The CUDA counterpart is now open at #535: GQA single-pass online-softmax attention with per-head QK-RMSNorm and NeoX rotate-half RoPE (matches your split-half reference), full 295B with routed experts SSD-streamed per token on a single RTX 4060 Ti 16GB at ~1.8 t/s. It stacks on the GLM CUDA backend in #517. One gotcha from our quant work that may be relevant to your mixed GGUF: published imatrices never cover blk.80 (the nextn/MTP layer), since imatrix generation runs normal forwards that skip it. Its experts need an explicit non-imatrix type (we used Q2_K) and nextn.eh_proj Q8_0, or llama-quantize dies at the very last tensors. Also |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
hy_v3) single-token Metal decode on residual tensors (cur_hc/after_*_hc) instead of per-layer bounce through batch buffers.n_tokens==1(reuse Flash helper); multi-token prefill unchanged.batch_routed_*scratch sized forDS4_N_DENSE_FF(13312).DS4_METAL_DISABLE_HY3_SINGLE_TOKEN_DECODE=1restores legacy batch path.Also on this branch (prior commits): Hy3 CPU/Metal GQA, split-half RoPE, plain residual, Q4 selected-slots for top-8 MoE, multi-token Q4 selected token-loop under streaming.
Published mixed GGUF (not in git)
Optional test artifact for ~128 GB machines:
hy3-heretic-80layer-Layers74-79Q4KExperts-mixed.gguf— ~89.8 GiB (96,456,123,360 bytes), derived fromtrohrbaugh/Hy3-heretic, a decensored/abliterated Hy3 checkpoint.Quantization layout:
IQ2_XXSgate/up +Q2_KdownQ4_Krouted expertsThe runtime changes are architecture-level Hy3 support and are not specific to the Heretic weights.
SHA-256:
2dc5f74b4784b6de374196918510935ddf102cdd72272e6832edd9c3c7c2ef2bTest plan
make ds4clean build./ds4 -m …mixed.gguf --metal --ctx 2048 --tokens 32 --temp 0 --nothink -p 'What is 2+2? Answer with only the number.'→4; prefill 20.22 t/s, generation 13.16 t/sDS4_TEST_MODEL=…mixed.gguf ./ds4_test --metal-kernels→OK(includes Hy3 GQA attention and split-half RoPE references)DS4_TEST_MODEL=…mixed.gguf DS4_TEST_MPP_EQ_CASE=short_italian_fact,short_code_completion,short_reasoning_plain ./ds4_test --metal-tensor-equivalence→ 3/3 cases with identical top-20 logits, zero RMS/max-absolute drift, and identical greedy tokensSHA-256 811ea7b48d1280ad9b3999c95ad29389d6d312db6148df541973d797076a50a8);DS4_METAL_DECODE_STAGE_PROFILEemittedpart=hy3_decodeonly for the optimized run, proving the A/B traversed different branches./ds4-eval -m …mixed.gguf --plain --nothink --case-sequence 76,…,92 --tokens 128 --temp 0 --seed 1completed the full 17-case COMPSEC slice with 13/17 passing; misses were cases 79, 80, 84, and 86. This is a regression signal, not an official benchmark score.Cross-domain evaluator limitation: selected GPQA, SuperGPQA, and AIME cases repeatedly exhausted 512–2048-token budgets before emitting the required final-answer marker. Those truncated runs are not reported as capability scores; the Hy3 prompt/budget policy still needs calibration before this evaluator can produce a meaningful cross-domain baseline for the checkpoint.