Hy3 (295B): CUDA backend support, GQA attention path#535
Open
giannisanni wants to merge 7 commits into
Open
Conversation
added 7 commits
July 7, 2026 08:20
Implements the CUDA side of GLM 5.2 inference on top of the glm5.2
branch engine: MLA attention, DSA indexer, compact KV, routed MoE with
IQ2_XXS gate/up/down dp4a kernels, and SSD expert streaming (io_uring
O_DIRECT fetch engine, host LFU expert cache, parallel fetch backfill).
Engine changes are minimal:
- allow DS4_BACKEND_CUDA through the GLM backend gate,
- gate the split value-projection path on the f16 compact cache: it
assumes the Metal f16 layout and fails the batch attention-lora encode
on CUDA's f32 cache.
The CUDA backend replaces the placeholder stubs with real
implementations and adds the GLM kernel implementations in
ds4_cuda_glm_{kv,indexer,attn,moe,stubs}.inc.
Includes the batch expert-tile LUT initialization fix also submitted
against main as antirez#513 (models with n_embd > 4096).
Tested on RTX 4060 Ti 16GB (sm_89), 32GB RAM host, Linux, against the
official GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf with
--ssd-streaming: greedy decode correct at ~0.3-0.4 t/s, 200-token batch
prefill correct at ~3.2 t/s, 600-token at ~6.5 t/s.
Engine side: DS4_MODEL_FAMILY_HY_V3 with the full shape profile (80+1 layers, GQA 64q/8kv head_dim 128, 192 experts top-8, dense layer 0, sigmoid router with expert bias and route-norm via the parameterized GLM router kernel at scale 2.826, rope theta 11158840, ctx 262144), llama.cpp-style tensor-name contract, layout/config validators, layer binder, and an engine gate that allows --inspect and refuses generation until inference is wired. CUDA side: ds4_cuda_gqa.inc adds the genuinely new piece, grouped-query attention (ds4 was MLA-only): per-head weighted QK-RMSNorm, rotate-half RoPE, per-kv-head f32 KV cache, two-pass-softmax causal GQA kernel, and a CPU-reference self-test (ds4 --gqa-selftest, no model file needed) covering a 13-token prefill and a 1-token decode against the populated cache. Maxima raised: DS4_MAX_LAYER 81, DS4_MAX_HEAD_KV 8, DS4_MAX_ROT 128.
… + gate (untested) Wires the hy_v3 generation path: hy3_state_alloc/free (per-layer f32 GQA KV caches), hy3_forward_token (attn_norm -> qkv proj -> QK-norm -> RoPE -> KV append -> GQA attention -> o_proj -> residual -> ffn_norm -> dense or router+shared+routed MoE -> residual -> output head), the argmax decode loop, dispatch from the graph generation entry, and the engine gate opened to --cuda. Syntax-clean (gcc -fsyntax-only); NOT yet built with nvcc and no converted model exists yet to run it. Reuses GLM router/shexp/streaming primitives and the new ds4_cuda_gqa.inc kernels.
vocab_load fell through to the DeepSeek special-token strings and died on the first real Hy3 gguf. The file carries tokenizer.ggml.bos/eos_token_id (120000/120025); resolve by id like the GLM branch does, chat-role and tool tokens stay unset for v1.
…mpute The two-pass kernel recomputed QK scores once per output dim in the V pass (~130x the FLOPs; at long context that re-reads K head_dim times past L1). Rewritten as online softmax: scores computed exactly once per chunk of 128 rows, shared-mem weights, per-dim V accumulation with running max/denominator rescale. Launch tightened to 128 threads. Self-test extended from 2 to 4 phases (cap 64 -> 256): a 200-token prefill and a decode at row 215 now cross the chunk boundary and exercise the rescale path. All phases pass at <1e-06 vs the CPU ref. Measured on Hy3 295B decode at short context (40-160 tokens): flat (1.80 -> 1.82 t/s) - expert streaming dominates there, as expected. The win is structural: no L1-spill cliff at long context, and this is the kernel batch prefill will sit on. Makefile: ds4_cuda.o now depends on the .inc files it includes; editing ds4_cuda_gqa.inc alone previously produced a silent stale build.
Wires DS4_MODEL_FAMILY_HY_V3 into the session engine and chat builders so the REPL and one-shot chat paths work like GLM, no --raw needed: - vocab_load resolves Hy3's real chat tokens (<hy_User>, <hy_Assistant>, <think:opensource> pair) instead of leaving them unset; reuses the GLM-only sop_id slot for the reasoning-mode marker. - chat prompt assembly (encode_chat_prompt, ds4_chat_append_message, ds4_chat_append_assistant_prefix) emits the no-think framing assistant+think_begin+think_end, matching the GGUF chat template. - ds4_session gains a persistent hy3_gpu_state: session_create allocs it once, session_sync forwards only the suffix when the prompt extends the running checkpoint (positional KV cache), session_eval decodes one token, session_free releases it. Speculative path falls back to plain eval (no MTP for Hy3 yet). Verified on substrate: multi-turn chat, turn 2 prefills only the 13 new delta tokens (not the full history), coherent assistant answers at ~1.7 t/s. CPU sessions rejected with a clear 'requires --cuda'. Skipped: reasoning_effort system marker (model follows markers without it at no_think default); tool-call framing; thinking-history reconstruction. Add when a use case needs them.
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Tencent Hy3 (
hy-v3, 295B total / 21B active MoE) support on the CUDA backend from #517. This stacks on that PR: merge #517 first, the first commit here is that branch.ds4_cuda_gqa.inc): single-pass online-softmax kernel for 64 query / 8 KV heads at head_dim 128, per-head QK-RMSNorm, NeoX rotate-half RoPE. First non-MLA attention family in ds4.hy-v3GGUFs: 192 experts with top-8 sigmoid router, expert bias and route norm (scale 2.826), 1 shared expert, dense FFN on layer 0, vocab 120832. bos/eos resolved from GGUF metadata token ids instead of DeepSeek literals.--gqa-selftest: 4-phase CPU-reference check of the attention kernel (prefill, decode, long-chunk, decode-after-long), all phases < 1e-06.Tested
RTX 4060 Ti 16GB, Ryzen 9900X, 32GB RAM, one Gen4 NVMe. Mixed GGUF (routed experts IQ2_XXS, attention/shared/dense Q8_0, https://huggingface.co/giannisan/Hy3-ds4-gguf):
make ds4clean build (CUDA 12.x, sm_89)--gqa-selftest4/4 phases pass (max diff 8.15e-07)blk.80 (nextn/MTP) binds when present but is not wired into decode: measurements on GLM show the accept loop cannot pay while expert streaming dominates the token clock. It can be enabled later without requantizing.
Complementary to #523, which adds the Metal path for the same architecture.