feat: implementing vulkan backend

[Farenheith]

Trying to solve #237 and #476.

Being extremely honest, I don't have the knowledge to contribute to such a big feature, so I've invested 20 CNY and tried to make it work using cloud DeepSeek v4 flash + hermes.

542674649 tokens and some hours of inference later, I got the changes in this branch.
It's not working yet; the model is being loaded into memory, but inference is falling into a segmentation fault. ssd-streaming is also failing.

I don't know if any of what Hermes did is valuable, but I hope something is. I'll try to proceed later, studying and proceeding, trying to make it work, but right now, this is as far as I can get.

Here's also the skill Hermes generated during the process.
vulkan-compute-backend-porting.zip.

If this code is on the right path, I'd appreciate some help with it

[Farenheith]

Skill updated with full debugging trace

Updated the vulkan-compute-backend-porting skill with all findings from the debugging session:

• Two-file stub pattern and the return 0 vs return 1 pitfall that caused "ffn batch encode failed"
• Descriptor set sharing bug between matmul_q8_0 and matmul_f16 (separate pipelines)
• vkResetDescriptorPool invalidating pre-allocated sets
• RADV radv_amdgpu_cs_finalize crash trace
• llama.cpp comparisons: TRANSIENT_BIT, CB rotation, pool cleanup, timeline semaphores
• All memory barrier and synchronization patterns from llama.cpp

Updated skill archive

[Farenheith]

Skill updated

I have updated the vulkan-compute-backend-porting skill with a full debugging trace covering all findings from this session. It now documents:

• The return 0 vs return 1 pitfall in _impl_gen.cpp that caused "ffn batch encode failed"
• Descriptor set sharing bug between matmul_q8_0 and matmul_f16 (separate pipelines, same DS handle)
• vkResetDescriptorPool invalidating pre-allocated descriptor sets
• RADV radv_amdgpu_cs_finalize crash at radv_amdgpu_cs.c:509 — null ib_buffer
• Queue family selection and TRANSIENT_BIT pool comparison with llama.cpp
• CB rotation + periodic pool cleanup pattern from llama.cpp
• Weight descriptor range capping
• Full stage-by-stage barrier analysis

Download updated skill archive

Note: the prefill GPU path works end-to-end. The decode still crashes RADV. The root cause is likely a Vulkan state detail that still differs from llama.cpp — any help debugging the radv_amdgpu_cs_finalize crash would be greatly appreciated.

[Farenheith]

Latest efforts report

The core problem is that ./ds4 --vulkan hangs after printing "CUDA using managed KV cache" — this is during ds4_session_create → metal_graph_alloc_raw_cap (tensor allocation) completes fine, then ds4_session_sync → metal_graph_prefill_layer_major → ds4_gpu_end_commands (submit_and_wait) hangs on GPU dispatch.

Key Findings

1. The hang is ENVIRONMENTAL, not code-related.

Even the clean HEAD commit (5f285d0) — which previously produced working output at 0.42 t/s — now also hangs with the same behavior.
I reverted all changes (git checkout -- ds4.c ds4_cli.c ...) and rebuilt: still hangs.
The kernel log confirms: amdgpu 0000:c4:00.0: [drm] device wedged, but recovered through reset (from earlier today, likely due to prior crash testing during our earlier MoE shader debug sessions).
2. GPU is partially functional but unstable.

✅ Basic Vulkan device creation/queue/submission works (tested via standalone C program).
✅ 1000 pipeline barriers submit and complete successfully.
✅ --head-test works (uses CPU compute only, no GPU dispatches).
✅ --first-token-test works (CPU compute only).
✅ --cpu mode works (2.03 t/s prefill, 2.22 t/s generation).
❌ Any GPU compute dispatch (via session-based generation) hangs indefinitely in vkQueueSubmit + vkWaitForFences.
3. Hang is non-deterministic.

With debug instrumentation, the hang occurs at different layers on different runs (layer 18, layer 20, layer 25, etc.) — classic sign of memory corruption / undefined behavior.
The shaders are valid SPIR-V (disassembles fine).
A standalone matmul_f16 dispatch test also fails (returned VK_ERROR_OUT_OF_HOST_MEMORY, but I need to double-check the error code).
4. What works on GPU:

Weight cache loading (80.76 GiB of tensor spans in ~25s)
All tensor allocations (60 layers × multiple buffers each)
Command buffer recording (all 60 layer encodes report ok=1)
Individual layer encoding succeeds (attention + FFN for each layer)
5. What's broken:

The final vkQueueSubmit + vkWaitForFences blocks forever
RADV debug flags (RADV_DEBUG=hang, RADV_DEBUG=cs) don't report any hang detection
Suspected Root Cause
The earlier GPU MoE shader debug sessions likely left the GPU in a wedged state. The driver "recovered" but the GPU may have persistent state corruption. The fact that basic Vulkan works but compute shaders hang points to a hardware/driver-level issue rather than a bug in our code.

Pending Work
The original tasks from the todo list are:

GPU MoE shaders — blocked by the GPU hang
Compressor — CPU implementation needed
Indexer — CPU implementation needed
Final test & tune — blocked by the GPU hang