log prototype#85
Draft
tjtanaa wants to merge 7 commits into
Draft
Conversation
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
… clean up sparse attention; removert most of the modeling file changes Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Add a comprehensive handoff note for the current DSV4 ROCm ATOM integration checkpoint, including runnable launch settings, accuracy/perf baselines, known invalid fast paths, active top-k/CSA-translate reuse experiment, KV/workspace context, validation gates, profiler flags, and next priorities. This captures the current state before handing work to another agent. Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Add a comprehensive handoff note for the current DSV4 ROCm ATOM integration checkpoint, including runnable launch settings, accuracy/perf baselines, known invalid fast paths, active top-k/CSA-translate reuse experiment, KV/workspace context, validation gates, profiler flags, and next priorities. This captures the current state before handing work to another agent. Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
…anged Add HCA compress skip optimization to the ATOM decode index path. When decode_hca_total hasn't changed between decode steps (which is ~127 out of 128 steps), the HCA compress section is still valid and doesn't need to be rewritten. The SWA prefix is still updated by calling write_v4_paged_decode_indices with hca_block_table=None (WRITE_HCA_HEAD=False in the Triton kernel). This optimization runs on ~25% of decode steps (when decode_csa_total changed but decode_hca_total hasn't) and saves ~0.05ms per affected decode step by avoiding the redundant HCA compress section write. Changes: * model_state.py: Add hca_compress_total field to track validity * rocm.py: Add HCA compress skip path in ensure_decode_indices between the "common_valid + hca_head" and full write paths Combined with VLLM_ROCM_DSV4_USE_AITER_HC_HEAD=1, this achieves 973.82 tok/s at C32 (1024/1024, 320 prompts) — beating the V1 native baseline of 970.59 tok/s while preserving GSM8K accuracy at 0.9462. Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Test Plan
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.