feat: harden Qwen 3.5/3.6 LoRA distributed training

[NolanHo]

Scope

Harden the native C++ Qwen 3.5/3.6 LoRA training path and add progressively verified distributed runtime slices. This remains a draft because the full Megatron-level MoE performance stack, PP/CP, cross-topology resharding, real multi-node validation, and a matched Megatron benchmark are not complete.

Delivered

• Fixed and dynamic multi-LoRA coverage for full attention, GDN, dense MLP, shared experts, and routed experts, with independent tenant optimizer clocks and checkpoint state.

• Independent TP, EP, DP, and expert-DP communicators with strict topology validation.

• TP x EP x DP source-coordinate handling for source-sharded and replicated-source EP policies. Server workers slice one global request by (DP,EP) source coordinate while TP peers share rows; dynamic requests require the complete global source batch.

• Frozen attention/GDN/dense MLP and vocabulary TP, ABI19 packed top-k sharded A2A, and ABI20 routed/shared expert tensor parallelism. The server composes EP expert narrowing before routed/shared TP sharding and rejects unsupported replicated-expert TP early.

• Checkpoint v5 with explicit rank order, five-axis coordinates, TP/EP compound tensor placements, fused routed gate/up segments, replica axes, and fixed/dynamic slot identities.

• Explicit per-save checkpoint generations, same-generation reuse rejection, content digests, and compact rank receipts. New receipt-aware v5 preflight reads O(world_size) compact metadata plus rank-local full state and fails closed when a receipt is missing.

• Shared CLI/server checkpoint implementation with rank-scoped save, same-topology restore, and v3/v4 plus pre-receipt v5 read compatibility.

• Transactional distributed server checkpoint save/load. Save holds one dispatch lock across all ranks, writes an exclusive sibling staging directory, validates the full receipt set through shadow-context hydration, and publishes only a fresh destination with RENAME_NOREPLACE. Load prepares communicator-attached shadow contexts without temporary-parameter broadcasts, then swaps live contexts only after every rank succeeds; partial commit fail-stops the worker group.

• Dynamic checkpoint restore validates each adapter's rank, alpha, target slots, tensor layout, Adam state, and private clock independently. Heterogeneous signatures may coexist after hydration; legacy training fails closed before forward unless selected adapters have homogeneous rank and canonical target sets.

• Dynamic Adam now preflights every active tenant and tensor before one out-of-place fused launch. Persistent per-adapter shadow buffers eliminate per-step allocation; live BF16 parameters, FP32 m/v, and private clocks swap only after launch validation. Production remains stream-asynchronous, while strict/test mode waits for completion and supports failure injection before commit.

• ABI23 adds live heterogeneous adapter registration and deterministic selected-tenant grouping by rank, canonical active slots, and tensor geometry. Tensor and borrowed host-i64 all-live/selected paths use v2. A later group failure rolls earlier parameter, FP32 m/v, and tenant-clock commits back through the persistent Adam shadows; per-group distributed success consensus prevents ranks whose homogeneous trainer returned from entering different subsequent groups.

• Deterministic fixed or tenant-specific dynamic adapter merge into standard PEFT, including correct [gate_all|up_all] reconstruction and dynamic optimizer clocks.

• Distributed CLI export with shared run identity, rank-isolated logs, unique attempt fencing, all-rank error reporting, completion markers, and atomic final directory publication.

• Fixed-LoRA CLI --resume-from restores projection-aware LoRA shards, CPU safetensors Adam m/v into CUDA/FP32 allocations, loss, and the native optimizer step after validating topology, adapter identity, and the canonical base-model path.

• Multi-node launch validation requires shared run/attempt IDs and shared output storage before workers start.

• EP IPC uses one absolute command deadline. An incomplete semaphore generation poisons the channel; the first IPC failure terminates and reaps the worker group, health changes to 503, and later dispatches fail immediately. Partial launch and graceful shutdown also have bounded worker cleanup.

• ABI22 adds a 64-byte aligned binary tensor slab between the HTTP parent and workers. The parent decodes and packs once; workers borrow validated int64 spans without per-worker vector deserialization, select topology-local rows, and enter one coarse C++ host-batch ABI for blocking H2D plus the full train/eval step. Legacy vector commands remain compatible.

• Tensor HTTP routes use a pre-extractor single-permit admission gate. This matches the already serialized IPC dispatch and prevents concurrent base64/decoded/slab buffers from multiplying peak memory while leaving health, checkpoint, and session control routes ungated.

• The EP worker group owns one explicit base-model session, clears rank-local state on delete, preserves independent dynamic adapters within that session, and returns the native logical training step through HTTP and benchmark paths.

• ABI24 separates heterogeneous signature forward/backward harvesting from one full-registry finalizer. All signature groups now share one token-normalization preflight, gradient synchronization boundary, out-of-place fused Adam transaction, and commit; fast-path finalization avoids activation chunk planning, registry metadata copies, and duplicate token-validity device fences.

Verification

• Local IPC tests passed 13/13; focused server tensor shape/slab tests passed 5/5; cargo check -p rustrain-server, the worker hot-path scan, and git diff --check passed. Existing repository warnings remain.

• Independent final review found no blocker/high issue. Its medium unbounded-request-memory finding was fixed with pre-decode tensor-route admission and re-reviewed successfully; remaining low risks are startup-only shm cleanup/naming hardening and missing full HTTP-to-worker multi-process coverage.

• H20 target: ABI22 single-GPU native smoke passed host/tensor eval loss parity with zero difference, fixed-LoRA training with zero Adam-oracle error, and selected dynamic multi-LoRA isolation with no unselected update.

• H20 target: transactional dynamic Adam failure smoke executed the shadow update, injected failure before commit, and preserved the selected LoRA B, FP32 m/v, and tenant clock exactly. Consecutive successful steps covered persistent shadow reuse and autograd leaf preservation.

• H20 target: heterogeneous restore smoke rejected ordinary mixed-signature registration, hydrated a rank-9 q_proj adapter beside rank-8 expert adapters, and rejected same-rank/different-target legacy training before forward while preserving existing tenant clocks.

• H20 target: ABI22 world8 TP2 x EP2 x DP2 tri-smoke passed checkpoint resume, selected adapter isolation, and Adam oracles across ranks.

• H20 target: ABI23 single-GPU smoke passed live mixed-signature registration, duplicate/unknown-ID registry preservation, tensor and all-live host-i64 v2 dispatch, successful mixed rank/target training, and exact cross-group rollback.

• H20 target: ABI23 world8 TP2 x EP2 x DP2 trained a rank-3 q_proj tenant with rank-4 multi-module tenants on every rank. Rank-0-only wrapper and pre-commit Adam failures reached all-rank consensus, restored parameters and clocks exactly, and the same contexts subsequently completed a successful heterogeneous step.

• H20 target: TP2 x EP2 fixed/dynamic native smoke passed routed and shared expert LoRA, tenant isolation, FP32 Adam-state oracles, and checkpoint/resume.

• H20 target: world8 TP2 x EP2 x DP2 passed both source-sharded and replicated-source native policies. Fixed updates reported zero Adam-oracle error on every rank; fixed resume restored parameters and Adam m/v with exact zero divergence and optimizer step 2.

• H20 target: dynamic multi-LoRA preserved tenant isolation and clocks [1, 0], matched the full-reference Adam oracle, and completed a second resume/update cycle.

• H20 target: checkpoint v5 TP2 x EP2 fixed merge passed save, fresh per-rank restore, and fused expert ordering; dynamic merge preserved tenant identity and optimizer clock.

• H20 target: ABI21 source-sharded world8 shadow resume passed all ranks. DP-nonzero fixed tensors were deliberately diverged before no-sync communicator attach and remained bitwise unchanged after attach; hydrate restored fixed/dynamic A/B, FP32 Adam m/v, and tenant step exactly, and continued training retained zero resume divergence. The ABI version, no-sync attach, and restore-adapter symbols are exported by the native library.

• ABI20 regressions passed pure EP2, TP2 x DP2, and GDN TP2.

• On the matched TP2 x EP2 fixed-LoRA workload, expert tensor parallelism reduced p50 from 8.742 ms to 6.923 ms and increased unique-token throughput from 58.57k/s to 73.95k/s.

• The H20 target does not have a Rust toolchain, so the Rust HTTP/IPC tests ran locally against the repository ABI1 environment; remote post-integration evidence covers the native host ABI and world8 oracle, not a full HTTP/shared-slab server run.

• H20 target: ABI24 single-GPU smoke proved two heterogeneous signatures trigger exactly one finalizer and one fused Adam launch. Group failure preserved counters and all fixed-accumulation flags/weights; all-live host-i64 dispatch used the same unified path.

• H20 target: ABI24 world8 TP2 x EP2 x DP2 passed rank-0-only group, pre-token-preflight, and pre-commit Adam failure injection without collective phase mismatch. Failure paths preserved parameters, Adam state, clocks, and pending accumulation; the same contexts then completed one-finalizer/one-Adam heterogeneous training.

Assessment and limitations

The verified subset has direct numerical and performance evidence, but this branch does not establish correctness or high performance for every Qwen 3.5/3.6 configuration. It has not completed real 35B/Qwen3.6 long-running training, real cross-node training, PP/CP, cross-topology resharding, or a matched Megatron benchmark. Base-model resume identity is canonical-path based rather than content-fingerprint based. Distributed server save currently requires a fresh destination and provides atomic visibility, but it does not yet provide immutable generation/LATEST rollover or power-loss fsync durability.

The server's parent/worker IPC is now binary, but HTTP ingress remains JSON/base64 and pageable host memory. Raw binary HTTP, pinned staging, asynchronous H2D/request overlap, and full remote HTTP benchmarking remain open. Heterogeneous selected training now shares one synchronization and Adam boundary, but each distinct rank/target signature still requires its own forward/backward harvest. Gradient synchronization remains a sequence of per-tensor collectives rather than a bucketed/grouped launch, and group/finalizer success consensus reads a GPU flag to CPU. The phase protocol covers tested rank-local failures before token preflight and before Adam commit; it is not recovery from an unrecoverable CUDA/NCCL failure midway through the multi-collective gradient synchronization region. Persistent shadows cost about 10 additional bytes per active LoRA parameter, so large-tenant capacity and throughput still need a matched benchmark.

Compared with Megatron-LM, rustrain still lacks sequence parallelism, fused permutation/inverse permutation, host-free split planning, communication/GEMM overlap, shared-expert overlap, PP/CP, fused QKV/gate-up, and a verified prebuilt DeepEP/TE/FLA integration. The implemented path uses the existing prebuilt PyTorch ABI1 and NCCL runtime; it introduces no Python JIT or source-built dependency.

Latest validated additions

• Added guarded TP2 dense fixed-LoRA sequence parallelism with embedding sequence scatter, column-path all-gather, row-path reduce-scatter, and loss gather. H20 two-rank smoke passed equal loss, local sequence ownership, collective counters, and a nonzero LoRA update. MoE, dynamic adapters, latent-rank LoRA, MTP, auxiliary loss, and CP/EP/DP/PP combinations remain fail-closed.
• Added a restricted PP2 fixed-LoRA non-interleaved 1F1B window with stage-local model ownership and grouped bidirectional activation/gradient P2P. Every forward microbatch performs cross-stage shape/dtype consensus before P2P. H20 pp-train-smoke passed loss/gradient/parameter parity and rejected a rank-divergent target shape collectively.
• Added selected dynamic multi-LoRA evaluation, per-tenant optimizer-step guards, and transactional recovery quarantine. A secondary recovery failure poisons the native context; TP/EP/DP health consensus propagates the quarantine to the worker group. Rust maps the optional ABI28 health symbol into a terminal EP result, so the parent coordinator becomes unavailable and reaps the group. H20 single-rank and world4 TP2 x EP2 poison smokes passed without collective hangs; local server tests passed 43/43 and IPC tests passed 18/18.

The remaining gap is narrower but still material: arbitrary/interleaved PP, PP-aware checkpoint placement, CP model execution, dynamic LoRA combined with PP/CP or sequence parallelism, fused/overlapped EP and dense kernels, raw/pinned asynchronous ingress, long-running real-model validation, and a matched Megatron throughput benchmark are not complete.

[NolanHo]

Follow-up validation (commit ae75152): native ABI bumped to 7 with qwen36_train_micro_step; Rust gradient_accumulation_steps now performs scaled micro-backward calls and applies NCCL/Adam only on the final micro-step. DP replicated LoRA reduction is token-count weighted; EP skips the DP reduction path. Multi-LoRA CE uses per-tenant token means and all activation chunks accumulate gradients before one logical Adam update. Local cargo check -p rustrain-qwen3-6 -p rustrain-server, cargo test -p rustrain-core --lib (8), and Qwen integration tests (6) pass; remote H20 C++ syntax compilation passes. Remaining parity gaps are explicit: no TP/CP/PP pipeline parallelism, no FP32 gradient accumulator/abort API for interrupted windows, and no full-model Qwen training benchmark.

[NolanHo]

Additional H20 validation after f65af15: rebuilt the native library with CUDA 13 headers plus fused CUDA objects and ran native_smoke under system PyTorch 2.5.1 ABI0. It passed MoE LoRA parity, dynamic multi-LoRA, dense/linear Qwen3.5 paths, and the new micro-step assertion (non-final backward leaves step/params unchanged; final backward increments step and updates params). This is ABI0 smoke evidence only; the project ABI1 PyTorch runtime was not mounted in this session.

[NolanHo]

最新增量：

• 新增 Megatron 风格 5D TP/PP/DP/EP/CP topology helper、rank/group/layer-range contract 和 launcher 环境规范化；这是拓扑基础，不代表 Qwen native 已启用 TP/PP/CP。
• ABI 升至 8，新增 qwen36_set_step_count；checkpoint load 现在恢复 C++ Adam bias-correction step。
• 验证：cargo test -p rustrain-parallel --lib 14/14；cargo test -p rustrain-core --lib 8/8；LD_LIBRARY_PATH=... cargo test -p rustrain-server --lib 2/2；Qwen integration 6/6；H20 ABI8 native_smoke 通过。
• 仍明确未完成：Qwen native TP/PP/CP、DeepEP/异步 A2A、FP32 accumulation、每 adapter 独立 optimizer step、rank-sharded topology checkpoint，以及 Megatron 同条件端到端吞吐对比。

[NolanHo]

ABI10 TP-only verification on H20 (123.57.26.97:28004): rebuilt C++/CUDA against PyTorch 2.5.1 ABI0, ran both RANK=0/1 with WORLD_SIZE=TP_SIZE=2, and both exited 0. Smoke covered MoE grouped/fallback, dense MLP, GDN/linear attention, dynamic multi-LoRA, selected-tenant isolation, and context reuse. Global LoRA rank 8 used distinct rank-4 slices; losses matched and each shard updated. This validates LoRA latent-rank TP only; frozen base weights remain replicated.

[NolanHo]

ABI11 evidence update (H20 123.57.26.97:28004):

• Single-rank native build + smoke passed FP32 accumulator storage, two-micro window, NaN/explicit abort cleanup, successful step commit, unsupported topology rejection, and dynamic-DP per-tenant batch rejection.
• Two-rank TP smoke passed on both ranks for MoE, dense MLP, GDN/linear attention, dynamic multi-LoRA, selected-tenant isolation, and the new guards (rank_statuses=0,0).
• cargo check -p rustrain-qwen3-6 -p rustrain-server -p rustrain-ipc passed; cargo test -p rustrain-server --lib passed 6/6.
• Same-topology TP checkpoint v3 tests cover v1/v2 compatibility, rank paths, topology mismatch, rank-shard mismatch, and fixed/dynamic metadata roundtrip.

The branch still explicitly does not claim full Megatron parity: base weights are replicated, PP/CP/multi-axis groups and variable-split EP token A2A are absent, prebuilt TE/DeepEP/flash-attn were unavailable on the target runtime, and no matched throughput benchmark has been run.

[NolanHo]

ABI11 DP/EP and optimizer verification update

Pushed b7050ea plus documentation d6f897c.

• H20 single-rank native smoke is green, including a standard Adam first-step parameter oracle. The test caught and the patch fixed the missing sqrt(1-beta2^t) factor in both fixed and dynamic fused Adam paths.
• H20 TP2 native smoke is green (rank_statuses=0,0), including repeated dynamic multi-LoRA invocations. The n_max rendezvous is invocation-scoped and atomically published.
• H20 DP2 native smoke is green (rank_statuses=0,0) for tenant token counts [1,3] and [3,1]: weighted m relative error 2.43e-8, grouped-expert error 2.27e-8, v error 7.33e-8, BF16 Adam delta error 0, and 7.96e-3 gap versus the old equal-count formula.
• H20 EP2 full-expert parity is green (rank_statuses=0,0) with distinct global expert base/LoRA slices. Loss, LoRA A/B updates, Adam m/v, and standard Adam first-step oracle all match the rank-local full-expert reference with zero reported difference.

The PR still intentionally does not claim Megatron parity: frozen base weights are replicated; PP/CP, multi-axis TP+DP/EP, variable-split token A2A, DeepEP/TE integration, cross-topology resharding, and matched throughput benchmarks remain open.

[NolanHo]

ABI11 DP/EP and optimizer verification update

Pushed b7050ea plus documentation d6f897c.

• H20 single-rank native smoke is green, including a standard Adam first-step parameter oracle. The test caught and the patch fixed the missing sqrt(1-beta2^t) factor in both fixed and dynamic fused Adam paths.
• H20 TP2 native smoke is green (rank_statuses=0,0), including repeated dynamic multi-LoRA invocations. The n_max rendezvous is invocation-scoped and atomically published.
• H20 DP2 native smoke is green (rank_statuses=0,0) for tenant token counts [1,3] and [3,1]: weighted m relative error 2.43e-8, grouped-expert error 2.27e-8, v error 7.33e-8, BF16 Adam delta error 0, and 7.96e-3 gap versus the old equal-count formula.
• H20 EP2 full-expert parity is green (rank_statuses=0,0) with distinct global expert base/LoRA slices. Loss, LoRA A/B updates, Adam m/v, and standard Adam first-step oracle all match the rank-local full-expert reference with zero reported difference.
• H20 gated variable-split A2A is green in legacy, replicated-source, and sharded-source modes (rank_statuses=0,0). Sharded mode uses disjoint source rows and [1,3] target-token counts; weighted global loss differs from the batch-2 full reference by about 9.6e-7, expert m/v maxima are 1.22e-5 / 3.92e-9, and standard Adam oracles are zero. The one 9.61e-4 parameter difference is an adjacent BF16 rounding bin, bounded separately.

The PR still intentionally does not claim Megatron parity: frozen base weights are replicated; PP/CP, multi-axis TP+DP/EP, dynamic-tenant A2A metadata, DeepEP/TE integration, cross-topology resharding, and matched throughput benchmarks remain open. The target host has no importable Megatron/Transformer Engine/DeepEP/flash-attn prebuilt packages, so no dependency installation or JIT workaround was used.

[NolanHo]

Native EP benchmark update

Fresh H20 ABI0 two-rank run of native_ep_bench.cpp passed both modes (rank_statuses=0,0) with seq=128, hidden=256, experts=8, intermediate=256, warmup=2, iters=10:

• Legacy EP: median 5.4752 ms / 5.4672 ms per rank; 46.4k processed tokens/s, 23.2k unique tokens/s.
• Sharded variable-split A2A: median 6.7192 ms / 6.7261 ms; 37.8k processed and unique tokens/s.

This is a synthetic native baseline, not a matched Megatron-LM/DeepEP/Transformer Engine benchmark. The result does not show an end-to-end A2A step-time win on this small workload; the main remaining gaps are GPU-only split planning, fused permutation, overlap, full Qwen3.6 shapes, and dynamic-tenant source metadata.

[NolanHo]

Latest validation (commit 3431c8b):

• Added opt-in QWEN36_FUSED_LORA_QKV_A=1: compatible Q/K/V LoRA adapters share one A-side batched matmul; heterogeneous layouts/ranks fall back, and the flag is in the distributed runtime hash.
• H20 TP2 smoke passed legacy/fused eval A/B, fixed/dynamic oracle parity, and rank-local flag mismatch fail-closed.
• H20 synthetic A/B is neutral-to-small and noisy (B=2,S=128 p50 9.354 -> 9.291 ms; larger repeated runs had no stable win), so the flag remains off by default.
• Dependency audit found no exact Torch 2.12.1+cu130 ABI1 prebuilt TE/FlashAttention/FLA/DeepEP artifact satisfying the no-JIT/no-build constraint; matched Megatron-LoRA throughput remains open.

The draft remains intentionally open because arbitrary PP/CP, true communication overlap, exact prebuilt Megatron kernels, and matched full-model throughput are not complete.

[NolanHo]

本轮更新（commit 7692e3e）：

• 为动态 LoRA 增加持久排序 adapter_id -> canonical slot 索引；distributed validator、selected-v2、expected-step、selected-eval 走 binary lookup。临时 selected/chunk registry detach 时索引显式失效，恢复 canonical vector 后恢复；add/rename/remove/checkpoint restore 同步维护。
• H20 ABI1（Torch 2.12.1+cu130, ABI1, H20 SM90）真实 native 编译通过。
• ./scripts/run_qwen36_native_tp_ep.sh local-smoke：exit 0。
• ./scripts/run_qwen36_native_tp_ep.sh smoke：4-rank TP2 x EP2，exit 0；包含 selected-v2、rollback/poison、collective fail-closed 负测。
• synthetic dynamic-MoE benchmark（TP2 x EP2, B=8,S=128,H=1024,I=2048, rank16, q/expert targets, 50 iters）：8 registered/8 active p50/p95 13.859/17.561 ms；1024 registered/8 active 14.390/19.579 ms，p50 增加约 3.8%。代价是 allocator reserved 约 1.04 -> 9.46 GiB/GPU；这是 registry scaling 证据，不是完整模型吞吐结论。

仍保持已知边界：完整 Megatron matched throughput、native DeepEP C ABI、TE/FLA 无 JIT 接入、任意 PP/CP/ring attention 尚未完成。

[NolanHo]

追加 MoE A2A 更新（commit 282fd4a）：

• token ID 与 expert ID 合并为一个 [rows,2] int64 metadata 消息；每个 peer 的 dispatch 从 hidden/token/expert 三个 NCCL 消息降为 hidden/metadata 两个。
• QWEN36_EP_A2A_PACKED_METADATA=0 可回退旧协议；flag 已纳入 distributed runtime hash，避免 rank 间协议分叉。
• H20 native 编译、local smoke、TP2 x EP2 distributed smoke 均 exit 0；split 与 packed 两种协议均完成 forward/backward benchmark。
• matched synthetic dynamic-MoE（B=8,S=128,H=1024,I=2048, 8 tenants, 50 iters）：split/packed p50 14.155/14.051 ms，p95 16.849/16.016 ms。p50 约 0.7%，按 noise-scale 结果处理，不宣称稳定吞吐提升。

该改动减少 NCCL enqueue，但 variable-count planning 仍需 host-visible counts；与 DeepEP 的 GPU-only layout/fused permutation/overlap 仍有实质差距。

[NolanHo]

追加 commit 140cdc8（已推送）。

• 默认 QWEN36_LAZY_DYNAMIC_OPTIMIZER_STATE=1：冷租户只保留 live LoRA + FP32 grad slab；首次有全局 token 才 materialize Adam m/v 与 rollback shadow。step-0 checkpoint 使用 optimizer_count=0，旧 full-state step-0 仍兼容；state getter/load 不分配 shadow。
• H20 TP2 x EP2 动态 MoE A/B（B=8,S=128,H=1024,I=2048, rank=16, 50 iters）：8 registered/8 active reserved 1.035 GiB/GPU；1024 registered/8 active 2.918-3.148 GiB/GPU，此前 full-residency 9.46 GiB/GPU。p50 14.017/14.058 ms，为噪声范围。
• H20 ABI1 local/distributed native smoke 退出 0；cargo check -p rustrain-qwen3-6 --lib 与 cargo check -p rustrain-server --lib 退出 0。
• GDN token-loop barrier 微优化：TP2 smoke/chunkwise/parity 退出 0；默认无 checkpoint p50 baseline/optimized 102.62/102.34 ms，stride32 replay 106.26/106.73 ms，只作为低风险 codegen 优化，不宣称稳定多百分比收益。

[NolanHo]

A2A prefix-plan optimization pushed in baedae0. After count all-gather, dispatch/combine autograd now reuses one CPU int64 send/receive prefix plan; removed repeated count-to-CPU, memcpy, vector allocation, and prefix scans without changing NCCL message order/layout. H20 validation: cargo check, local-smoke, and distributed smoke all exit 0; MoE parity diff=0, fixed Adam oracle error=0, heterogeneous v2 all four ranks ok. Matched TP2xEP2 dynamic B8/S128/H1024 A/B: baseline mean/p50/p95 13.974047/13.792658/15.242697 ms; prefix 13.977255/14.008014/14.735662 ms; memory unchanged. This is deterministic host-bookkeeping cleanup, not a stable EP2 end-to-end speedup.

[NolanHo]

Pushed 330b470 (perf: fuse packed MoE weighted unpermute).

This adds opt-in QWEN36_MOE_FUSED_UNPERMUTE=1 for packed EP A2A. CUDA reconstructs the original [token, top_k] assignment order and performs the weighted merge directly; custom autograd emits gradients for both returned activations and routing weights. The runtime hash includes the flag, and legacy/non-packed behavior is unchanged. The flag remains disabled by default.

H20 ABI1 verification:

• cargo check -p rustrain-qwen3-6 --lib: exit 0
• local native smoke: exit 0; MoE fallback/grouped diff 0 and fixed Adam oracle error 0
• TP2 x EP2 distributed smoke: exit 0; fixed Adam error 0 on all ranks; resume parameter/m/v diff 0 at step 2; dynamic isolation diff 0 and Adam error 0; heterogeneous-v2 and recovery-poison passed on all ranks
• matched dynamic B8/S128/H1024, 10 warmup/50 iterations: fused mean/p50/p95 14.063/13.963/15.248 ms; fallback 14.154/14.132/15.223 ms; resident/reserved memory unchanged

The roughly 0.6% mean difference is noise-scale, so this is documented as an optional launch/temporary reduction rather than a stable throughput claim. It does not close the remaining DeepEP/GPU-only metadata/communication-overlap gap.

[NolanHo]

Pushed 11343a1 (perf: pool dynamic Adam transaction shadows).

This addresses the dynamic multi-LoRA rotation residency issue. QWEN36_DYNAMIC_ADAM_SHADOW_POOL=1 (default, included in the distributed runtime hash) keeps tenant-owned Adam m/v persistent but leases the six-tensor out-of-place rollback destination from a context-level pool only for active transactions. The selected-v2 finalizer retains leases through the outer rank consensus, then releases after success or rollback; ordinary/pipeline finalizers release on scope exit. Exact layouts reuse idle slots, and a free slot is reshaped before the pool grows, so mixed rank/target layouts are bounded by simultaneous active pairs. QWEN36_DYNAMIC_ADAM_SHADOW_POOL=0 preserves the old behavior.

H20 ABI1 evidence:

• cargo check -p rustrain-qwen3-6 --lib: exit 0
• pool-enabled local and TP2 x EP2 distributed smoke: exit 0; fixed Adam error 0, resume parameter/m/v diff 0 at step 2, dynamic isolation diff 0 and Adam error 0, heterogeneous-v2 all ranks ok, recovery-poison all ranks ok
• pool-disabled local smoke: fixed Adam error 0, transactional failure/heterogeneous rollback/recovery all ok
• 1024 registered/8 active, B8/S128/H1024, TP2 x EP2, complete 128-step rotation: old path allocated/reserved/max-resident 8.439/8.813, 9.029, 11.363 GiB; pooled 5.028/5.447, 5.637, 7.971 GiB; p50/p95 21.179/32.224 -> 16.752/26.057 ms
• second complete 128-step rotation: allocated remained 5.028 GiB (reserved 5.879 GiB), showing no growth with historical tenant activation

Persistent Adam m/v still grows for tenants that actually train; complete CPU/GPU optimizer-state paging/offload remains a later requirement for very large tenant fleets.

[NolanHo]

H20 ABI1 verification for commit 04befa4:

• QWEN36_MOE_FUSED_LOCAL_PERMUTE=1 local smoke and TP2xEP2 distributed smoke passed; fixed Adam error 0, resume parameter/m/v diff 0, heterogeneous-v2 ok, recovery-poison ok.
• cargo check -p rustrain-qwen3-6 --lib passed.
• Matched dynamic B=8,S=128,H=1024,I=2048, rank 16, packed A2A plus fused weighted-unpermute, 10 warmup/50 iterations: fused mean/p50/p95 13.342/13.156/15.808 ms; fallback 13.956/13.846/14.792 ms. Resident memory was unchanged within allocator noise.
• The mean/p50 gain is useful but p95 was not stable in this run, so the flag remains opt-in/default-off. It removes local host-side permutation work but does not implement Megatron/DeepEP GPU-only dispatch or communication-compute overlap.

The audit/progress docs record the remaining Megatron gaps, including CPU optimizer-state paging, interleaved PP, general/ring CP, DeepEP/TE integration, request/compute overlap, and real-model long-run throughput.

[NolanHo]

追加两项已验证改进：

• 43e46e6：新增 opt-in QWEN36_EP_A2A_COUNT_OVERLAP=1，独立 metadata stream 将 count all-gather/compact D2H 与 token packing 重叠。H20 TP2xEP2 local/distributed smoke 通过。matched dynamic B8/S128 mean on/off 13.625/13.807 ms；更大 B8/S512 为 52.758/52.772 ms，因此保持默认关闭，不宣称稳定端到端加速。
• 4a501a8：ABI30 dynamic Adam CPU checkpoint snapshot。首次导出完整 FP32 m/v 到 host 并按 optimizer clock 复用，server save 不再通过 getter 重新 materialize 全部 GPU state；restore 暂不改变。H20 ABI1 qwen/server cargo check、local smoke、TP2xEP2 distributed smoke、step0/step1、重复 snapshot、resume parity 均通过。

H20 依赖核查确认当前 PyTorch 2.12.1+cu130 ABI1 环境无匹配 TE/DeepEP/FLA/standalone flash-attn 预编译包；PyTorch bundled Flash-SDPA 可用。未安装依赖、未引入 JIT。文档已明确剩余 Megatron 差距。

[NolanHo]

ABI31 后续切片已完成并推送（fac291a）：dynamic Adam restore 改为一次性 bulk host import。完整 stage-local CPU FP32 m/v 先校验后原子发布，restore 后 GPU resident count=0；首次选中 tenant 才按 pair hydrate。native smoke 覆盖 partial import 原子失败、CPU export 不触发 hydrate、首次训练 resident=active_pairs*4、loss/参数/m/v resume parity。H20 ABI1 cargo check、local smoke、TP2xEP2 distributed smoke 均通过；磁盘 checkpoint 格式不变，仍未实现训练后 LRU eviction/完整 optimizer paging。

[NolanHo]

ABI32/host-paging follow-up is pushed on audit/qwen35-qwen36-megatron (6c160bb, 73ba2a4, a8f1909).

• Added opt-in QWEN36_DYNAMIC_ADAM_HOST_PAGING=1 with QWEN36_DYNAMIC_ADAM_RESIDENT_BYTES budget and LRU eviction of cold, unselected tenant Adam m/v to the existing host-authoritative snapshot.
• Added read-only qwen36_get_adapter_optimizer_resident_bytes_v1; paging is post-commit/best-effort, default-off, and failures retain resident state without changing a committed result.
• H20 ABI1 TP2xEP2 smoke passed the budget=0 eviction, CPU snapshot/no hydrate, first-use hydrate, and resident-oracle exact loss/parameter/m/v parity. Existing heterogeneous-v2 and poison-recovery checks also passed. H20 local smoke passed Qwen3.5/Qwen3.6 MoE, GDN, LoRA and Adam oracles.
• H20 Rust ABI1 cargo check -p rustrain-qwen3-6 --lib -p rustrain-server passed with the project Python 3.12/PyTorch 2.12.1+cu130/ABI1 environment (warnings only).

This is bounded native selected-path host paging, not yet Megatron distributed optimizer/offload, asynchronous prefetch, PP-window eviction, or a matched full-model throughput result.

[NolanHo]

Added and pushed commit 0305f94 for arbitrary-CP dense full-attention ring attention.

• QWEN36_CP_FULL_ATTENTION_RING=1: NCCL P2P KV rotation, FP32 online softmax, custom backward with owner-return dK/dV accumulators.
• CP2 and CP4 H20 ABI1 ring smokes pass against CP1 oracle, including LoRA q/k/v/o gradients, Adam m/v/params, padding, and fail-closed rank flag mismatch.
• Legacy CP2 KV gather remains mutually exclusive oracle path. Qwen3.6 hybrid GDN remains CP2-only; ring is correctness/memory milestone, not Megatron-throughput claim.
• H20 ABI1 cargo check -p rustrain-qwen3-6 --lib -p rustrain-server passes.