metal: stop expert-miss readahead racing the pread pool (+13% GLM streaming decode)

[jasontitus]

During steady-state SSD-streaming decode the eval thread spends much of its
time in fcntl(F_RDADVISE): every expert-cache miss advises the gate/up/down
ranges and then immediately hands the exact same ranges to the pread pool.
The advise fcntl is synchronous on the eval thread, while the pool reads the
bytes in parallel either way, so the advise only adds stall time. A 5 s
thread sample during GLM 5.2 streaming decode (the same three call sites
exist on the glm5.2 branch):

eval thread	share
fcntl(F_RDADVISE) in readahead_range (miss paths)	40%
GPU waitUntilCompleted	44%
pread pool wait	11%
CPU	5%

Commits

1. metal: skip streaming expert miss readahead by default — gates the
   three miss-path advise sites (single miss, batch load, early load).
   DS4_METAL_ENABLE_STREAMING_EXPERT_MISS_READAHEAD=1 restores the old
   behavior. The prefill/layer readahead sites are untouched.
2. metal: report pread pool dispatch stats in the timing summary — one
   extra line under the existing DS4_METAL_STREAMING_EXPERT_TIMING_SUMMARY
   gate: dispatches, tasks, bytes, effective queue depth (sum of per-task
   pread ms / pool wall ms) and delivered bandwidth. This is how the numbers
   below were measured; makes streaming IO regressions visible without
   dtrace.
3. metal: keep expert pread threads on the important IO tier — reader
   threads inherit the parent QoS, so under taskpolicy -b / background
   launchd jobs every miss pays the throttled IO tier. Pin them to
   user-initiated QoS + IOPOL_IMPORTANT
   (DS4_METAL_DISABLE_STREAMING_EXPERT_PREAD_QOS opts out).

Measurements

One machine only: M1 Ultra Mac Studio, 128 GB RAM, 8 TB SSD (~7.2 GB/s
sequential), Metal backend.

GLM-5.2-UD-IQ2_XXS (rebased onto glm5.2, where the change applies
clean; --ssd-streaming --ssd-streaming-cache-experts 48GB --ssd-streaming-full-layers 16, prefill 2048 + 128 greedy):

	prefill t/s	steady decode t/s
glm5.2	20.9	2.34
+ this change	26.7 (+27%)	2.74 (+13%)

DeepSeek-V4-Flash Q4KExperts imatrix streaming on this branch
(--ssd-streaming-cache-experts 48GB, prefill 512 + 128 greedy): no
measurable difference (5.01 vs 5.21 gen t/s, inside the run-to-run spread —
two back-to-back pristine runs varied 2.66 vs 3.11 on the shorter form).
Expected: only ~25 GB of this quant misses RAM, so the miss rate — and the
advise tax — is small. The win concentrates on models far bigger than RAM.

Flash IQ2XXS imatrix resident (no --ssd-streaming): bit-identical
logits and identical speed (25.7 t/s decode) — the changed code never runs
outside streaming mode.

Correctness: every A/B pair above was run with
--dump-frontier-logits-dir and the full frontier logits compared
byte-for-byte — identical in all cases (the streaming baseline itself is
deterministic: two pristine runs match bit-for-bit).

Also tried while measuring: splitting each expert read into 2-8 striped
preads to raise queue depth. It lifts large preload dispatches
(5.1 -> 6.8 GB/s) but does not help decode misses, so it is not included.

Tests run

make clean && make            # zero warnings
./ds4_test --server           # OK
./ds4_test --metal-kernels    # OK
./ds4_test --logprob-vectors  # pre-existing ERR (long_code_audit) with the
                              # IQ2XXS imatrix quant; identical on unpatched
                              # main, unrelated to this change

Speed/parity A/Bs with ds4-bench as above.

[andreaborio]

We ran the PR branch on the cell the writeup extrapolates to ("the win
concentrates on models far bigger than RAM"): DeepSeek on main with model
much bigger than RAM. On that cell the default flip goes the other way.

Box: M5 Pro 64 GB, macOS 26, Flash IQ2XXS (80.8 GiB file, cache budget 33.3
GiB), --ssd-streaming, temp 0. Paired runs alternating the arms back to
back, in both orders, on the PR branch as-is, so the QoS pinning is active
in both arms and only DS4_METAL_ENABLE_STREAMING_EXPERT_MISS_READAHEAD
differs. A 3 s thread sample during decode shows fcntl frames only in the
advise arm, so the gate does what it says.

Short chat prompt (11 tokens, -n 96), 6 pairs, expert traffic identical
across arms:

	prefill t/s	gen t/s
PR default (miss advise off)	3.79-3.92	11.7-12.4
advise restored	4.29-4.74	11.1-13.0

Generation is a wash, but time to first token (short-prompt prefill is
mostly streaming ~35 GiB of missed experts) is 13-22% better with the advise
on, 6/6 pairs, no overlap in either order. Long prefill (ds4-bench, ctx 2048

• 32 gen) points the same way, smaller: advise on wins prefill 5/5 pairs,
  +4% mean (0-7%), gen overlapping.

Your dispatch stats show why. Same bytes in both arms (35.52 GiB, 16167
tasks, 3167 dispatches), but advise off pool_gbps 10.8 task_gbps 2.2, advise
on pool_gbps 23 task_gbps 7.4. Each pread completes 3-4x faster when the
advise ran first, so here the advise buys real prefetch, presumably because
the kernel keeps more of an advised range in flight than a single
synchronous pread gets.

Why the stall you sampled does not show up here: the two branches have
opposite defaults for the async selected load. On main,
metal_graph_use_iq2_selected_async_load is on for every streaming run
(env is opt-out), so the miss advise mostly lands on the async worker: in
our sample ~95% of the fcntl frames sit on
metal_graph_selected_async_load_worker_main and fcntl is ~0.5% of the eval
thread. On glm5.2, glm_graph_use_streaming_selected_async_load is
opt-in, so by default the GLM advise is synchronous on the eval thread,
which is exactly where your 40% sample lives and why skipping it pays there.

So the miss advise is not the problem, issuing it on the eval thread is.
Suggestion: keep the advise default on main (or gate the skip on the
synchronous path only), and on the glm5.2 line fix it by moving the advise
off the eval thread (or enabling the async load) rather than dropping the
prefetch. Happy to re-run any variant on this box.

The dispatch stats are a great addition, they are what made this visible at
all. FWIW we measured the same QoS pinning independently on this box a few
days ago: neutral in foreground runs, which fits the taskpolicy/background
rationale.