FP16 matmul kernel is 2-18x slower than FP32; Fp16Codec.decode dominates the inner loop

[michalharakal]

Summary

PanamaVectorFp16MatmulKernel runs at ~0.5 GFLOP/s regardless of matrix shape or batch size, making FP16 KEEP_NATIVE 2-18x slower than the FP32 SGEMM it replaces. The sibling PanamaVectorBf16MatmulKernel, structurally identical, is 1.5-2.1x faster than FP32.

The difference is the per-element decode, not the kernel structure or the dispatch.

Follow-up to #884 / #886.

Measurements

Intel i7-9750H (AVX2, no AVX-512), 12 threads, OpenJDK 21.0.11, engine at ddad83a. Median ms per call, weight already in [in, out] so chooseQuantizedMatmul dispatches to the narrow kernel:

shape (in x out)	batch	fp32	fp16	bf16
2048 x 2048	1	3.210	18.601	1.591
2048 x 2048	16	16.449	297.873	11.125
4096 x 4096	1	40.335	73.793	21.472
4096 x 4096	16	95.954	1166.224	66.227
4096 x 11008	1	111.922	197.337	60.372
4096 x 11008	16	272.889	3146.712	186.360
11008 x 4096	1	109.242	198.977	57.749
11008 x 4096	16	274.547	3172.639	177.142

FP16 holding ~0.5 GFLOP/s flat while work scales 16x with batch is the signature of being compute-bound on a scalar operation in the inner loop, not memory-bound.

Root cause

Both kernels fill a scratch lane array scalar-wise before the vector FMA. PanamaVectorBf16MatmulKernel:

scratch[lane] = Float.fromBits(((hi shl 8) or lo) shl 16)

Three integer ops, branch-free, trivially inlined.

PanamaVectorFp16MatmulKernel (skainet-backends/skainet-backend-cpu/src/jvmMain/kotlin/sk/ainet/exec/kernel/PanamaVectorFp16MatmulKernel.kt):

scratch[lane] = Fp16Codec.decode((hi shl 8) or lo)

Fp16Codec.decode (skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/types/NarrowFloatCodec.kt) is a three-way when on the exponent, and the subnormal arm contains a data-dependent do/while renormalization loop. In a hot inner loop executed once per weight element, that is branchy and unpredictable.

The codec itself is correct and fine for general use — the problem is only its cost on this path.

Suggested fix

On the JVM, Float.float16ToFloat(short) has been available since JDK 20 and is an intrinsic that lowers to vcvtph2ps on hardware with F16C. The kernel is already JVM-only (jvmMain) and already requires JDK 21 via isJdk21Plus(), so it can call it directly without touching the shared codec:

scratch[lane] = Float.float16ToFloat((((hi shl 8) or lo).toShort()))

Better still would be converting a whole ShortVector per iteration instead of filling scratch lane by lane, which would remove the scalar loop entirely — but the intrinsic alone should close most of the gap.

A branch-free fallback for the common-path decode in Fp16Codec would also help non-JVM targets, where the same decode is used by ScalarFp16MatmulKernel.

Impact

Until this is addressed, FP16 KEEP_NATIVE is a performance regression rather than an optimization on this hardware class, while BF16 KEEP_NATIVE delivers the expected bandwidth win. Consumers choosing a narrow format for speed should currently prefer BF16.

Reproducing

NarrowFloatMatmulBenchmark in SKaiNET-transformers (llm-inference/llama/src/jvmTest, branch chore/validate-engine-0.38.0-snapshot):

./gradlew :llm-inference:llama:jvmTest --tests "*NarrowFloatMatmulBenchmark*" \
  -PuseMavenLocalSkainet=true -Dskainet.bench.narrow=true --rerun-tasks -i

It asserts chooseQuantizedMatmul's preconditions before timing, so the numbers are the kernel and not a silent fallback to the generic path.

[michalharakal]

Related: #888 — narrow-float weights never reach this kernel in practice, because transpose has no NarrowFloatTensorData arm and widens them elementwise first. The two are independent fixes but both are needed for FP16 KEEP_NATIVE to be worth enabling.