Follow-up to #884, which added FP16 matmul at the scalar (0) and Panama Vector (50) tiers. BF16 additionally has a native FFM (100) kernel, so on any machine where the native provider wins, FP16 silently runs a tier slower than BF16.
The asymmetry is correct but easy to misread: benchmarking the two formats today compares a Panama kernel against a native one, not FP16 against BF16.
Scope
Notes for whoever picks this up
The SPI is already in place — Fp16MatmulKernel : NarrowFloatMatmulKernel with the same signature as the BF16 kernel, including the asymmetric stride convention (float strides for A/out, byte offsets and strides for B). KernelProvider.matmulFp16() defaults to null, so adding the native kernel is purely additive; the dispatch in DefaultCpuOpsJvm already selects by codec and needs no change.
Decode is the whole difference from BF16. BF16 widens with a bit shift; binary16 needs exponent rebiasing, gradual-underflow handling for subnormals, and the 65504 ceiling. In C the natural options are the _Float16 type (GCC/Clang on ARMv8.2-A+ and x86 with F16C) or an explicit bit-twiddling path for portability. Whichever is chosen, accumulate in float, never in half — the SPI contract and every existing implementation do this, and it is what PyTorch/JAX/tensor cores do.
Fp16Codec in skainet-lang-core is the reference semantics to match: round-to-nearest-ties-to-even, overflow to ±Inf, gradual underflow, NaN payload preserved rather than collapsed to Inf. NarrowFloatCodecTest pins all of those and is the spec the C code must agree with.
Worth noting on ordering: the on-device target (Torq T1) is bf16-native and has no FP16 support at all, and the board's IREE VM has no EXT_F16 — so this is a host-side CPU performance item, not something the deployment path is waiting on. Prioritise accordingly.
Related
🤖 Generated with Claude Code
Follow-up to #884, which added FP16 matmul at the scalar (0) and Panama Vector (50) tiers. BF16 additionally has a native FFM (100) kernel, so on any machine where the native provider wins, FP16 silently runs a tier slower than BF16.
The asymmetry is correct but easy to misread: benchmarking the two formats today compares a Panama kernel against a native one, not FP16 against BF16.
Scope
native/src/fp16_matmul.cexportingskainet_fp16_matmul, mirroringbf16_matmul.c.NativeFp16MatmulKernel(JVM FFM binding) mirroringNativeBf16MatmulKernel.matmulFp16()in the native provider, priority 100.ScalarFp16MatmulKernel, mirroringNativeBf16MatmulKernelParityTest.KernelSupportMatrixTestto cover FP16.Notes for whoever picks this up
The SPI is already in place —
Fp16MatmulKernel : NarrowFloatMatmulKernelwith the same signature as the BF16 kernel, including the asymmetric stride convention (float strides forA/out, byte offsets and strides forB).KernelProvider.matmulFp16()defaults tonull, so adding the native kernel is purely additive; the dispatch inDefaultCpuOpsJvmalready selects bycodecand needs no change.Decode is the whole difference from BF16. BF16 widens with a bit shift; binary16 needs exponent rebiasing, gradual-underflow handling for subnormals, and the 65504 ceiling. In C the natural options are the
_Float16type (GCC/Clang on ARMv8.2-A+ and x86 with F16C) or an explicit bit-twiddling path for portability. Whichever is chosen, accumulate infloat, never in half — the SPI contract and every existing implementation do this, and it is what PyTorch/JAX/tensor cores do.Fp16Codecinskainet-lang-coreis the reference semantics to match: round-to-nearest-ties-to-even, overflow to ±Inf, gradual underflow, NaN payload preserved rather than collapsed to Inf.NarrowFloatCodecTestpins all of those and is the spec the C code must agree with.Worth noting on ordering: the on-device target (Torq T1) is bf16-native and has no FP16 support at all, and the board's IREE VM has no
EXT_F16— so this is a host-side CPU performance item, not something the deployment path is waiting on. Prioritise accordingly.Related
PanamaVectorBf16MatmulKernel(IntVector.lanewise(LSHL, 16).reinterpretAsFloats(), flagged in its own KDoc). Independent of this issue, and BF16-only — the shift trick does not apply to binary16.🤖 Generated with Claude Code