diff --git a/CHANGELOG.md b/CHANGELOG.md index a36f69f7..fca59a3c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,8 +2,23 @@ ## [Unreleased] +## [0.38.0] - 2026-07-30 + ### Added +- **Narrow-float (BF16 + FP16) weights kept packed.** A shared `NarrowFloatCodec` layer + (`Bf16Codec`/`Fp16Codec`) plus `NarrowFloatDenseTensorData`/`Fp16DenseTensorData` let SafeTensors F16 + and GGUF F16/BF16 weights load `KEEP_NATIVE` — two bytes per element at rest instead of widening to + FP32 at load. `DefaultCpuOpsJvm` dispatches to format-specific matmul kernels by codec, so the packed + weight reaches the kernel rather than a widened copy. Narrow floats are a *storage* width only: + kernels widen to f32 lanes, accumulate in f32, and narrow on store. +- **Native (FFM) FP16 matmul kernel.** `skainet_fp16_matmul` joins the existing BF16 kernel in + `skainet-backend-native-cpu`, wired through `NativeKernelProvider.matmulFp16()`. Until now the + provider carried `matmulBf16` but no FP16 counterpart, so BF16 resolved to the native kernel at + priority 100 while FP16 silently cascaded to the JVM Panama kernel at 50 — which read as a slow + kernel and was a missing one. `KernelProvider.supports` gains the matching `"Float16"` arm, absent + while every other matmul dtype was present. + - **First-class dynamic dimensions (`Dim`).** A new `sk.ainet.lang.tensor.Dim` vocabulary makes "dynamic extent" explicit instead of an overloaded `-1`: `Dim.DYNAMIC` is a reserved sentinel (`Int.MIN_VALUE`) **distinct from reshape's `-1` = infer**, with the dynamic-aware shape arithmetic (`concat`, `compatible`, @@ -53,6 +68,49 @@ (`VoidTensorOps.calculateConcatShape`) and the emitter (`ShapeOperationsConverter` concat) now keep the concatenated axis dynamic when any operand's extent there is dynamic, instead of numerically summing it (which turned a growing cache `? ++ 1` into a bogus static `0`). +- **Narrow-float weights transpose for free.** `NarrowFloatInputMajorTensorData` stores a rank-2 narrow + weight input-major, so the `[out, in]` → `[in, out]` transpose that `Linear.onForward` performs on + every call is a zero-copy reinterpretation of the same buffer. Projections are stored `[out, in]` but + the narrow matmul dispatch needs `[in, out]`, and transposing a row-major narrow tensor previously + walked it elementwise through boxed `get()` and widened to FP32 — 4.4 s for a 4096x11008 projection, + per weight, per token, which made `KEEP_NATIVE` slower than not using it. A row-major narrow buffer + deliberately still takes the generic path: swapping its shape would silently yield a different matrix + rather than the transpose. +- **Native narrow-float kernels read the weight once per matmul.** Both the BF16 and FP16 kernels tile + the `j` dimension at `m > 1` and widen each weight row once per tile, instead of walking the whole + weight matrix once per row of the input. Accumulation into any output element stays `p` ascending, so + results are bit-identical to the previous formulation. At `m == 1` both keep the straight pass — there + is nothing to amortize and tiling costs ~15% there. +- **`Fp16Codec.decode` is straight-line.** The subnormal arm no longer renormalizes in a data-dependent + loop; binary16 subnormals are `mant * 2⁻²⁴` with both factors exact in FP32, so one multiply lets the + hardware renormalize. A NaN now decodes **quiet**, matching `encode` (which already never emitted a + signaling binary16 NaN) and the hardware conversion the JVM kernel uses. This changes 1022 patterns — + the signaling NaNs — and nothing else; without it the JVM and every other target would disagree on + them. + +### Fixed + +- **FP16 matmul was 2–18x slower than the FP32 SGEMM it replaces.** The cause was dispatch, not + arithmetic: `NativeKernelProvider` had no `matmulFp16()`, so FP16 fell through to the JVM kernel while + BF16 ran natively. Head to head the two JVM kernels are within ~15% of each other. FP16 is now + 1.5–1.7x *faster* than FP32. +- **CI and supply-chain hardening.** Least-privilege permissions on the build workflow, the docs MathJax + npm install pinned by version, and a `logback-classic` bump. + +### Performance + +Measured on an i7-9750H (AVX2), OpenJDK 21, median ms per call, `4096x11008` projection: + +| | batch 1 | batch 16 | +|---|---|---| +| FP32 SGEMM | 108.7 | 271.1 | +| BF16 | 57.4 | 143.5 | +| FP16 | 71.2 | 159.3 | + +Both narrow formats now beat the FP32 SGEMM — BF16 by 1.8–1.9x, FP16 by 1.5–1.7x. Note these kernels +are compute-bound on the FMA chain at batch 16, not bandwidth-bound: cutting weight traffic 16x bought +only 9–19%, so the next win is a blocked microkernel or `bfdot`/`bfmmla` on ARMv8.6-A+, not more layout +work. ## [0.37.0] - 2026-07-25 diff --git a/README.md b/README.md index 130061d0..785f29b5 100644 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ Add the core dependencies (Gradle Kotlin DSL): ```kotlin dependencies { // Recommended: import the umbrella BOM and drop versions on the engine modules. - implementation(platform("sk.ainet:skainet-bom:0.37.0")) + implementation(platform("sk.ainet:skainet-bom:0.38.0")) implementation("sk.ainet.core:skainet-lang-core") implementation("sk.ainet.core:skainet-backend-cpu") @@ -287,20 +287,19 @@ val withoutLabel = dataPipeline() --- -## What's New in 0.37.0 +## What's New in 0.38.0 -- **`Lstm` layer** — single-layer, batch-first LSTM built from existing primitives only (no new `TensorOps` op, traces to StableHLO without a dedicated converter), with `torch.nn.LSTM`-compatible gate order and an explicit caller-owned `LstmState` + `step()` API for transducer prediction networks. -- **Training essentials** — `Dropout` now performs real inverted dropout under a training-phase context (it was an identity placeholder), optimizers expose a mutable `lr` plus a `linearWarmupCosineDecay` LR schedule, `Linear` supports bias-less projections (`nn.Linear(bias=False)` equivalent) and is `open` for LoRA-style adapters. -- **Attention scale fix** — `scaledDotProductAttention` at its default scale multiplied every score by zero on the CPU backend, collapsing softmax to a uniform average; it now resolves to `1/sqrt(headDim)` as documented. -- **Autograd correctness** — `CrossEntropyLoss` no longer detaches the tape (gradients reached the predictions in neither target path), and `softmax`/`logSoftmax`/`variance` backward now work for rank ≥ 3. -- **Android native IO** — `skainet-io-core` and `skainet-io-safetensors` gain `androidNative` targets (arm64 and arm32). -- **Reproducible, hardened CI** — every GitHub Action pinned to a commit hash, the docs Docker image pinned by digest with exact npm package versions, and `allTests` split into parallel per-target jobs to end OOM flakes. +- **Streaming KV-cache decode (dynamic dimensions)** — a first-class `Dim` vocabulary makes "dynamic extent" explicit instead of an overloaded `-1`, and the StableHLO emitter renders it as an MLIR `?`. One compiled vmfb now serves every autoregressive decode step with a growing cache, instead of one fixed cache length. Verified end-to-end: the full FunctionGemma `with_past` decode graph and the Moonshine v2 decoder (dynamic self *and* cross caches) self-compile from the DSL to a CPU vmfb — graphs that could not be compiled before. Static graphs are emitted byte-for-byte unchanged. +- **Narrow-float (BF16 + FP16) weights kept packed** — SafeTensors F16 and GGUF F16/BF16 weights load `KEEP_NATIVE`, two bytes per element at rest instead of widening to FP32, and reach format-specific matmul kernels still packed. Narrow floats are a storage width only: kernels widen to f32 lanes and accumulate in f32. +- **Both narrow formats now beat the FP32 SGEMM** — BF16 by 1.8–1.9x, FP16 by 1.5–1.7x on a 4096x11008 projection. Getting there took a zero-copy transpose for input-major weights (the per-token transpose previously widened the tensor elementwise, 4.4 s per projection), a native FFM FP16 kernel to match the existing BF16 one, and tiling both kernels so the weight is read once per matmul rather than once per input row. +- **Allocation-free shape-only tracing** — `VoidTensorOps` propagates shapes through a `ShapeOnlyTensorData` that allocates no backing buffer, so a dynamic extent flows through a whole decode trace instead of throwing on a negative-size allocation. -### Previously, in 0.36.0 +### Previously, in 0.37.0 -- **Kotlin 2.4.0 toolchain** — KSP 2.3.10 and Dokka 2.2.0 aligned to the new compiler. No public API changes. -- **`permute` replay fix (`ComputeGraphExecutor`)** — a traced `permute(t, axes)` now replays with its recorded axes instead of being dispatched as a plain last-two-dims transpose. -- **REUSE / SPDX license-compliance setup** — `REUSE.toml` + `LICENSES/`, a CI compliance workflow, and a REUSE status badge. +- **`Lstm` layer** — single-layer, batch-first LSTM built from existing primitives only, with `torch.nn.LSTM`-compatible gate order and a caller-owned `LstmState` + `step()` API. +- **Training essentials** — real inverted `Dropout`, mutable optimizer `lr` plus `linearWarmupCosineDecay`, bias-less and `open` `Linear`. +- **Attention scale fix** — `scaledDotProductAttention` at its default scale multiplied every score by zero on the CPU backend; it now resolves to `1/sqrt(headDim)` as documented. +- **Autograd correctness** — `CrossEntropyLoss` no longer detaches the tape, and `softmax`/`logSoftmax`/`variance` backward now work for rank ≥ 3. See [CHANGELOG.md](CHANGELOG.md) for details and the full release history. @@ -326,6 +325,11 @@ We love contributions! Whether it's a new operator, documentation, or a bug fix: Browse the full codebase documentation on [DeepWiki](https://deepwiki.com/SKaiNET-developers/SKaiNET). +### Contributors (0.38.0) + +- **Michal Harakal** ([@michalharakal](https://github.com/michalharakal)) — dynamic tensor dimensions for streaming KV-cache decode (#891), shared narrow-float BF16/FP16 layer (#886), zero-copy transpose for input-major narrow weights (#895), native FP16 matmul kernel (#896), read-once weight tiling in the native narrow kernels (#897) +- **[@MacOS](https://github.com/MacOS)** — least-privilege permissions on the build workflow (#899), MathJax npm install pinned by version (#889) + ### Contributors (0.37.0) - **Michal Harakal** ([@michalharakal](https://github.com/michalharakal)) — `Lstm` layer (#824), `Dropout` masking (#867), LR schedules (#866), optional/open `Linear` (#870, #875), SDPA scale fix (#880), autograd fixes (#877), `argMax` DAG spec (#878), tokenizer + `gather` fixes (#879), Android native IO targets (#836, #842, #845) diff --git a/docs/antora.yml b/docs/antora.yml index 8b452002..0830d93b 100644 --- a/docs/antora.yml +++ b/docs/antora.yml @@ -15,7 +15,7 @@ asciidoc: framework_name: SKaiNET # Current SKaiNET release — bump once per release; referenced as # {skainet_version} in dependency snippets (blocks need subs="attributes+"). - skainet_version: 0.37.0 + skainet_version: 0.38.0 ksp_version: 2.2.21-2.0.5 dokka_version: 2.1.0 asciidoctorj_version: 3.0.0 diff --git a/gradle.properties b/gradle.properties index 2a00dfce..7b371181 100644 --- a/gradle.properties +++ b/gradle.properties @@ -1,5 +1,5 @@ GROUP=sk.ainet.core -VERSION_NAME=0.37.0 +VERSION_NAME=0.38.0 POM_DESCRIPTION=SKaiNET POM_URL=https://github.com/SKaiNET-developers/skainet/