Skip to content

Board-verify the FunctionGemma KV-cache decode loop on the SL2610 #12

Description

@michalharakal

The KV-cache 2-graph decode (GemmaKvDecoder, GEMMA_KV=1) is CPU-verified (transformers PR #245) but the
board loop is an UNVERIFIED draft. On the first SL2610 run, confirm each silent-corruption trap:

  • Per-block K-vs-V output orderGemmaKvDecoder.kFirstInOutput=false ((V,K) per return-SSA); flip to true if the first run yields garbage tokens.
  • --output=@file formatIreeRuntime.invokeFiles assumes RAW bytes; strip the .npy header on read if the board writes NumPy.
  • gemma_with_past input arg order — trace-derived; confirm against the compiled vmfb.
  • Dynamic-concat — the x?x256 sentinel-relax leaves shape inference to iree-compile; confirm the vmfb accepts it (else fixed-pad+mask fallback).
  • --task_topology_group_count accepted by the board iree-run-module (gated to revert).
  • vmfbs built with the Torq-fork iree-compile (g165e12a).

Done: the loop reproduces the oracle [262146,236769,3255,718,498,1373,262152,106], PERF-LOGBOOK.md shows the O(seq²)→O(seq) collapse vs ~6000 ms/token, then make GEMMA_KV=1 the default in Pipeline.kt.

Runbook: sl2610-function-calling/docs/GEMMA-KV-BOARD-LOOP.md. Tracker: docs/GEMMA-KV-INT8.md.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions