int8 weight-only quant (`GEMMA_QUANT=int8`) is export-verified on CPU (transformers PR #245). On-device, confirm: - [ ] Numeric quality of per-row symmetric int8 from Q5_K (oracle token-for-token check). - [ ] Decode speed vs bf16. - [ ] RAM claim: irpa 831 → ~415 MiB on the 1.9 GB board. - [ ] `iree-compile` (Torq-fork) accepts the in-graph dequant (`i8→f32 × scale`). Tracker: `sl2610-function-calling/docs/GEMMA-KV-INT8.md`.
int8 weight-only quant (
GEMMA_QUANT=int8) is export-verified on CPU (transformers PR #245). On-device, confirm:iree-compile(Torq-fork) accepts the in-graph dequant (i8→f32 × scale).Tracker:
sl2610-function-calling/docs/GEMMA-KV-INT8.md.