handy-computer · brauliobo · Jul 25, 2026 · Jul 25, 2026
@@ -56,9 +56,11 @@ value at the default context; for the effective limit under a lowered
 
 | Families | Limit | Behavior |
 | --- | --- | --- |
-| whisper, parakeet (all variants), voxtral_realtime | none (`max_audio_ms = 0`) | Long audio is windowed internally and stitched (whisper), processed by an unbounded/streaming encoder (parakeet, voxtral_realtime), or padded if short. No practical limit; all three **ignore `n_ctx`** (it cannot lower a limit they don't have). whisper and parakeet never error on length. voxtral_realtime has one exception — its absolute `dec_max_position` cap (~2.9 h, see below): a clip past it returns `INPUT_TOO_LONG` (one-shot/batch), and a stream that reaches it flags `was_truncated`. |
+| whisper, canary-1b-v2, parakeet (all variants), voxtral_realtime | none (`max_audio_ms = 0`) | Long audio is windowed internally and stitched (whisper, canary-1b-v2), processed by an unbounded/streaming encoder (parakeet, voxtral_realtime), or padded if short. No practical input limit. whisper, canary-1b-v2, and parakeet never error on input length. voxtral_realtime has one exception — its absolute `dec_max_position` cap (~2.9 h, see below): a clip past it returns `INPUT_TOO_LONG` (one-shot/batch), and a stream that reaches it flags `was_truncated`. |
 
-Whisper slices audio into 30 s windows with prev-context stitching; parakeet's
+Whisper slices audio into 30 s windows with prev-context stitching;
+canary-1b-v2 uses 30-40 s windows, selects low-energy acoustic boundaries,
+and stitches a 1 s overlap; parakeet's
 conformer is effectively unbounded (the encoder positional table is recomputed
 per run, not a fixed wall). This holds for every parakeet variant, including the
 cache-aware streaming RNN-T member (`nemotron-3.5-asr-streaming-0.6b`): the
@@ -71,7 +73,7 @@ need and do not have a length gate.
 
 | Families | Limit source | Behavior |
 | --- | --- | --- |
-| qwen3_asr, canary_qwen, funasr_nano, granite, granite_nar, voxtral, cohere, canary | decoder context window (`dec_max_position_embeddings` / `dec_max_seq`), or the encoder positional table (`enc_pos_emb_max_len`, for cohere/canary) — all from GGUF | KV cache grows to fit, clamped to the model's true max. Over-length input is **rejected before the decode** (or before the encoder, where the encoder table is the binding limit) with `TRANSCRIBE_ERR_INPUT_TOO_LONG`. |
+| qwen3_asr, canary_qwen, funasr_nano, granite, granite_nar, voxtral, cohere, canary except canary-1b-v2 | decoder context window (`dec_max_position_embeddings` / `dec_max_seq`), or the encoder positional table (`enc_pos_emb_max_len`, for cohere/canary) — all from GGUF | KV cache grows to fit, clamped to the model's true max. Over-length input is **rejected before the decode** (or before the encoder, where the encoder table is the binding limit) with `TRANSCRIBE_ERR_INPUT_TOO_LONG`. |
 
 These families wrap an LLM-style decoder whose context window
 (`audio_tokens + prompt + generation`) is the binding constraint. The number of
@@ -144,10 +146,12 @@ self-KV / output budget. In those families `transcribe_session_get_limits()`
 reports a smaller `effective_n_ctx` and `max_kv_bytes` when `n_ctx` is lowered,
 but `effective_max_audio_ms` stays pinned to the encoder input bound.
 
-**Chunked / unbounded families (bucket 1) ignore `n_ctx` entirely.** whisper,
-parakeet, and voxtral_realtime have no lowerable context ceiling, so a non-zero
-`n_ctx` is a documented no-op and `transcribe_session_get_limits()` keeps
-reporting them unbounded. voxtral_realtime is the subtle case: it *does* have an
+Most chunked / unbounded families ignore `n_ctx` entirely. whisper, parakeet,
+and voxtral_realtime have no lowerable context ceiling, so a non-zero `n_ctx`
+is a documented no-op. canary-1b-v2 is the exception: its input remains
+unbounded because each window gets a fresh decoder context, while `n_ctx` can
+still lower that per-window output-token budget and cause `OUTPUT_TRUNCATED`.
+voxtral_realtime is the other subtle case: it *does* have an
 absolute decoder position cap (`dec_max_position`, ~2.9 h of audio), but that is
 the model's true RoPE wall — not a memory ceiling a caller can lower — so
 `n_ctx` does not narrow it (its decoder KV is a constant-memory sliding ring;

@@ -23,8 +23,9 @@ The model takes a 16 kHz mono WAV and produces a transcript. Supports:
 This is the broadest-coverage canary variant. The other multilingual
 variants (180m-flash, 1b-flash) cover only English/German/Spanish/French.
 
-Not a streaming model. Word and segment timestamps from the upstream
-model are not exposed in the v1 port.
+Not a streaming model. Word and segment timestamps are available when the
+GGUF includes the upstream auxiliary CTC aligner. Older GGUFs without the
+aligner remain text-only and reject explicit timestamp requests.
 
 See NVIDIA's [model card](https://huggingface.co/nvidia/canary-1b-v2)
 for training data, intended use, and upstream evaluation methodology.
@@ -44,6 +45,9 @@ pinned 2026-05-08.
 | Q5_K_M | [canary-1b-v2-Q5_K_M.gguf](https://huggingface.co/handy-computer/canary-1b-v2-gguf/resolve/main/canary-1b-v2-Q5_K_M.gguf) | 798 MB | 1.93% |
 | Q4_K_M | [canary-1b-v2-Q4_K_M.gguf](https://huggingface.co/handy-computer/canary-1b-v2-gguf/resolve/main/canary-1b-v2-Q4_K_M.gguf) | 701 MB | 1.91% |
 
+These published artifacts predate embedded-aligner support and remain text-only.
+Use the [conversion recipe](#convert) below to create a timestamp-enabled GGUF.
+
 WER is measured on the full LibriSpeech test-clean split (2620 utterances)
 with greedy decoding and no external LM. F32 reference baseline: 1.92%.
 NVIDIA's self-reported number on the upstream model card is 2.18%; our
@@ -58,7 +62,7 @@ cmake --build build
 
 # ASR (any supported language)
 build/bin/transcribe-cli \
-  -m models/canary-1b-v2/canary-1b-v2-Q8_0.gguf \
+  -m models/canary-1b-v2/canary-1b-v2-timestamps-Q8_0.gguf \
   -l en \
   samples/jfk.wav
 
@@ -74,6 +78,13 @@ build/bin/transcribe-cli \
   --task translate \
   -l en --target-language de \
   samples/jfk.wav
+
+# Word timestamps (requires a GGUF converted with the auxiliary aligner)
+build/bin/transcribe-cli \
+  -m models/canary-1b-v2/canary-1b-v2-Q8_0.gguf \
+  -l en --timestamps word \
+  --output-json transcript.json \
+  samples/jfk.wav
 ```
 
 If your audio is not already 16 kHz mono WAV, convert it first:
@@ -82,6 +93,11 @@ If your audio is not already 16 kHz mono WAV, convert it first:
 ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav
 ```
 
+There is no practical per-call audio limit. Inputs over 40 seconds are
+processed in 30-40 second windows selected at low-energy acoustic boundaries,
+with 1 second overlap, and stitched into one transcript; word and segment
+timestamps remain on the original audio timeline.
+
 CLI flags specific to canary:
 
 - `--pnc` / `--no-pnc` — punctuation & capitalization. Note: canary-1b-v2
@@ -90,7 +106,13 @@ CLI flags specific to canary:
 - `-l <code>` — source language code (one of the 25 supported BCP-47
   codes).
 - `--task translate` + `--target-language <code>` — switch to translation
-  mode.
+  mode. Translation returns text only; explicit timestamp requests are rejected
+  because the auxiliary CTC model aligns source-language speech.
+- `--timestamps word|segment` — run the embedded CTC aligner. Timestamp-enabled
+  v2 conversions advertise word support; GGUFs converted from stripped sources
+  without `timestamps_asr_model` advertise none.
+- `--output-json <path>` — write structured segment and nested word timings in
+  integer milliseconds for subtitle integrations.
 
 ## Performance
 
@@ -158,13 +180,15 @@ For the full porting writeup, see
 
 ```bash
 uv run --project scripts/envs/canary \
-  scripts/convert-canary.py nvidia/canary-1b-v2 --repo-id nvidia/canary-1b-v2
+  scripts/convert-canary.py nvidia/canary-1b-v2 \
+  models/canary-1b-v2/canary-1b-v2-timestamps-F32.gguf
 ```
 
 ### Quantize
 
 ```bash
-uv run scripts/quantize-all.py models/canary-1b-v2/canary-1b-v2-F32.gguf
+uv run scripts/quantize-all.py \
+  models/canary-1b-v2/canary-1b-v2-timestamps-F32.gguf
 ```
 
 ### Validate

@@ -44,11 +44,13 @@ each per-variant doc has direct download links.
 
 ## Input limits
 
-Every variant accepts up to about **6.7 minutes (400 s)** of 16 kHz mono audio
-per call — the encoder's positional table is the binding limit, shared across
-the family. Longer audio is rejected up front with
-`TRANSCRIBE_ERR_INPUT_TOO_LONG` rather than silently truncated; split it into
-shorter segments. See the [input-length contract](../input-limits.md).
+`canary-1b-v2` has no practical per-call limit. Audio over 40 seconds is split
+into 30-40 second windows at low-energy acoustic boundaries with 1 second
+overlap, then stitched by subword-token agreement; word timestamps are rebased
+onto the original timeline. The other variants retain their encoder positional-table
+limit of about **6.7 minutes (400 s)** and reject longer audio up front with
+`TRANSCRIBE_ERR_INPUT_TOO_LONG`. See the
+[input-length contract](../input-limits.md).
 
 ## Quick start
 

@@ -140,7 +140,7 @@ uv run scripts/bench/run.py \
   - canary-1b-v2 / 1b-flash / 180m-flash: `[<source_lang>, <target_lang>, <task>, <pnc>, <toggle_timestamps>]` (5 slots, `task='asr'|'ast'`).
 - Output head: LM head over the concatenated SP vocabulary. Decoding is beam search by default for the original canary-1b (beam=5, length_penalty=1.0) and greedy by default for the flash variants (beam=1).
 - Tokenizer: concatenated SentencePiece — one SP model per language concatenated into a single vocabulary. canary-1b-v2 is 16,384 pieces; flash/180m-flash/1b vocab sizes are not stated on model cards (Stage 2 fills from .nemo).
-- Audio length contract: native ≤40 s direct inference. <1 s is symmetrically zero-padded to 1 s. >40 s is handled by an external chunked inference script with 1 s overlap (canary-1b-v2 chunk len defaults to 40 s; canary-1b-flash 10 s; canary-180m-flash 10 s). **Long-form / streaming is out of scope for the v1 port.**
+- Audio length contract: native ≤40 s direct inference. <1 s is symmetrically zero-padded to 1 s. Upstream handles >40 s with overlapping chunks. transcribe.cpp implements that path for canary-1b-v2 with low-energy boundaries inside a 30-40 s window, 1 s overlap, and subword-LCS stitching; streaming remains unsupported. The other variants retain the encoder positional-table input cap.
 
 ## Capabilities (from intake)
 
@@ -177,8 +177,8 @@ See per-variant `reports/porting/canary/<variant>/intake.json::known_risks`. Fam
 4. **NeMo FilterbankFeatures applies preemph=0.97 BEFORE windowing/STFT.** Standard NeMo ASR frontend trap. Per-feature normalization (per mel band, across time) — not per-utterance global stats.
 5. **Cross-attention from transformer decoder to encoder output.** Padding mask on the cross-attention keys must propagate from encoder input lengths after the factor-8 subsampling.
 6. **Decoding default differs by variant.** canary-1b: beam=5, length_penalty=1.0. flash variants: beam=1 greedy. Mismatch with upstream's measured-config beam will move WER.
-7. **Audio length contract.** Native ≤40 s; <1 s is zero-padded to 1 s; long-form needs an external chunker. Streaming is not native — out of scope for v1.
-8. **Timestamps are not native to the AED.** canary-1b-v2 / 1b-flash / 180m-flash advertise word + segment timestamps but produce them via a side CTC aligner inside the .nemo archive, not from the AED itself. Out of scope for v1.
+7. **Audio length contract.** Native ≤40 s; <1 s is zero-padded to 1 s. canary-1b-v2 long-form uses internal 30-40 s overlapping windows; the other variants still expose the encoder positional-table cap. Streaming is not native.
+8. **Timestamps are not native to the v2 AED.** canary-1b-v2 aligns the AED transcript with a side CTC Conformer stored in the `.nemo`. The aligner tokenizer IDs must remain identical to the AED tokenizer IDs. Flash timestamp-token decoding remains out of scope.
 9. **License divergence.** canary-1b is CC-BY-NC-4.0; the rest are CC-BY-4.0. Converter must surface license correctly in `general.license` so non-commercial restrictions on canary-1b ride along with the GGUF.
 10. **.nemo archive distribution.** No HF `config.json` / `tokenizer_config.json` / `generation_config.json` (except canary-1b-flash's encoder-only HF Transformers shim). Converter must mirror the parakeet pattern of unpacking the .nemo before extraction.
 
@@ -209,8 +209,9 @@ flagged in the row notes.
 | Translate | EN→de | `build/bin/transcribe-cli -m models/<variant>/<variant>-F32.gguf --language en --translate --target-language de samples/jfk.wav` | non-empty plausible German translation of JFK | PASS — verified on all four variants 2026-05-07 (`--target-language` wired through `params->target_language`; canary-1 uses explicit `<\|translate\|>` task token, canary2 infers from src!=tgt) |
 | Translate | EN↔es / EN↔fr | same as above with `--target-language es\|fr` | non-empty plausible translation in the requested language | ACCEPTED GAP — no es/fr reference samples shipped; runtime path identical to the de case |
 | Translate | X→EN for 24 langs (canary-1b-v2 only) | `build/bin/transcribe-cli -m models/canary-1b-v2/canary-1b-v2-F32.gguf --language <X> --translate --target-language en samples/<X>.wav` | non-empty plausible English translation | ACCEPTED GAP — multilingual sample set not shipped; runtime path identical to the EN→de case |
-| Segment timestamps | only canary-1b-v2 / 1b-flash / 180m-flash | n/a — not exposed by runtime in v1 | n/a | SKIP — not exposed by runtime (timestamps decoder path / `_timestamps_asr_model` aligner not ported in v1; intake known_risks notes timestamps are explicitly experimental upstream) |
-| Word timestamps | only canary-1b-v2 / 1b-flash / 180m-flash | same | n/a | SKIP — not exposed by runtime (same reason) |
+| Segment timestamps | canary-1b-v2 | `build/bin/transcribe-cli -m models/canary-1b-v2/canary-1b-v2-F32.gguf --timestamps segment samples/jfk.wav` | non-zero `segments:` rows matching NeMo within one 80 ms encoder frame | PASS — embedded CTC aligner matched NeMo on `samples/jfk.wav` 2026-07-23 |
+| Word timestamps | canary-1b-v2 | `build/bin/transcribe-cli -m models/canary-1b-v2/canary-1b-v2-F32.gguf --timestamps word samples/jfk.wav` | non-zero `words:` rows matching NeMo within one 80 ms encoder frame | PASS — 22 words matched NeMo within one frame; covered by the golden manifest and gated real-model API smoke 2026-07-23 |
+| Word/segment timestamps | canary-1b-flash / canary-180m-flash | n/a | n/a | SKIP — upstream AED timestamp-token path is not exposed by the runtime |
 | Streaming | n/a | not advertised by upstream | n/a | SKIP — not advertised |
 | Voice activity detection | n/a | not advertised by upstream | n/a | SKIP — not advertised |
 | Speaker diarization | n/a | not advertised by upstream | n/a | SKIP — not advertised |
@@ -222,7 +223,7 @@ flagged in the row notes.
 - Canary's FastConformer encoder shares the rel_pos relative-shift implementation that Parakeet already needs. The C++ encoder code can probably be re-used with parameter-only changes (n_layers, ffn_dim, etc.).
 - Use `samples/jfk.wav` for the English smoke test. German/Spanish/French sample clips and the wider 25-language multilingual set need to be sourced for canary-1b-v2 — pull from FLEURS or Common Voice during Stage 2 dumper setup.
 - canary-1b is CC-BY-NC-4.0. Do not bake "Apache-2.0" or default license metadata into its converter output; surface the non-commercial restriction in `general.license`.
-- canary-1b-v2 ships timestamps via a separate CTC aligner inside the .nemo archive (`_timestamps_asr_model`). The model card says these can be deleted to save memory; Stage 2 should decide whether to keep them in the converted GGUF or strip (default: strip — timestamps out of scope for v1).
+- canary-1b-v2 ships timestamps via a separate CTC aligner inside the `.nemo` archive (`timestamps_asr_model`, deliberately excluded from the AED `state_dict`). The converter embeds it in the same GGUF under `aligner.*`; older v2 GGUFs without those tensors remain text-only. The F32 aligner adds about 2.33 GiB before quantization.
 - **Stage 3 tensor-name decisions** (`scripts/convert-canary.py`):
   - Encoder block names mirror `src/arch/parakeet/weights.cpp` verbatim (FastConformer module class is shared); the converter is a pass-through with no transposes.
   - Decoder layer names use `dec.layer.{i}.{norm1, self_attn.{q,k,v,o}, norm2, cross_attn.{q,k,v,o}, norm3, ffn.{up,down}}` rather than NeMo's `first_sub_layer`/`second_sub_layer`/`third_sub_layer`. The latter is fine for source comprehension but unfriendly to a reader of the future Stage 4 `src/arch/canary/` code.