handy-computer · brauliobo · Jul 25, 2026
@@ -23,8 +23,9 @@ The model takes a 16 kHz mono WAV and produces a transcript. Supports:
 This is the broadest-coverage canary variant. The other multilingual
 variants (180m-flash, 1b-flash) cover only English/German/Spanish/French.
 
-Not a streaming model. Word and segment timestamps from the upstream
-model are not exposed in the v1 port.
+Not a streaming model. Word and segment timestamps are available when the
+GGUF includes the upstream auxiliary CTC aligner. Older GGUFs without the
+aligner remain text-only and reject explicit timestamp requests.
 
 See NVIDIA's [model card](https://huggingface.co/nvidia/canary-1b-v2)
 for training data, intended use, and upstream evaluation methodology.
@@ -44,6 +45,9 @@ pinned 2026-05-08.
 | Q5_K_M | [canary-1b-v2-Q5_K_M.gguf](https://huggingface.co/handy-computer/canary-1b-v2-gguf/resolve/main/canary-1b-v2-Q5_K_M.gguf) | 798 MB | 1.93% |
 | Q4_K_M | [canary-1b-v2-Q4_K_M.gguf](https://huggingface.co/handy-computer/canary-1b-v2-gguf/resolve/main/canary-1b-v2-Q4_K_M.gguf) | 701 MB | 1.91% |
 
+These published artifacts predate embedded-aligner support and remain text-only.
+Use the [conversion recipe](#convert) below to create a timestamp-enabled GGUF.
+
 WER is measured on the full LibriSpeech test-clean split (2620 utterances)
 with greedy decoding and no external LM. F32 reference baseline: 1.92%.
 NVIDIA's self-reported number on the upstream model card is 2.18%; our
@@ -58,7 +62,7 @@ cmake --build build
 
 # ASR (any supported language)
 build/bin/transcribe-cli \
-  -m models/canary-1b-v2/canary-1b-v2-Q8_0.gguf \
+  -m models/canary-1b-v2/canary-1b-v2-timestamps-Q8_0.gguf \
   -l en \
   samples/jfk.wav
 
@@ -74,6 +78,13 @@ build/bin/transcribe-cli \
   --task translate \
   -l en --target-language de \
   samples/jfk.wav
+
+# Word timestamps (requires a GGUF converted with the auxiliary aligner)
+build/bin/transcribe-cli \
+  -m models/canary-1b-v2/canary-1b-v2-Q8_0.gguf \
+  -l en --timestamps word \
+  --output-json transcript.json \
+  samples/jfk.wav
 ```
 
 If your audio is not already 16 kHz mono WAV, convert it first:
@@ -90,7 +101,13 @@ CLI flags specific to canary:
 - `-l <code>` — source language code (one of the 25 supported BCP-47
   codes).
 - `--task translate` + `--target-language <code>` — switch to translation
-  mode.
+  mode. Translation returns text only; explicit timestamp requests are rejected
+  because the auxiliary CTC model aligns source-language speech.
+- `--timestamps word|segment` — run the embedded CTC aligner. Timestamp-enabled
+  v2 conversions advertise word support; GGUFs converted from stripped sources
+  without `timestamps_asr_model` advertise none.
+- `--output-json <path>` — write structured segment and nested word timings in
+  integer milliseconds for subtitle integrations.
 
 ## Performance
 
@@ -158,13 +175,15 @@ For the full porting writeup, see
 
 ```bash
 uv run --project scripts/envs/canary \
-  scripts/convert-canary.py nvidia/canary-1b-v2 --repo-id nvidia/canary-1b-v2
+  scripts/convert-canary.py nvidia/canary-1b-v2 \
+  models/canary-1b-v2/canary-1b-v2-timestamps-F32.gguf
 ```
 
 ### Quantize
 
 ```bash
-uv run scripts/quantize-all.py models/canary-1b-v2/canary-1b-v2-F32.gguf
+uv run scripts/quantize-all.py \
+  models/canary-1b-v2/canary-1b-v2-timestamps-F32.gguf
 ```
 
 ### Validate

@@ -178,7 +178,7 @@ See per-variant `reports/porting/canary/<variant>/intake.json::known_risks`. Fam
 5. **Cross-attention from transformer decoder to encoder output.** Padding mask on the cross-attention keys must propagate from encoder input lengths after the factor-8 subsampling.
 6. **Decoding default differs by variant.** canary-1b: beam=5, length_penalty=1.0. flash variants: beam=1 greedy. Mismatch with upstream's measured-config beam will move WER.
 7. **Audio length contract.** Native ≤40 s; <1 s is zero-padded to 1 s; long-form needs an external chunker. Streaming is not native — out of scope for v1.
-8. **Timestamps are not native to the AED.** canary-1b-v2 / 1b-flash / 180m-flash advertise word + segment timestamps but produce them via a side CTC aligner inside the .nemo archive, not from the AED itself. Out of scope for v1.
+8. **Timestamps are not native to the v2 AED.** canary-1b-v2 aligns the AED transcript with a side CTC Conformer stored in the `.nemo`. The aligner tokenizer IDs must remain identical to the AED tokenizer IDs. Flash timestamp-token decoding remains out of scope.
 9. **License divergence.** canary-1b is CC-BY-NC-4.0; the rest are CC-BY-4.0. Converter must surface license correctly in `general.license` so non-commercial restrictions on canary-1b ride along with the GGUF.
 10. **.nemo archive distribution.** No HF `config.json` / `tokenizer_config.json` / `generation_config.json` (except canary-1b-flash's encoder-only HF Transformers shim). Converter must mirror the parakeet pattern of unpacking the .nemo before extraction.
 
@@ -209,8 +209,9 @@ flagged in the row notes.
 | Translate | EN→de | `build/bin/transcribe-cli -m models/<variant>/<variant>-F32.gguf --language en --translate --target-language de samples/jfk.wav` | non-empty plausible German translation of JFK | PASS — verified on all four variants 2026-05-07 (`--target-language` wired through `params->target_language`; canary-1 uses explicit `<\|translate\|>` task token, canary2 infers from src!=tgt) |
 | Translate | EN↔es / EN↔fr | same as above with `--target-language es\|fr` | non-empty plausible translation in the requested language | ACCEPTED GAP — no es/fr reference samples shipped; runtime path identical to the de case |
 | Translate | X→EN for 24 langs (canary-1b-v2 only) | `build/bin/transcribe-cli -m models/canary-1b-v2/canary-1b-v2-F32.gguf --language <X> --translate --target-language en samples/<X>.wav` | non-empty plausible English translation | ACCEPTED GAP — multilingual sample set not shipped; runtime path identical to the EN→de case |
-| Segment timestamps | only canary-1b-v2 / 1b-flash / 180m-flash | n/a — not exposed by runtime in v1 | n/a | SKIP — not exposed by runtime (timestamps decoder path / `_timestamps_asr_model` aligner not ported in v1; intake known_risks notes timestamps are explicitly experimental upstream) |
-| Word timestamps | only canary-1b-v2 / 1b-flash / 180m-flash | same | n/a | SKIP — not exposed by runtime (same reason) |
+| Segment timestamps | canary-1b-v2 | `build/bin/transcribe-cli -m models/canary-1b-v2/canary-1b-v2-F32.gguf --timestamps segment samples/jfk.wav` | non-zero `segments:` rows matching NeMo within one 80 ms encoder frame | PASS — embedded CTC aligner matched NeMo on `samples/jfk.wav` 2026-07-23 |
+| Word timestamps | canary-1b-v2 | `build/bin/transcribe-cli -m models/canary-1b-v2/canary-1b-v2-F32.gguf --timestamps word samples/jfk.wav` | non-zero `words:` rows matching NeMo within one 80 ms encoder frame | PASS — 22 words matched NeMo within one frame; covered by the golden manifest and gated real-model API smoke 2026-07-23 |
+| Word/segment timestamps | canary-1b-flash / canary-180m-flash | n/a | n/a | SKIP — upstream AED timestamp-token path is not exposed by the runtime |
 | Streaming | n/a | not advertised by upstream | n/a | SKIP — not advertised |
 | Voice activity detection | n/a | not advertised by upstream | n/a | SKIP — not advertised |
 | Speaker diarization | n/a | not advertised by upstream | n/a | SKIP — not advertised |
@@ -222,7 +223,7 @@ flagged in the row notes.
 - Canary's FastConformer encoder shares the rel_pos relative-shift implementation that Parakeet already needs. The C++ encoder code can probably be re-used with parameter-only changes (n_layers, ffn_dim, etc.).
 - Use `samples/jfk.wav` for the English smoke test. German/Spanish/French sample clips and the wider 25-language multilingual set need to be sourced for canary-1b-v2 — pull from FLEURS or Common Voice during Stage 2 dumper setup.
 - canary-1b is CC-BY-NC-4.0. Do not bake "Apache-2.0" or default license metadata into its converter output; surface the non-commercial restriction in `general.license`.
-- canary-1b-v2 ships timestamps via a separate CTC aligner inside the .nemo archive (`_timestamps_asr_model`). The model card says these can be deleted to save memory; Stage 2 should decide whether to keep them in the converted GGUF or strip (default: strip — timestamps out of scope for v1).
+- canary-1b-v2 ships timestamps via a separate CTC aligner inside the `.nemo` archive (`timestamps_asr_model`, deliberately excluded from the AED `state_dict`). The converter embeds it in the same GGUF under `aligner.*`; older v2 GGUFs without those tensors remain text-only. The F32 aligner adds about 2.33 GiB before quantization.
 - **Stage 3 tensor-name decisions** (`scripts/convert-canary.py`):
   - Encoder block names mirror `src/arch/parakeet/weights.cpp` verbatim (FastConformer module class is shared); the converter is a pass-through with no transposes.
   - Decoder layer names use `dec.layer.{i}.{norm1, self_attn.{q,k,v,o}, norm2, cross_attn.{q,k,v,o}, norm3, ffn.{up,down}}` rather than NeMo's `first_sub_layer`/`second_sub_layer`/`third_sub_layer`. The latter is fine for source comprehension but unfriendly to a reader of the future Stage 4 `src/arch/canary/` code.

@@ -17,35 +17,72 @@
 #include <cstdio>
 #include <cstdlib>
 #include <cstring>
+#include <filesystem>
 #include <fstream>
+#include <sstream>
 #include <string>
 #include <vector>
 
 namespace {
 
-// Minimal JSON string escape: covers the characters MUST be escaped by
-// the JSON spec (quote, backslash, control chars). Transcribed text is
-// short UTF-8 in practice; we don't need unicode escaping.
+// JSON string escape with UTF-8 validation. A truncated decoder can stop in
+// the middle of a byte sequence; replace malformed bytes instead of emitting
+// syntactically invalid JSON.
 std::string json_escape(const char * s) {
     std::string out;
-    for (const char * p = s ? s : ""; *p; ++p) {
+    for (const char * p = s ? s : ""; *p;) {
         const unsigned char c = static_cast<unsigned char>(*p);
         if (c == '"') {
             out += "\\\"";
+            ++p;
         } else if (c == '\\') {
             out += "\\\\";
+            ++p;
         } else if (c == '\n') {
             out += "\\n";
+            ++p;
         } else if (c == '\r') {
             out += "\\r";
+            ++p;
         } else if (c == '\t') {
             out += "\\t";
+            ++p;
         } else if (c < 0x20) {
             char buf[8];
             std::snprintf(buf, sizeof(buf), "\\u%04x", c);
             out += buf;
-        } else {
+            ++p;
+        } else if (c < 0x80) {
             out += static_cast<char>(c);
+            ++p;
+        } else {
+            int n = 0;
+            if (c >= 0xC2 && c <= 0xDF) {
+                n = 2;
+            } else if (c >= 0xE0 && c <= 0xEF) {
+                n = 3;
+            } else if (c >= 0xF0 && c <= 0xF4) {
+                n = 4;
+            }
+            bool valid = n > 0;
+            for (int i = 1; valid && i < n; ++i) {
+                const unsigned char continuation = static_cast<unsigned char>(p[i]);
+                valid                            = continuation >= 0x80 && continuation <= 0xBF;
+            }
+            if (valid && n == 3) {
+                const unsigned char second = static_cast<unsigned char>(p[1]);
+                valid                      = (c != 0xE0 || second >= 0xA0) && (c != 0xED || second <= 0x9F);
+            } else if (valid && n == 4) {
+                const unsigned char second = static_cast<unsigned char>(p[1]);
+                valid                      = (c != 0xF0 || second >= 0x90) && (c != 0xF4 || second <= 0x8F);
+            }
+            if (valid) {
+                out.append(p, static_cast<size_t>(n));
+                p += n;
+            } else {
+                out += "\xEF\xBF\xBD";
+                ++p;
+            }
         }
     }
     return out;
@@ -123,6 +160,85 @@ std::string batch_segments_json(const transcribe_session * ctx, int i) {
     return out;
 }
 
+const char * timestamp_kind_json_name(transcribe_timestamp_kind kind) {
+    switch (kind) {
+        case TRANSCRIBE_TIMESTAMPS_NONE:
+            return "none";
+        case TRANSCRIBE_TIMESTAMPS_SEGMENT:
+            return "segment";
+        case TRANSCRIBE_TIMESTAMPS_WORD:
+            return "word";
+        case TRANSCRIBE_TIMESTAMPS_TOKEN:
+            return "token";
+        case TRANSCRIBE_TIMESTAMPS_AUTO:
+            return "auto";
+    }
+    return "unknown";
+}
+
+bool write_result_json(const std::string &        path,
+                       const transcribe_session * ctx,
+                       const std::string &        language,
+                       transcribe_status          status) {
+    const char *       detected  = transcribe_detected_language(ctx);
+    const char *       lang      = (detected != nullptr && detected[0] != '\0') ? detected : language.c_str();
+    const auto         kind      = transcribe_returned_timestamp_kind(ctx);
+    const bool         truncated = transcribe_was_truncated(ctx);
+    std::ostringstream json;
+    json << "{\"status\":\"" << json_escape(transcribe_status_string(status))
+         << "\",\"truncated\":" << (truncated ? "true" : "false")
+         << ",\"aborted\":" << (status == TRANSCRIBE_ERR_ABORTED ? "true" : "false") << ",\"text\":\""
+         << json_escape(transcribe_full_text(ctx)) << "\",\"language\":\"" << json_escape(lang)
+         << "\",\"timestamp_kind\":\"" << timestamp_kind_json_name(kind) << "\",\"segments\":[";
+
+    const bool emit_segments = kind != TRANSCRIBE_TIMESTAMPS_NONE || transcribe_n_speaker_segments(ctx) > 0;
+    const int  n_segments    = emit_segments ? transcribe_n_segments(ctx) : 0;
+    for (int i = 0; i < n_segments; ++i) {
+        if (i > 0) {
+            json << ',';
+        }
+        transcribe_segment segment;
+        transcribe_segment_init(&segment);
+        if (transcribe_get_segment(ctx, i, &segment) != TRANSCRIBE_OK) {
+            std::fprintf(stderr, "error: cannot read segment %d for JSON output\n", i);
+            return false;
+        }
+        json << "{\"t0_ms\":" << segment.t0_ms << ",\"t1_ms\":" << segment.t1_ms;
+        if (segment.speaker_id > 0) {
+            json << ",\"speaker_id\":" << segment.speaker_id;
+        }
+        json << ",\"text\":\"" << json_escape(segment.text) << "\",\"words\":[";
+        for (int j = 0; j < segment.n_words; ++j) {
+            if (j > 0) {
+                json << ',';
+            }
+            transcribe_word word;
+            transcribe_word_init(&word);
+            if (transcribe_get_word(ctx, segment.first_word + j, &word) != TRANSCRIBE_OK) {
+                std::fprintf(stderr, "error: cannot read word %d for JSON output\n", segment.first_word + j);
+                return false;
+            }
+            json << "{\"t0_ms\":" << word.t0_ms << ",\"t1_ms\":" << word.t1_ms << ",\"text\":\""
+                 << json_escape(word.text) << "\"}";
+        }
+        json << "]}";
+    }
+    json << "]}\n";
+
+    std::ofstream out(std::filesystem::u8path(path), std::ios::binary | std::ios::trunc);
+    if (!out) {
+        std::fprintf(stderr, "error: cannot open JSON output %s\n", path.c_str());
+        return false;
+    }
+    out << json.str();
+    out.close();
+    if (out.fail()) {
+        std::fprintf(stderr, "error: failed writing JSON output %s\n", path.c_str());
+        return false;
+    }
+    return true;
+}
+
 // ",\"raw_text\":\"...\"" fragment: the model's pre-cleanup decode
 // (transcribe_raw_text). Emitted only when it differs from the clean text,
 // so JSONL stays compact for families with no post-processing.
@@ -197,6 +313,7 @@ struct cli_args {
     std::string                language;
     std::string                target_language;   // --target-language: target lang for translation
     std::string                batch_file;        // --batch: one wav path per line
+    std::string                output_json_path;  // --output-json: structured single-file result
     int                        batch_size   = 0;  // --batch-size: >1 groups utterances into
                                                   // transcribe_run_batch calls (offline only).
                                                   // 0/1 keeps the per-file serial loop.
@@ -295,6 +412,8 @@ void print_usage(const char * argv0) {
                  "  --timestamps TYPE     timestamps: auto, none, segment, word, token (default: auto)\n"
                  "  --batch FILE          batch mode: FILE has one wav path per line\n"
                  "  --batch-jsonl         output one JSON line per file (for batch)\n"
+                 "  --output-json PATH    write a structured single-file result with nested\n"
+                 "                        segment and word timings in milliseconds\n"
                  "  --batch-size N        group N utterances into one transcribe_run_batch\n"
                  "                        call (offline only; 0/1 = per-file serial loop)\n"
                  "  --initial-prompt TEXT (whisper) initial prompt text for context biasing\n"
@@ -518,6 +637,12 @@ bool parse_args(int argc, char ** argv, cli_args & out) {
             out.batch_file = v;
         } else if (a == "--batch-jsonl") {
             out.batch_jsonl = true;
+        } else if (a == "--output-json") {
+            const char * v = take_value(a.c_str());
+            if (!v) {
+                return false;
+            }
+            out.output_json_path = v;
         } else if (a == "--batch-size") {
             const char * v = take_value(a.c_str());
             if (!v) {
@@ -656,6 +781,14 @@ bool parse_args(int argc, char ** argv, cli_args & out) {
         std::fprintf(stderr, "error: cannot combine positional audio.wav with --batch\n");
         return false;
     }
+    if (!out.output_json_path.empty() && !out.batch_file.empty()) {
+        std::fprintf(stderr, "error: --output-json is only available for single-file runs\n");
+        return false;
+    }
+    if (!out.output_json_path.empty() && out.model_path.empty()) {
+        std::fprintf(stderr, "error: --output-json requires --model\n");
+        return false;
+    }
     if (out.stream_chunk_ms > 0 && out.repeat > 1) {
         std::fprintf(stderr, "error: --stream-chunk-ms cannot be combined with --repeat\n");
         return false;
@@ -1236,7 +1369,8 @@ int main(int argc, char ** argv) {
         // CLI prints the live tentative text on each such feed.
         // Families that only commit at finalize keep result_changed
         // false until the finalize call.
-        transcribe_status run_st = TRANSCRIBE_OK;
+        transcribe_status run_st        = TRANSCRIBE_OK;
+        bool              json_write_ok = true;
         if (args.stream_chunk_ms > 0) {
             struct transcribe_capabilities caps;
             transcribe_capabilities_init(&caps);
@@ -1400,6 +1534,10 @@ int main(int argc, char ** argv) {
                                 (tok.text != nullptr) ? tok.text : "");
                 }
             }
+            if (!args.output_json_path.empty() &&
+                !write_result_json(args.output_json_path, ctx, args.language, run_st)) {
+                json_write_ok = false;
+            }
         }
 
         transcribe_print_timings(ctx);
@@ -1418,7 +1556,7 @@ int main(int argc, char ** argv) {
         transcribe_session_free(ctx);
         transcribe_model_free(model);
 
-        if (run_st != TRANSCRIBE_OK) {
+        if (run_st != TRANSCRIBE_OK || !json_write_ok) {
             return EXIT_FAILURE;
         }
     } else {