[TRTLLM-14778][perf] Add feature-mode encoder CUDA graphs for fixed-shape encoders (Whisper)

docs/source/models/encoder-decoder.md

@@ -45,7 +45,7 @@ The following table describes the supported and recommended configurations.
 | Beam search | Yes with V1 | Configure `max_beam_width` when constructing `LLM`, then set `use_beam_search=True` in `SamplingParams`. |
 | Attention backend | `TRTLLM` | Use this backend for encoder-decoder models. It is required when `tensor_parallel_size > 1`. |
 | Decoder CUDA graphs | Yes, except in FP32 | `CudaGraphConfig` captures decoder work. V1 supports greedy and beam search; V2 supports its single-beam path. FP32 encoder-decoder models decline capture at engine init and log a warning instead of failing. |
-| Encoder CUDA graphs | Yes | Set `encoder_cuda_graph_config=EncodeCudaGraphConfig(...)` and `encoder_max_batch_size`. Usually set `encoder_max_batch_size` lower than `max_batch_size`. The `TRTLLM` attention backend is required. |
+| Encoder CUDA graphs | Yes | Set `encoder_cuda_graph_config=EncodeCudaGraphConfig(...)` and `encoder_max_batch_size`. Usually set `encoder_max_batch_size` lower than `max_batch_size`. The `TRTLLM` attention backend is required. Text encoders also require `num_tokens` and `seq_lens`; a fixed-shape feature encoder such as Whisper derives both from the model and needs only `batch_sizes`. |
 | Overlap scheduler | Yes | Enabled by default. V1 supports greedy decoding and beam search; V2 remains limited to `max_beam_width=1`. |
 | Tensor parallelism | Yes | Use `tensor_parallel_size > 1` with `attn_backend="TRTLLM"`. Attention head counts must be divisible by the TP size. |
 | Pipeline parallelism | No | Keep `pipeline_parallel_size=1`. |
@@ -437,6 +437,34 @@ size, total packed tokens, and maximum sequence length. The
 limit. With beam search, decoder graph batch sizes must cover the active
 decoder sequences after beam expansion.
 
+Which encoder buckets you must supply depends on the model. A text encoder,
+such as BART or T5, packs a variable number of tokens per request, so
+`num_tokens` and `seq_lens` are part of its key space and are required; leaving
+either unset is an error at engine construction. An encoder whose input is a
+fixed-shape per-request feature tensor, such as Whisper's audio waveform,
+produces the same number of encoder positions for every request, so both lists
+follow from the model and are derived rather than configured. For those models
+`batch_sizes` alone enables capture, and any `num_tokens` or `seq_lens` you set
+is ignored:
+
+```python
+from tensorrt_llm.llmapi import EncodeCudaGraphConfig
+
+
+llm = LLM(
+    model="openai/whisper-large-v3",
+    backend="pytorch",
+    attn_backend="TRTLLM",
+    max_batch_size=8,
+    encoder_max_batch_size=8,
+    encoder_cuda_graph_config=EncodeCudaGraphConfig(batch_sizes=[1, 2, 4, 8]),
+    # ... the remaining Whisper settings from "Transcribe audio with Whisper"
+)
+```
+
+Batch sizes that do not fit `encoder_max_num_tokens` divided by the model's
+encoder output length are dropped, and the encoder stays eager when none fit.
+
 `max_batch_size` controls the total decoder concurrency, while
 `encoder_max_batch_size` controls encoder microbatch admission. For better
 performance, tune `encoder_max_batch_size`, `encoder_max_num_tokens`, and the
@@ -656,6 +684,13 @@ that the encoder graph buckets cover the request shape, and that
 `attn_backend="TRTLLM"`. Unsupported shapes and attention backends fall back to
 eager encoder execution.
 
+### `num_tokens` or `seq_lens` unset is rejected at engine construction
+
+A text encoder needs both bucket lists, so the engine raises rather than
+silently running eager. Supply them, or drop `encoder_cuda_graph_config` if you
+do not want encoder graphs. A fixed-shape feature encoder such as Whisper does
+not hit this: it derives both from the model and needs only `batch_sizes`.
+
 ### Output quality differs from the Hugging Face example
 
 Confirm that the source uses the task prefix and language settings expected by

tensorrt_llm/_torch/models/modeling_whisper.py

@@ -654,6 +654,17 @@ def _build_decoder_prompt(self) -> List[int]:
             forced = self.processor.get_decoder_prompt_ids(no_timestamps=True)
         return [int(start_id)] + [int(tok) for _, tok in sorted(forced)]
 
+    def get_decoder_prefix_len(self) -> int:
+        """Tokens every request's decoder prompt starts with.
+
+        Mixed encoder/decoder CUDA graphs capture at this query length, and a
+        mismatch makes every mixed batch miss its graph silently. This reports
+        the checkpoint default, so a request carrying a text prompt of a
+        different length (see `_resolve_decoder_prompt`) misses the mixed graph
+        and runs that batch eagerly.
+        """
+        return len(self._build_decoder_prompt())
+
     def _resolve_decoder_prompt(self, prompt_text: Optional[str]) -> List[int]:
         """Checkpoint-default forced prompt, or the user's decoder prompt.
 
@@ -880,6 +891,27 @@ def __pp_init__(self):
     def config(self):
         return self.model_config.pretrained_config
 
+    def encoder_graph_spec(self) -> Tuple[Tuple[int, ...], torch.dtype, int]:
+        """Fixed-shape encoder contract for enc-dec encoder CUDA graphs.
+
+        Every Whisper encoder request is an fp32 waveform zero-padded by
+        `WhisperInputProcessor` to the fixed window that yields exactly
+        ``max_source_positions`` encoder positions — so the encoder graph key
+        degenerates to the batch size.
+
+        The window is derived rather than hardcoded: the conv stem halves the
+        STFT frame count, so ``n_samples = max_source_positions * 2 *
+        hop_length``. ``hop_length`` comes from the checkpoint's feature
+        extractor, the same source the input processor validates its own
+        window against.
+
+        Returns ``(per_request_feature_shape, dtype, fixed_seq_len)``.
+        """
+        fixed_seq_len = int(self.config.max_source_positions)
+        hop_length = int(self.model.encoder.log_mel.hop_length)
+        n_samples = fixed_seq_len * 2 * hop_length
+        return ((n_samples,), torch.float32, fixed_seq_len)
+
     def forward(
         self,
         attn_metadata: AttentionMetadata,