Skip to content

Add CLI option to set target-time parameter for batch measurement loop - #439

Open
oleksandr-pavlyk wants to merge 12 commits into
NVIDIA:mainfrom
oleksandr-pavlyk:support-batch-target-time
Open

Add CLI option to set target-time parameter for batch measurement loop#439
oleksandr-pavlyk wants to merge 12 commits into
NVIDIA:mainfrom
oleksandr-pavlyk:support-batch-target-time

Conversation

@oleksandr-pavlyk

@oleksandr-pavlyk oleksandr-pavlyk commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

new --batch-target-time CLI option

This PR adds support for --batch-target-time CLI option:

* `--batch-target-time <seconds>`
  * Target accumulated GPU time for batched measurements.
  * Default is 0.5 seconds.
  * `<seconds>` must be finite and positive.
  * Batched measurements continue until both `--min-samples` and the
    accumulated GPU-time target are satisfied, unless `--timeout` is reached
    first.
  * Applies to the most recent `--benchmark`, or all benchmarks if specified
    before any `--benchmark` arguments.

This option controls batch size in batch measurement loop by specifying its target duration.

Batch measurement loop termination condition did not change: both the number of launches executed and the time executing them should exceed thresholds: num_samples >= m_min_samples && total_cuda_time >= m_batch_target_time.

Improved error on unrecognized CLI option

An error handling of unrecognized CLI options is improved:

$ ./build/bin/nvbench.example.cpp17.throughput --stopping-criterion entropy --min-time 1

NVBench encountered an error:

nvbench/option_parser.cu:1180: Error handling option `--min-time 1`:
nvbench/option_parser.cu:141: --min-time is not valid for the active stopping criterion 'entropy'.
  Current criterion 'entropy' accepts: --max-angle, --min-r2.

Change in behavior of batch measurements when no prior cold measurements exist

We used to overwrite the value like so: m_min_time = std::max( std::midpoint(m_min_time, m_timeout), m_min_time * 5);.

In default settings (min-time of 0.5 seconds and timeout of 15 seconds) this would change value to 2.5 seconds.
Given then user may hit this branch by using state::exec(exec_tag::hot, launchable); in the benchmark generator, this behavior is very surprising when --batch-target-time is explicitly specified.

For this reason, and because the option is now explicit, this overwrite has been removed.

Relationship to other work

This PR is an alternative to #424. The advantages of this solutions are:

  • --batch-target-time option is global, i.e., it is valid irrespective of which stopping criterion is considered active by option parser.
  • it allows controlling batched measurements loop independently from specifying stopping criterion which is only applied to cold/cpu_only measurements loops
  • --min-time option continues on as stdrel-criterion parameter.

Closes #408

The option specifies target accumulated GPU time for batched
measurements.

This option is used to determine batch size and to decide
when batched measurements loop terminates. (total accummulated
time must exceed the target time and the number of launches
executed should >= min_samples).
If time estimate results in batch size smaller
than m_min_samples, use that instead so that batch
does not degenerated into single launch and one does
not observe the benefit of using warm L2-cache.

Setting it this way permits batched measurement to complete
in a single iteration of out loop.

Also, update batch_size only after exit condition has been
checked, and we know that m_batch_target_time is greater
than m_total_cuda_time. Handle the possibility of `m_total_cuda_time`
being zero.
@oleksandr-pavlyk

This comment was marked as outdated.

@coderabbitai

This comment was marked as outdated.

@coderabbitai

coderabbitai Bot commented Jul 28, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Summary by CodeRabbit

  • New Features
    • Added configurable batched-measurement targeting via the --batch-target-time CLI option (default 0.5s).
    • Exposed batch target time in C++/Python benchmark/state APIs and option decorators.
    • Included batch_target_time in emitted JSON results.
  • Improvements
    • Updated hot-trial batching to stop based on accumulated GPU-time vs batch target time, with more robust batch-size prediction.
    • Refreshed timeout/warning messaging and related “skip time” help to reference batch target time.
  • Bug Fixes
    • Added validation to reject non-finite or non-positive values, with expanded parsing/decorator tests.

Walkthrough

Adds configurable batch target time for GPU measurements through C++ and Python APIs, CLI parsing, JSON serialization, hot-measurement scheduling, documentation, and tests.

Changes

Batched measurements

Layer / File(s) Summary
Batch target configuration contracts
nvbench/detail/validate_batch_target_time.cuh, nvbench/benchmark_base.*, nvbench/state.*, nvbench/json_printer.cu
Benchmark and state objects validate, store, expose, initialize, clone, and serialize batch target time.
Hot measurement batch scheduling
nvbench/detail/measure_hot.*
Hot measurements predict bounded batch sizes and stop after target CUDA time and minimum sample count are reached.
CLI parsing and documentation
nvbench/option_parser.cu, docs/cli_help.md
The CLI parses and validates --batch-target-time, reports active criterion parameters consistently, and documents the option and timeout criteria.
Python API and validation coverage
python/src/py_nvbench.cpp, python/cuda/bench/_decorators.py, python/test/test_cuda_bench.py, testing/option_parser.cu
Python bindings and decorators expose batch target time, with tests for APIs, decorators, documentation, valid values, invalid inputs, and criterion-parameter errors.

Assessment against linked issues

Objective Addressed Explanation
Provide an explicit batch measurement option for batched minimum-time configuration [#408]
Reject invalid criterion parameters with an error identifying the active criterion and accepted parameters [#408]
Move the batched minimum-time setting out of the stdrel criterion model [#408] The new --batch-target-time option is modeled independently, but the existing --min-time criterion parameter remains in the CLI model.

Comment @coderabbitai help to get the list of available commands.

@oleksandr-pavlyk

This comment was marked as outdated.

@coderabbitai

This comment was marked as outdated.

coderabbitai[bot]

This comment was marked as resolved.

When an unknown CLI option is encountered, we try it as a stopping
criterion parameter. The error message now states the option that
tripped the error, active stopping criterion and its parameters
@oleksandr-pavlyk

This comment was marked as outdated.

@coderabbitai

This comment was marked as outdated.

@oleksandr-pavlyk oleksandr-pavlyk changed the title Add CLI option to set targe-time parameter for batch measurement loop Add CLI option to set target-time parameter for batch measurement loop Jul 28, 2026
coderabbitai[bot]

This comment was marked as resolved.

@oleksandr-pavlyk

This comment was marked as outdated.

@coderabbitai

This comment was marked as outdated.

coderabbitai[bot]

This comment was marked as resolved.

The floor existed to prevent batch-size from degenerating to 1,
which we now cap at 4 launches, so use of time floor is unnecessary.

Removed no-cold measurements overwrite of batch target time, as it
leads to non-transparent to the user significant increase in the
target time used (say target time is default at 0.5 seconds, and
timeout is default at 15 seconds, the overwrite would bump target
time to 2.5 seconds).

Present day NVBench only starts measure_hot after measure_cold has
completed by default. If test is skipped/encountered errors during
cold measurement loop, the measure_hot is not reached.

The only way for a user to do batched measurements with cold
measurments unavailable would be to use

state::exec(nvbench::exec_tag::hot, launchable);

But even in this case we want to honor requested
`--batch-target-time` value.
@oleksandr-pavlyk

This comment was marked as outdated.

@coderabbitai

This comment was marked as outdated.

coderabbitai[bot]

This comment was marked as resolved.

@oleksandr-pavlyk
oleksandr-pavlyk marked this pull request as ready for review July 28, 2026 15:29
@oleksandr-pavlyk

This comment was marked as outdated.

@coderabbitai

This comment was marked as outdated.

coderabbitai[bot]

This comment was marked as resolved.

@oleksandr-pavlyk oleksandr-pavlyk added the pre-commit.ci autofix Trigger autofix job for precommit.ci gate label Jul 28, 2026
@pre-commit-ci pre-commit-ci Bot removed the pre-commit.ci autofix Trigger autofix job for precommit.ci gate label Jul 28, 2026
@copy-pr-bot

This comment was marked as outdated.

Also test --batch-target-time nan
@oleksandr-pavlyk
oleksandr-pavlyk force-pushed the support-batch-target-time branch from 8708b47 to ece54e4 Compare July 28, 2026 16:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Batched min-time is modeled as a stdrel criterion parameter, causing unintuitive CLI behavior with other stopping criteria

1 participant