Skip to content

fix: replace hand-rolled quickselect with select_nth_unstable_by - #407

Open
agene0001 wants to merge 4 commits into
statrs-dev:mainfrom
agene0001:fix/163-quickselect-nan
Open

fix: replace hand-rolled quickselect with select_nth_unstable_by#407
agene0001 wants to merge 4 commits into
statrs-dev:mainfrom
agene0001:fix/163-quickselect-nan

Conversation

@agene0001

@agene0001 agene0001 commented Jul 26, 2026

Copy link
Copy Markdown

select_inplace implements the Numerical Recipes quickselect by hand. With NaN in the slice, its partition loop walks past the end of the array — the crash reported in #163 (a segfault under the pre-0.13 unsafe version, an index-out-of-bounds panic today). Reproducer that panics on main:

let mut v: Vec<f64> = (0..1000).map(|i| (i as f64 * 0.7).sin()).collect();
for i in (0..1000).step_by(7) { v[i] = f64::NAN; }
Data::new(v).quantile(0.5); // panicked at src/statistics/slice_statistics.rs:106:28

Replacing it with <[T]>::select_nth_unstable_by(rank, f64::total_cmp) fixes it two ways:

  • NaN gets a defined position in a total order, so the search stays in bounds (previous behaviour with NaN was arbitrary).
  • Worst case is O(n) (introselect) instead of the NR quickselect's quadratic worst case.

Performance

The first version of this PR claimed ~4.4x. That was measured on large random input only, and it was wrong as a general statement — thanks @day01 for asking for a reproducible benchmark, which is what surfaced it.

Shape matters as much as size. The NR quickselect took its pivot from a median of three, which on sorted or reverse-sorted input lands on the true median and finishes in a single partition, so it beat plain introselect on exactly those shapes. This PR therefore checks for order first, which recovers that and improves on it — the answer becomes an index rather than a partition.

benches/order_statistics.rs gains a selection scaling group sweeping size against shape, written against only the public Data API so it compiles on any branch and --save-baseline works across a checkout. Against main, medians of the Criterion intervals:

shape 1024 65536 1048576
random +12% −36% −76%
sorted −3% −0% +15%
reversed −37% −29% −26%
organ_pipe −46% −64% −71%
duplicates −19% −64% −79%

Twelve of fifteen improve. Sorted input is roughly a wash below a million elements and 15% slower at a million, where the total_cmp scan costs more per element than the median-of-three pivot it replaces. Small random input is 12% slower, where introselect's setup costs more than the old quickselect's.

Keeping the fast paths still beats dropping them: without them, sorted is +75% to +89% and reversed +13% to +30% against main.

Tests

The regression test covers NaN scattered through a large slice, all-NaN input, and NaN at the exact positions the old partition tripped on.

Added on review: every order statistic, the median, and the quantile endpoints checked against a sorted reference over eight shapes and nine sizes, plus permutation invariance. These compare bit patterns, not values, so a wrong sign of zero is a failure — which is how the signed_zeros shape pins @day01's catch that <= considers [-1.0, +0.0, -0.0, 1.0] sorted and returns the wrong zero.

Existing order-statistic/median/quantile tests (including test_median_robust_on_infinities) pass unchanged.

Closes #163

🤖 Generated with Claude Code

`select_inplace` implemented the Numerical Recipes quickselect by hand. With
NaN in the slice its partition loop walks past the end of the array, which is
the crash reported in statrs-dev#163 (a segfault under the older unsafe version, an
index-out-of-bounds panic today):

    let mut v: Vec<f64> = (0..1000).map(|i| (i as f64 * 0.7).sin()).collect();
    for i in (0..1000).step_by(7) { v[i] = f64::NAN; }
    Data::new(v).quantile(0.5);   // panics on main

`<[T]>::select_nth_unstable_by` with `total_cmp` fixes this three ways: NaN
gets a defined position in a total order, the worst case is O(n) rather than
quadratic, and it is ~4.4x faster at 1e6 elements.

Closes statrs-dev#163
agene0001 added a commit to agene0001/statrs that referenced this pull request Jul 26, 2026
Brings the integration branch to the exact union of PRs statrs-dev#407-statrs-dev#420: adds the
NaN regression test written for statrs-dev#407 (statrs-dev#163), relocates the
Geometric near-1 test to match the prec PR, and aligns two comment/tolerance
details.
The test allocated its input with `Vec`, which does not exist under
`--no-default-features`: the crate is `no_std` without `alloc`, and there is
no `extern crate alloc`. `core::array::from_fn` and array literals give the
same data with no allocation.

CI did not catch this because the test job runs only default features, and
the feature-powerset job passes `--no-dev-deps`, so test code is never
compiled without std.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@day01

day01 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Could you extend the existing benches/order_statistics.rs benchmark and include a reproducible Criterion between main and this branch? The PR reports a 4.4x improvement, but currently there is no benchmark setup or i cannot find it.

@day01

day01 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

additionally, based ontest, is focused on edge case, can you test positive scenario with values?

Follow-up to @day01's review on statrs-dev#407.

Adds a `selection scaling` group to benches/order_statistics.rs sweeping size
against input shape, written against only the public `Data` API so the same
file compiles on any branch and `--save-baseline` works across a checkout.
The existing group measures one fixed size on random data, which is not enough
to reproduce a performance claim.

Running it against main showed the claim in this PR was wrong. The shapes are
why: the Numerical Recipes quickselect took its pivot from a median of three,
which on sorted or reverse-sorted input lands on the true median and finishes
in one partition. It beat plain introselect on exactly those shapes -- by 75
to 89% on sorted input at every size measured -- while losing badly on large
random, organ-pipe and duplicate-heavy input. The honest summary of the
original change was "much faster on some shapes, nearly twice as slow on
others", not "~4x faster".

So test for order first, which recovers what median-of-three was doing and
improves on it: the answer becomes an index rather than a partition. Both
checks stop at the first out-of-order pair, so unordered input pays a couple
of comparisons, and `median` and `quantile` benefit twice since they select
twice. `<=` is used rather than `total_cmp`, which is cheaper per element and
still correct -- any NaN makes some comparison false, so a slice containing
one never takes a fast path.

With that, 14 of 15 cases beat main (medians of the Criterion intervals):

    shape        1024     65536    1048576
    random       +19%     -28%     -76%
    sorted       -34%     -30%     -24%
    reversed     -56%     -51%     -52%
    organ_pipe   -46%     -65%     -71%
    duplicates   -15%     -63%     -78%

The exception is small random input, where introselect's setup costs more than
the old quickselect's; 2.0us against 2.4us at n = 1024. Left alone, since the
reason for this PR is that the NR partition walks off the end of the array on
NaN, not throughput.

Also adds the positive-value tests asked for. The existing ones cover NaN,
infinities and constant sequences -- they pin that nothing panics but say
nothing about the values. The new ones check every order statistic, the
median, and the quantile endpoints against a sorted reference over seven
shapes and nine sizes, plus permutation invariance. These cover the new fast
paths, which return an index rather than partitioning: mutating the descending
path to index from the wrong end fails both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Comment thread benches/order_statistics.rs Outdated
/// file compiles on any branch. To compare two implementations:
///
/// ```text
/// git checkout main && cargo bench --bench order_statistics -- --save-baseline main

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it shouldnt be here, i asked only for results of benches. like in one other of your prs I prepared.

Comment thread src/statistics/slice_statistics.rs Outdated
// containing one never takes a fast path and falls through to the
// general case, and without NaN the two orderings agree.
let slice = self.0.as_mut();
if slice.is_sorted_by(|a, b| a <= b) {

@day01 day01 Jul 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

<= does not define the same ordering as total_cmp for signed zero. For [-1.0, +0.0, -0.0, 1.0], order_statistic(2) returns +0.0, while the total_cmp reference is -0.0. Please use total_cmp consistently for both fast paths or remove them.

Comment thread src/statistics/slice_statistics.rs Outdated
if slice.is_sorted_by(|a, b| a <= b) {
return slice[rank];
}
if slice.is_sorted_by(|a, b| a >= b) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as up

Comment thread src/statistics/slice_statistics.rs Outdated
if end <= rank {
low = begin;
}
// `<=` rather than `total_cmp` here, which is both cheaper per element

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude things :)

Comment thread src/statistics/slice_statistics.rs Outdated
/// Selection agrees with sorting, on ordinary values across a range of
/// shapes and sizes.
///
/// The counterpart to the NaN and infinity tests below, which pin that

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

may we make it more human desc?

Comment thread src/statistics/slice_statistics.rs Outdated
// NaNs sort past +inf) rather than the arbitrary result the previous
// Numerical Recipes partition produced, which is the point of the change.
//
// The ordered fast paths are not incidental. The NR quickselect took its

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this desc is needed r u sure?

Comment thread src/statistics/slice_statistics.rs Outdated
fn test_order_statistics_match_sorted_reference() {
fn shaped(shape: &str, n: usize) -> Vec<f64> {
match shape {
// deterministic pseudo-random, no rand dependency needed

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have rand, i dont know why we have to forget about rand in tests, deps are welcome especially in tests

let data = shaped(shape, n);
group.throughput(Throughput::Elements(n as u64));
group.bench_function(format!("median/{shape}/{n}"), |b| {
b.iter_batched(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

iter_batched excludes data.clone() from the measurement. Are the numbers in the PR description from a different benchmark?

@day01 is right, and the counterexample is exactly the one given: `<=`
considers `[-1.0, +0.0, -0.0, 1.0]` sorted, because `+0.0 <= -0.0` holds, so
the fast path indexed straight into it and `order_statistic(2)` returned
`+0.0` where the `total_cmp` reference gives `-0.0`. My reasoning for `<=`
covered NaN and missed signed zero entirely.

Both fast paths now use `total_cmp`, matching the ordering the general path
selects by. The value tests gain a `signed_zeros` shape and compare bit
patterns rather than values, since `-0.0 == +0.0` would have hidden this;
restoring `<=` now fails with `signed_zeros/3: order_statistic(2) = 0, want
-0`.

That costs the cheaper scan, so the benchmark moves. Against main, medians of
the Criterion intervals:

    shape        1024     65536    1048576
    random       +12%     -36%     -76%
    sorted        -3%      -0%     +15%
    reversed     -37%     -29%     -26%
    organ_pipe   -46%     -64%     -71%
    duplicates   -19%     -64%     -79%

Twelve of fifteen improve. Sorted input is now roughly a wash below a million
elements and 15% slower at a million, where the `total_cmp` scan costs more
per element than the median-of-three pivot it replaces. Keeping the fast paths
is still much better than dropping them, which would put sorted back at +75%
to +89% and reversed at +13% to +30%.

Also drops the run instructions from the benchmark file and trims the
commentary, per review.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@agene0001

Copy link
Copy Markdown
Author

Good catch on the signed zero — that's a real bug and you gave the exact counterexample. <= reports [-1.0, +0.0, -0.0, 1.0] as sorted because +0.0 <= -0.0 holds, so the fast path indexed straight in and returned +0.0 where the total_cmp reference gives -0.0. My reasoning for <= covered NaN and missed signed zero completely.

Both fast paths now use total_cmp. The value tests gained a signed_zeros shape and now compare bit patterns rather than values, since -0.0 == +0.0 would have hidden it — restoring <= fails with signed_zeros/3: order_statistic(2) = 0, want -0.

On iter_batched: yes, the clone is excluded, and that's intended — the measurement should be selection, not allocation. The numbers came from this benchmark via --baseline main, but the ones in the description were from the <= version, so they were stale. total_cmp costs more per element; current figures (medians of the Criterion intervals), and I've updated the description to match:

shape 1024 65536 1048576
random +12% −36% −76%
sorted −3% −0% +15%
reversed −37% −29% −26%
organ_pipe −46% −64% −71%
duplicates −19% −64% −79%

Twelve of fifteen improve. Sorted input is roughly a wash below a million elements and 15% slower at a million, where the total_cmp scan costs more than the median-of-three pivot it replaces. Keeping the fast paths still beats dropping them, which would put sorted back at +75% to +89% and reversed at +13% to +30% — but if you'd rather have the simpler code, say so and I'll drop them.

Also removed the run instructions from the bench file and trimmed the commentary, and switched the tests to rand.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

segfault in select_inplace from v0.12

2 participants