Enable parquet metadata prefetching by default - #23307
Closed
TomAugspurger wants to merge 4 commits into
Closed
Conversation
This reverts commit 3c56d8d.
…data members (NVIDIA#23267) Route IO builders through option setters because they are responsible for validation. Also fixed validation in `set_quoting`. Authors: - Vukasin Milovanovic (https://github.com/vuule) Approvers: - Muhammad Haseeb (https://github.com/mhaseeb123) - David Wendt (https://github.com/davidwendt) - Misiu Godfrey (https://github.com/misiugodfrey) URL: NVIDIA#23267
…IA#23147) Closes NVIDIA#23146 This PR adds a `direct_inner_join` free function to libcudf, the first step of the perfect hash join effort in NVIDIA#23126. The keys are a single `UINT32` column per side, produced by a prior perfect hashing pass such as `cudf::key_remapping`, dictionary encoding, or dense integer primary keys. The right keys act as a perfect hash of the right rows: a lookup table of caller-specified `capacity` entries maps each key value to its row index and each left key probes that table directly, so the join performs no hashing or key comparison at all. - The caller controls the memory footprint via the explicit `capacity` argument. All key values must be in `[0, capacity)` and the right keys must be distinct; behavior is undefined otherwise. - Inner join only, as a free function: no table reuse across probes is needed, so there is no join object. - The build scatters right row indices into the lookup table with `cub::DeviceTransform::Fill` + `cub::DeviceFor::Bulk`; the probe is a single `cub`-based `copy_if` pass emitting the matched index pairs. - A new `JOIN_NVBENCH` benchmark compares `inner_join`, `distinct_hash_join`, and `direct_inner_join` on identical conforming input; results in the comment below. Authors: - Yunsong Wang (https://github.com/PointKernel) Approvers: - Tianyu Liu (https://github.com/kingcrimsontianyu) - Shruti Shivakumar (https://github.com/shrshi) - Muhammad Haseeb (https://github.com/mhaseeb123) URL: NVIDIA#23147
This updates the cudf-polars configuration to enable parquet metadata
prefetching by default. It can be disabled through the environment
variable
CUDF_POLARS__PARQUET_OPTIONS__PREFETCH_FILE_METADATA=0
or through Python when constructing the ParquetOptions.
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
github-actions
Bot
added
Python
TomAugspurger
added
improvement
Contributor
Author
|
I'm seeing a surprising amount of GIL contention during metadata prefetching: 
I think the majority of this is around |
Contributor
Author
|
Subsumed by #23558. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This updates the cudf-polars configuration to enable parquet metadata prefetching by default. It can be disabled through the environment variable
or through Python when constructing the ParquetOptions.
We need to do some benchmarking to make sure this is beneficial everywhere.
Closes #22826
Description
Checklist