feat(parquet): add WithDictionaryCostFallback writer property - #1036
Open
tom-eon wants to merge 1 commit into
Open
feat(parquet): add WithDictionaryCostFallback writer property#1036tom-eon wants to merge 1 commit into
tom-eon wants to merge 1 commit into
Conversation
tom-eon
force-pushed
the
parquet-dict-cost-fallback-option
branch
from
July 28, 2026 14:38
76fdf25 to
f2a7481
Compare
The pre-first-page cost-based dictionary fallback compares uncompressed sizes, so it can discard a dictionary that would have won after page compression. Allow callers that decide dictionary usage themselves to opt out; the DictionaryPageSizeLimit fallback is unaffected.
tom-eon
force-pushed
the
parquet-dict-cost-fallback-option
branch
from
July 28, 2026 15:18
f2a7481 to
6135548
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
The pre-first-page dictionary cost check (
checkDictionarySizeLimit, mirroring parquet-mr'sFallbackValuesWriter.shouldFallBack) compares uncompressed sizes: the dictionary is discarded whendictSize + encodedSize >= rawSize. With a page compressor configured this can discard dictionaries that clearly win after compression — bit-packed dictionary indices are near-incompressible but small, while PLAIN-encoded values often compress far less than the raw-size comparison assumes. In our benchmarks (ZSTD level 3), a mid-cardinalityTime64column written PLAIN after the fallback was 2× larger than the same column dictionary-encoded (7.1 MB vs 3.6 MB compressed); mid-cardinality integer columns lost 20%+.Callers that decide dictionary usage from their own measurements (e.g. by trial-encoding a sample of the actual data with the actual codec) currently have no way to make an explicit
WithDictionaryFor(col, true)stick — the heuristic silently overrides it.Alternative considered: making the check itself compression-aware — e.g. trial-compressing both branches of the first data page and keeping the winner, or letting callers supply per-column compressibility estimates for the comparison. That fixes the misprediction for everyone rather than only for callers who opt out, but it costs an extra encode+compress of the first page per column (or a new estimate-supplying API surface), needs a guard band to avoid paying it on clear-cut cases, and still guesses for callers who already measured. The opt-out is minimal, preserves the default exactly, and doesn't preclude a compression-aware check later — if there's interest in that direction I'm happy to discuss it in a follow-up.
What changes are included in this PR?
A new per-column writer property following the existing global/per-column property pattern (like
WithDictionaryDefault/WithDictionaryFor):parquet.WithDictionaryCostFallback(enabled bool)— sets the default for all columns (stored onColumnProperties, defaulttrue, existing behavior unchanged);parquet.WithDictionaryCostFallbackFor(path string, enabled bool)/WithDictionaryCostFallbackPath(path ColumnPath, enabled bool)— per-column override;WriterProperties.DictionaryCostFallbackEnabledFor(path string) bool— resolution, per-column entry falling back to the default.When disabled for a column, the pre-first-page cost-based fallback is skipped for that column, so a requested dictionary is kept; the
DictionaryPageSizeLimitoverflow fallback remains in effect regardless of the setting. The per-column granularity matters for callers that measure encodings per column: columns they measured keep their explicit choice, while unmeasured columns retain the heuristic as a safety net.Are these changes tested?
Yes — new tests in
parquet/file/dict_cost_fallback_test.gocovering: (1) default behavior still discards a non-paying dictionary, (2) with the option disabled an explicitly requested dictionary is kept, (3) a per-column disable keeps that column's dictionary while (4) a disable scoped to a different column leaves the fallback active, and (5) theDictionaryPageSizeLimitfallback still applies when the cost fallback is disabled. Existingparquet/fileandparquet/internal/encodingsuites pass.Are there any user-facing changes?
New opt-in writer properties; default behavior is unchanged.