Skip to content

feat(parquet): add WithDictionaryCostFallback writer property - #1036

Open
tom-eon wants to merge 1 commit into
apache:mainfrom
tom-eon:parquet-dict-cost-fallback-option
Open

feat(parquet): add WithDictionaryCostFallback writer property#1036
tom-eon wants to merge 1 commit into
apache:mainfrom
tom-eon:parquet-dict-cost-fallback-option

Conversation

@tom-eon

@tom-eon tom-eon commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

The pre-first-page dictionary cost check (checkDictionarySizeLimit, mirroring parquet-mr's FallbackValuesWriter.shouldFallBack) compares uncompressed sizes: the dictionary is discarded when dictSize + encodedSize >= rawSize. With a page compressor configured this can discard dictionaries that clearly win after compression — bit-packed dictionary indices are near-incompressible but small, while PLAIN-encoded values often compress far less than the raw-size comparison assumes. In our benchmarks (ZSTD level 3), a mid-cardinality Time64 column written PLAIN after the fallback was 2× larger than the same column dictionary-encoded (7.1 MB vs 3.6 MB compressed); mid-cardinality integer columns lost 20%+.

Callers that decide dictionary usage from their own measurements (e.g. by trial-encoding a sample of the actual data with the actual codec) currently have no way to make an explicit WithDictionaryFor(col, true) stick — the heuristic silently overrides it.

Alternative considered: making the check itself compression-aware — e.g. trial-compressing both branches of the first data page and keeping the winner, or letting callers supply per-column compressibility estimates for the comparison. That fixes the misprediction for everyone rather than only for callers who opt out, but it costs an extra encode+compress of the first page per column (or a new estimate-supplying API surface), needs a guard band to avoid paying it on clear-cut cases, and still guesses for callers who already measured. The opt-out is minimal, preserves the default exactly, and doesn't preclude a compression-aware check later — if there's interest in that direction I'm happy to discuss it in a follow-up.

What changes are included in this PR?

A new per-column writer property following the existing global/per-column property pattern (like WithDictionaryDefault / WithDictionaryFor):

  • parquet.WithDictionaryCostFallback(enabled bool) — sets the default for all columns (stored on ColumnProperties, default true, existing behavior unchanged);
  • parquet.WithDictionaryCostFallbackFor(path string, enabled bool) / WithDictionaryCostFallbackPath(path ColumnPath, enabled bool) — per-column override;
  • WriterProperties.DictionaryCostFallbackEnabledFor(path string) bool — resolution, per-column entry falling back to the default.

When disabled for a column, the pre-first-page cost-based fallback is skipped for that column, so a requested dictionary is kept; the DictionaryPageSizeLimit overflow fallback remains in effect regardless of the setting. The per-column granularity matters for callers that measure encodings per column: columns they measured keep their explicit choice, while unmeasured columns retain the heuristic as a safety net.

Are these changes tested?

Yes — new tests in parquet/file/dict_cost_fallback_test.go covering: (1) default behavior still discards a non-paying dictionary, (2) with the option disabled an explicitly requested dictionary is kept, (3) a per-column disable keeps that column's dictionary while (4) a disable scoped to a different column leaves the fallback active, and (5) the DictionaryPageSizeLimit fallback still applies when the cost fallback is disabled. Existing parquet/file and parquet/internal/encoding suites pass.

Are there any user-facing changes?

New opt-in writer properties; default behavior is unchanged.

@tom-eon
tom-eon requested a review from zeroshade as a code owner July 27, 2026 22:42
@tom-eon
tom-eon force-pushed the parquet-dict-cost-fallback-option branch from 76fdf25 to f2a7481 Compare July 28, 2026 14:38
The pre-first-page cost-based dictionary fallback compares uncompressed
sizes, so it can discard a dictionary that would have won after page
compression. Allow callers that decide dictionary usage themselves to
opt out; the DictionaryPageSizeLimit fallback is unaffected.
@tom-eon
tom-eon force-pushed the parquet-dict-cost-fallback-option branch from f2a7481 to 6135548 Compare July 28, 2026 15:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant