feat: support zstd metadata compression by wirybeaver · Pull Request #2894 · apache/iceberg-rust

wirybeaver · 2026-07-25T23:06:29Z

Which issue does this PR close?

Part of #2411. This implements the "Support zstd metadata compression codec" tracking item without closing the broader v3 epic.

What changes are included in this PR?

Accept write.metadata.compression-codec=zstd case-insensitively, using the existing default zstd level.
Always write canonical .zstd.metadata.json names (.gz.metadata.json for gzip). Trailing .metadata.json.zstd and .metadata.json.gz forms are accepted only when parsing existing locations for interoperability, and subsequent versions are canonicalized.
Detect gzip and standard zstd frames from content when reading metadata, independent of filenames.
Compress metadata with zstd when writing while preserving codec/location consistency checks.
Add an iceberg-go zstd metadata fixture and interoperability, error-path, canonicalization, transition, and round-trip coverage.

No catalog-specific, dependency, lockfile, or format-version changes are included.

Are these changes tested?

Yes:

cargo test -p iceberg --lib — 1473 passed
cargo fmt --all -- --check
cargo clippy -p iceberg --all-targets --all-features -- -D warnings

Add zstd metadata codec validation, canonical filename generation, alternate suffix parsing, and content-based frame detection for reads. Cover interoperability, codec transitions, truncated frames, and semantic round trips. Part of apache#2411.

xanderbailey

Nice PR, thanks for working on this! Left a comment on style, let me know what you think

xanderbailey · 2026-07-27T21:31:23Z

+        let zstd_suffix = CompressionCodec::zstd_default().suffix()?;
+        let metadata_suffix = ".metadata.json";
+
+        let (stripped, compression_codec) = if let Some(stripped) =
+            file_name.strip_suffix(&format!("{metadata_suffix}{zstd_suffix}"))
+        {
+            (stripped, CompressionCodec::zstd_default())
+        } else if let Some(stripped) =
+            file_name.strip_suffix(&format!("{metadata_suffix}{gzip_suffix}"))
+        {
+            (stripped, CompressionCodec::gzip_default())
+        } else if let Some(stripped) = file_name.strip_suffix(metadata_suffix) {
+            if let Some(stripped) = stripped.strip_suffix(zstd_suffix) {
+                (stripped, CompressionCodec::zstd_default())
+            } else if let Some(stripped) = stripped.strip_suffix(gzip_suffix) {
+                (stripped, CompressionCodec::gzip_default())
+            } else {
+                (stripped, CompressionCodec::None)
+            }
        } else {
-            (stripped, CompressionCodec::None)
+            return Err(Error::new(
+                ErrorKind::Unexpected,
+                format!("Invalid metadata file ending: {file_name}"),
+            ));


Suggested change

let zstd_suffix = CompressionCodec::zstd_default().suffix()?;

let metadata_suffix = ".metadata.json";

let (stripped, compression_codec) = if let Some(stripped) =

file_name.strip_suffix(&format!("{metadata_suffix}{zstd_suffix}"))

{

(stripped, CompressionCodec::zstd_default())

} else if let Some(stripped) =

file_name.strip_suffix(&format!("{metadata_suffix}{gzip_suffix}"))

{

(stripped, CompressionCodec::gzip_default())

} else if let Some(stripped) = file_name.strip_suffix(metadata_suffix) {

if let Some(stripped) = stripped.strip_suffix(zstd_suffix) {

(stripped, CompressionCodec::zstd_default())

} else if let Some(stripped) = stripped.strip_suffix(gzip_suffix) {

(stripped, CompressionCodec::gzip_default())

} else {

(stripped, CompressionCodec::None)

}

} else {

(stripped, CompressionCodec::None)

return Err(Error::new(

ErrorKind::Unexpected,

format!("Invalid metadata file ending: {file_name}"),

));

let codecs = [

CompressionCodec::zstd_default(),

CompressionCodec::gzip_default(),

];

let (stripped, compression_codec) = codecs

.into_iter()

.find_map(|codec| {

strip_metadata_suffix(file_name, codec.suffix().ok()?).map(|s| (s, codec))

})

.or_else(|| {

file_name

.strip_suffix(METADATA_SUFFIX)

.map(|s| (s, CompressionCodec::None))

})

.ok_or_else(|| {

Error::new(

ErrorKind::Unexpected,

format!("Invalid metadata file ending: {file_name}"),

)

})?;

fn strip_metadata_suffix<'a>(file_name: &'a str, codec_suffix: &str) -> Option<&'a str> { file_name .strip_suffix(METADATA_SUFFIX) .and_then(|s| s.strip_suffix(codec_suffix)) .or_else(|| { file_name .strip_suffix(codec_suffix) .and_then(|s| s.strip_suffix(METADATA_SUFFIX)) }) }

I wonder if this kind of thing reads a little better, this is very nested and hard to follow as it currently stands

Done in fdc645c. I added the suggested strip_metadata_suffix helper and iterator-based parsing, using the shared suffix-to-codec mapping so the canonical and trailing forms stay easy to extend.

xanderbailey · 2026-07-27T21:38:28Z

On closer look at the Java docs https://iceberg.apache.org/docs/latest/configuration/#write-properties, I don't actually see zstd here... am I missing something?

xanderbailey · 2026-07-27T21:49:06Z

Go has it... apache/iceberg-go#1020

They claim in that pr that py-iceberg and java have it but I can't find it...

wirybeaver · 2026-07-28T02:15:15Z

Go has it... apache/iceberg-go#1020

They claim in that pr that py-iceberg and java have it but I can't find it...

Yeah, I refer to the iceberg-go. And #2411 also mentioned this zstd support for metadata

blackmwk · 2026-07-28T09:37:39Z

+        let zstd_suffix = CompressionCodec::zstd_default().suffix()?;
+        let metadata_suffix = ".metadata.json";
+
+        let (stripped, compression_codec) = if let Some(stripped) =
+            file_name.strip_suffix(&format!("{metadata_suffix}{zstd_suffix}"))
+        {
+            (stripped, CompressionCodec::zstd_default())
+        } else if let Some(stripped) =
+            file_name.strip_suffix(&format!("{metadata_suffix}{gzip_suffix}"))
+        {
+            (stripped, CompressionCodec::gzip_default())
+        } else if let Some(stripped) = file_name.strip_suffix(metadata_suffix) {
+            if let Some(stripped) = stripped.strip_suffix(zstd_suffix) {
+                (stripped, CompressionCodec::zstd_default())
+            } else if let Some(stripped) = stripped.strip_suffix(gzip_suffix) {
+                (stripped, CompressionCodec::gzip_default())
+            } else {
+                (stripped, CompressionCodec::None)
+            }
        } else {
-            (stripped, CompressionCodec::None)
+            return Err(Error::new(
+                ErrorKind::Unexpected,
+                format!("Invalid metadata file ending: {file_name}"),
+            ));


blackmwk · 2026-07-28T09:40:19Z

                CompressionCodec::None.name(),
-                CompressionCodec::gzip_default().name()
+                CompressionCodec::gzip_default().name(),
+                CompressionCodec::zstd_default().name()


I'm not a big fan of doing it this way, it's hard to extend and maitain. Ideally, we should have a static map in compression file, like TABLE_METADATA_SUFFIX_TO_COMPRESSION, TABLE_METADATA_MAGIC_TO_COMPRESSION, TABLE_METADATA_SUPPORTED_COMPreSSION, and reuse them in all places.

Done in fdc645c. I centralized the supported metadata codecs, suffix mappings, and magic mappings in compression.rs, then reused them for property validation, location parsing, and content-based detection. The write path now relies on the validated codec directly as well.

Reuse shared supported-codec, suffix, and magic mappings across metadata property validation, location parsing, and I/O. Simplify alternate suffix parsing with a dedicated helper.

feat: support zstd metadata compression

a75b1f9

Add zstd metadata codec validation, canonical filename generation, alternate suffix parsing, and content-based frame detection for reads. Cover interoperability, codec transitions, truncated frames, and semantic round trips. Part of apache#2411.

wirybeaver mentioned this pull request Jul 25, 2026

EPIC: v3 Support Tracking #2411

Open

xanderbailey reviewed Jul 27, 2026

View reviewed changes

blackmwk reviewed Jul 28, 2026

View reviewed changes

refactor: centralize metadata compression mappings

fdc645c

Reuse shared supported-codec, suffix, and magic mappings across metadata property validation, location parsing, and I/O. Simplify alternate suffix parsing with a dedicated helper.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: support zstd metadata compression - #2894

feat: support zstd metadata compression#2894
wirybeaver wants to merge 2 commits into
apache:mainfrom
wirybeaver:iceberg-rs-zstd

wirybeaver commented Jul 25, 2026 •

edited

Loading

Uh oh!

xanderbailey left a comment

Uh oh!

xanderbailey Jul 27, 2026

Uh oh!

blackmwk Jul 28, 2026

Uh oh!

wirybeaver Jul 28, 2026

Uh oh!

xanderbailey commented Jul 27, 2026 •

edited

Loading

Uh oh!

xanderbailey commented Jul 27, 2026 •

edited

Loading

Uh oh!

wirybeaver commented Jul 28, 2026 •

edited

Loading

Uh oh!

blackmwk Jul 28, 2026

Uh oh!

blackmwk Jul 28, 2026

Uh oh!

wirybeaver Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

wirybeaver commented Jul 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

What changes are included in this PR?

Are these changes tested?

Uh oh!

xanderbailey left a comment

Choose a reason for hiding this comment

Uh oh!

xanderbailey Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

blackmwk Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

wirybeaver Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

xanderbailey commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xanderbailey commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wirybeaver commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blackmwk Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

blackmwk Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

wirybeaver Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wirybeaver commented Jul 25, 2026 •

edited

Loading

xanderbailey commented Jul 27, 2026 •

edited

Loading

xanderbailey commented Jul 27, 2026 •

edited

Loading

wirybeaver commented Jul 28, 2026 •

edited

Loading