fix: report plaintext size/etag for multipart objects in listings#131
Merged
Conversation
ListObjects resolved plaintext size from the backend object's user metadata, which is only stamped on the simple PUT path. Multipart objects carry their plaintext size in the .meta sidecar (user metadata is fixed at CreateMultipartUpload), so listings reported the ciphertext size. Sync clients comparing sizes (scylla-manager's rclone) saw every multipart object as changed and re-uploaded it on every pass — doubling backup transfer and defeating snapshot dedup entirely. _process_list_objects now falls back to the sidecar when the head metadata lacks plaintext-size, returning the same synthetic etag that HEAD and CompleteMultipartUpload report. Resolved attributes are kept in a process-local LRU keyed by (bucket, key, backend etag) — an overwrite changes the backend etag, which invalidates the entry — and CompleteMultipartUpload primes the cache so fresh uploads list without extra round-trips.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ListObjectsresolves an object's plaintext size from the backend object'splaintext-sizeuser metadata — but that metadata is only stamped on the simple PUT path. Multipart objects can't carry it (user metadata is fixed atCreateMultipartUpload, before the final size is known); their plaintext size lives in the.s3proxy-internal/<key>.metasidecar, which_process_list_objectsnever read. Listings therefore reported the ciphertext/framed size for every multipart object.Impact, observed on the ScyllaDB backup cluster: scylla-manager's rclone compares listed sizes to decide what to skip. Every multipart object (all large sstables) always looked changed, so every retry pass and every subsequent backup run re-uploaded the entire multipart dataset. Tables show ~180–199% progress with 0 failed bytes; the daily backup ships ~2× its data, runs 30h+, and dedup only works for small (simple-PUT) files. HEAD already resolves the sidecar correctly — only LIST was broken.
Fix
_process_list_objects: when head metadata lacksplaintext-size, load the multipart sidecar and reporttotal_plaintext_sizeplus the same synthetic etag that HEAD and CompleteMultipartUpload return. Raw-size fallback remains for objects with neither (legacy/unencrypted, where the backend size is already correct).state/attr_cache.py) keyed by(bucket, key, backend etag)caches resolved attributes, so repeated listings (a backup client walking the same directories each pass) skip the per-object round-trips entirely — including the pre-existing per-object HEAD. Overwrites change the backend etag, which invalidates the entry without coordination.CompleteMultipartUploadprimes the cache from its response etag, so freshly uploaded objects list with no extra reads.Tests
tests/unit/test_list_multipart_plaintext_size.py:Expected effect on the Scylla backups: retry passes and subsequent runs dedup instead of re-uploading — roughly halving transfer and bringing the daily run back inside its window.