Skip to content

fix: report plaintext size/etag for multipart objects in listings#131

Merged
ServerSideHannes merged 2 commits into
mainfrom
fix/list-multipart-plaintext-size
Jul 12, 2026
Merged

fix: report plaintext size/etag for multipart objects in listings#131
ServerSideHannes merged 2 commits into
mainfrom
fix/list-multipart-plaintext-size

Conversation

@ServerSideHannes

Copy link
Copy Markdown
Owner

Problem

ListObjects resolves an object's plaintext size from the backend object's plaintext-size user metadata — but that metadata is only stamped on the simple PUT path. Multipart objects can't carry it (user metadata is fixed at CreateMultipartUpload, before the final size is known); their plaintext size lives in the .s3proxy-internal/<key>.meta sidecar, which _process_list_objects never read. Listings therefore reported the ciphertext/framed size for every multipart object.

Impact, observed on the ScyllaDB backup cluster: scylla-manager's rclone compares listed sizes to decide what to skip. Every multipart object (all large sstables) always looked changed, so every retry pass and every subsequent backup run re-uploaded the entire multipart dataset. Tables show ~180–199% progress with 0 failed bytes; the daily backup ships ~2× its data, runs 30h+, and dedup only works for small (simple-PUT) files. HEAD already resolves the sidecar correctly — only LIST was broken.

Fix

  • _process_list_objects: when head metadata lacks plaintext-size, load the multipart sidecar and report total_plaintext_size plus the same synthetic etag that HEAD and CompleteMultipartUpload return. Raw-size fallback remains for objects with neither (legacy/unencrypted, where the backend size is already correct).
  • New process-local LRU (state/attr_cache.py) keyed by (bucket, key, backend etag) caches resolved attributes, so repeated listings (a backup client walking the same directories each pass) skip the per-object round-trips entirely — including the pre-existing per-object HEAD. Overwrites change the backend etag, which invalidates the entry without coordination.
  • CompleteMultipartUpload primes the cache from its response etag, so freshly uploaded objects list with no extra reads.

Tests

tests/unit/test_list_multipart_plaintext_size.py:

  • sidecar resolution for multipart objects (size + synthetic etag)
  • raw-size fallback when no sidecar exists
  • simple-PUT user metadata still wins without a sidecar lookup
  • repeat listings served from cache (no extra HEAD/GET)
  • cache scoped to backend etag (overwrite re-resolves)
  • failed HEAD falls back and does not poison the cache
  • LRU eviction + empty-etag guard
  • end-to-end complete → cache primed (reuses the complete-recovery fixtures)

Expected effect on the Scylla backups: retry passes and subsequent runs dedup instead of re-uploading — roughly halving transfer and bringing the daily run back inside its window.

ListObjects resolved plaintext size from the backend object's user
metadata, which is only stamped on the simple PUT path. Multipart
objects carry their plaintext size in the .meta sidecar (user metadata
is fixed at CreateMultipartUpload), so listings reported the ciphertext
size. Sync clients comparing sizes (scylla-manager's rclone) saw every
multipart object as changed and re-uploaded it on every pass — doubling
backup transfer and defeating snapshot dedup entirely.

_process_list_objects now falls back to the sidecar when the head
metadata lacks plaintext-size, returning the same synthetic etag that
HEAD and CompleteMultipartUpload report. Resolved attributes are kept
in a process-local LRU keyed by (bucket, key, backend etag) — an
overwrite changes the backend etag, which invalidates the entry — and
CompleteMultipartUpload primes the cache so fresh uploads list without
extra round-trips.
@ServerSideHannes ServerSideHannes merged commit 2e1a856 into main Jul 12, 2026
11 checks passed
@ServerSideHannes ServerSideHannes deleted the fix/list-multipart-plaintext-size branch July 12, 2026 16:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant