feat(storage): add runner script and docs for appendable upload benchmark - #6171
feat(storage): add runner script and docs for appendable upload benchmark#6171vsharonlynn wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a new benchmark crate, storage-benchmark-appendable-object, to measure Google Cloud Storage appendable upload performance under various chunk sizes. The feedback suggests creating the output directory recursively before writing the CSV report to prevent potential runtime failures, and adding a lifecycle rule to the bucket setup instructions in the README to automatically delete uploaded objects and avoid unnecessary storage costs.
| gcloud storage buckets create gs://${BUCKET_NAME} \ | ||
| --location=${ZONE} \ | ||
| --placement=zone \ | ||
| --default-storage-class=RAPID \ | ||
| --hierarchical-namespace \ | ||
| --uniform-bucket-level-access | ||
| ``` |
There was a problem hiding this comment.
Since each run of the benchmark uploads multiple large objects (e.g., 100 x 100 MiB = 10 GiB), these objects will accumulate in the zonal bucket and incur storage costs. It is highly recommended to configure a lifecycle rule on the bucket to automatically delete objects after 1 day.
| gcloud storage buckets create gs://${BUCKET_NAME} \ | |
| --location=${ZONE} \ | |
| --placement=zone \ | |
| --default-storage-class=RAPID \ | |
| --hierarchical-namespace \ | |
| --uniform-bucket-level-access | |
| ``` | |
| gcloud storage buckets create gs://${BUCKET_NAME} \ | |
| --location=${ZONE} \ | |
| --placement=zone \ | |
| --default-storage-class=RAPID \ | |
| --hierarchical-namespace \ | |
| --uniform-bucket-level-access | |
| # Add a lifecycle rule to delete objects after 1 day to avoid storage costs | |
| echo '{"rule": [{"action": {"type": "Delete"}, "condition": {"age": 1}}]}' > lifecycle.json | |
| gcloud storage buckets update gs://${BUCKET_NAME} --lifecycle-file=lifecycle.json | |
| rm lifecycle.json |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #6171 +/- ##
=======================================
Coverage 96.58% 96.59%
=======================================
Files 266 266
Lines 66852 66852
=======================================
+ Hits 64571 64574 +3
+ Misses 2281 2278 -3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
b64cbd3 to
1371208
Compare
1371208 to
fa01ebe
Compare
This PR comes after PR #6170.
Issue #5716.