feat: emit StorageEnvelope compressed_data as msgpack bin (protocol 1.1) (LAB-866) - #59
Merged
Merged
Conversation
….1) (LAB-866) Writer flip, rollout step 3 of the readers-first plan (LAB-764): serde_bytes on compressed_data only — checksum stays array-of-ints per the protocol's normative scope exclusion. Re-encode byte-identity now asserts against the *_bin vector set; decode-identity keeps running against BOTH sets forever. dual_decode's local twin struct becomes the legacy-reader stand-in, keeping the permanent proof that pre-1.1 readers accept bin wire. 64 MiB incompressible (hot_path bench, new 64mib_incompressible group): envelope overhead 1.508x -> 1.0039x, encode ~7.3x faster, decode ~11.2x faster, store() e2e 158 -> 567 MiB/s. GH #54 close deferred to stage 5 (SDK parity); see protocol PR #44 and cachekit-core#57 (stage 2). Expert panel (high stakes) passed: AAD independence, compressed_data- only scope, format/checksum untouched, bin length headers bounds-checked pre-allocation all CONFIRMED. Fixture-level bin16/bin32 width-boundary vector (deferred MUST) tracked as LAB-868.
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (5)
WalkthroughThe PR documents the msgpack bin writer format, updates compatibility tests to validate legacy and current readers, and adds a deterministic 64 MiB envelope codec benchmark. ChangesWire-format compatibility
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related issues
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Kody Review CompleteGreat news! 🎉 Keep up the excellent work! 🚀 Kody Guide: Usage and ConfigurationInteracting with Kody
Current Kody ConfigurationReview OptionsThe following review options are enabled or disabled:
|
![kodus-27b[bot]](https://avatars.githubusercontent.com/u/223823690?s=60&v=4){kind=small}
Contributor
Author
|
@coderabbitai review |
Contributor
✅ Action performedReview finished.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
27Bslash6
added a commit
to cachekit-io/cachekit-ts
that referenced
this pull request
Jul 29, 2026
…-901) (#91) Closes LAB-901. Parent: LAB-764 stage 5 (protocol 1.1 rollout, TypeScript leg). ## What Bump `cachekit-core` 0.3.0 → 0.4.0 in both native bindings — `cachekit-core-ts` (NAPI) and `cachekit-core-wasm` — and prove the protocol 1.1 wire change through the real TS stack. Core 0.4.0 emits `StorageEnvelope.compressed_data` as msgpack **bin** ([core#59](cachekit-io/cachekit-core#59), spec [protocol#44](cachekit-io/protocol#44)); pre-0.4.0 array-of-integers envelopes remain readable forever (dual-read). No ts-side serialization code changes — the codec is single-sourced in cachekit-core. The diff is the two Cargo pins + locks, the re-vendored spec vectors, and tests. ## Byte-verification (proven, not asserted) - **Re-vendored** `protocol/test-vectors/wire-format.json` v1.1.0 (6 legacy + 6 `*_bin` vectors) and now drive the **full vector set through both lanes**: NAPI (Node lane, `test/protocol`) and wasm inside real workerd (`test/workers`). - **bin emit**: `pack` output equals every `*_bin` ground-truth envelope byte-for-byte, in both bindings — which also pins the two bindings byte-identical to each other and to every other SDK. - **Legacy read**: all 6 legacy vectors plus the pre-0.4.0 canonical fixture decode through the real retrieve path (permanent dual-read proof). - **Marker assert**: fresh packs of arbitrary payloads (bin8- and bin16-sized, incompressible included) carry the msgpack bin marker and round-trip identically. - Local verification against ground truth before the fixtures were written: freshly built NAPI and wasm artifacts both reproduce `simple_string_bin` exactly and decode the legacy `simple_string` envelope. ## Checks - Node lane: 659 passed | 1 skipped. Workers lane (workerd): 97 passed. Lint, prettier, type-check green. - wasm size budget: 54,992 B gzipped (< 100 KB), API drift check clean. - `.secrets.baseline` refreshed — hex test vectors are line-pinned there (pre-existing pattern), and the vector additions moved them. ## Docs gate No README or docs.cachekit.io statement describes the `compressed_data` field encoding (checked SDK READMEs + docs repo `envelope`/`compressed_data` hits); container-level claims ("LZ4 + xxHash3-64", byte-compatibility across NAPI/wasm) remain true. Test-file headers updated to protocol v1.1 with the dual-read note. Nothing else needed changing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Writer flip — rollout step 3 of the protocol 1.1 readers-first plan (LAB-764, decision record
decisions/envelope-bin-encoding.md, spec PR cachekit-io/protocol#44). Resolves LAB-866. References #54 (GitHub close deferred until stage 5 confirms SDK parity).The diff
src/byte_storage.rs:#[serde(with = "serde_bytes")]onStorageEnvelope.compressed_data— the one functional change.checksum: [u8;8],original_size,formatuntouched (normative scope exclusions).serde_byteswas already a non-optional dependency; zero dep changes.tests/wire_format_vectors.rs: re-encode byte-identity (store-level + codec-level) switched from the legacy set to the*_binset, exactly at the switch points stage 2 (test: prove dual-decode of legacy + bin envelopes, re-pin protocol 1.1 fixture (LAB-791) #57) marked. Decode-identity keeps running against BOTH sets, forever.tests/dual_decode.rs: reader roles inverted — the crate struct is now the serde_bytes shape, so the local twin becameStorageEnvelopeLegacy(plainVec<u8>), permanently proving legacy readers acceptbinand legacy wire still decodes. Width-boundary tests (bin8/bin16/bin32) now exercise the real writer. A re-encode assert that became an exact duplicate ofenvelope_codec_roundtrip_byte_identicalwas deleted (panel cut).benches/hot_path.rs: new64mib_incompressiblegroup (envelope codec + store/retrieve e2e) — the LAB-510-harness proof workload, with a self-enforcing incompressibility assert.README.md: wire-format paragraph now states the writer emitsbin.Benchmark proof (64 MiB incompressible, this machine, criterion
--baseline legacy)Reproduces the LAB-764 decision-memo reference measurement (1.508→1.0039×, encode 6–7×, decode 9–11×, store() off the ~150 MB/s msgpack-encode ceiling).
The v0.4.0 RELEASE PR (release-please's) must not merge before the LAB-865 release.yml fix is on main. Status: #58 is already merged ✓ — so the remaining order is simply: merge this PR → merge the release PR. This PR does not touch
.github/.feat:type minor-bumps 0.3.0 → 0.4.0.Expert panel (crypto/protocol gate, high stakes) — PASSED
compressed_dataonly; byte-identity against protocol-pinned*_binvectors proves checksum/original_size/format bytes unchanged.formatuntouched: CONFIRMED.retrieve()/validate()cap input at 512 MiB pre-deserialize; rmp-serde 1.3.1read_slicebounds-checks declared bin lengths against the buffer before any allocation.incompressible()dedup via#[path]include (couples bench to test internals to save 12 deterministic lines).Docs gate
README wire-format section updated (writer emission + scope exclusion). Protocol spec already correct (protocol 1.1). Rust doc-comments in the two proof test files rewritten to post-flip reality. SDK-side docs are stage 5 (per rollout plan). No other doc surface claims the legacy encoding — verified by repo-wide grep for pre-flip tense.
Summary by CodeRabbit
Compatibility
Documentation
Testing
Performance