feat: allow backing an ArrayBuffer/SharedArrayBuffer with embedder-owned memory - #5466
Conversation
…ned memory Adds a public API to create `ArrayBuffer`s and `SharedArrayBuffer`s whose bytes alias an embedder-supplied memory region instead of a Boa-owned allocation, enabling zero-copy sharing of byte regions (Wasm linear memories, mmap'd files, GPU-mapped buffers) between JavaScript and native host code: - `JsArrayBuffer::from_external_ptr` / `ArrayBuffer::from_external_data` - `JsSharedArrayBuffer::from_external_ptr` / `SharedArrayBuffer::from_external_ptr` - `is_external()` on all four types Internally, `ArrayBuffer`'s data slot becomes a `BufferData` enum with `Owned(AlignedVec<u8>)`/`External` variants, and `SharedArrayBuffer`'s inner buffer becomes a `SharedData` enum with the same shape. All existing accessors work on both variants, so typed arrays, `DataView` and `Atomics` work transparently over external memory. Externally-backed buffers are always fixed-length and cannot be detached, resized, grown nor transferred; those operations throw a `TypeError`. Boa never allocates, grows nor frees the external region. Closes boa-dev#5447 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Test262 conformance changes
Tested main commit: |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #5466 +/- ##
===========================================
+ Coverage 47.24% 62.86% +15.61%
===========================================
Files 476 530 +54
Lines 46892 59167 +12275
===========================================
+ Hits 22154 37195 +15041
+ Misses 24738 21972 -2766 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Shine-neko
left a comment
There was a problem hiding this comment.
Thanks for tackling this so quickly! I tested this branch end-to-end embedding Boa, backing WebAssembly.Memory.prototype.buffer with a wasm linear memory via from_external_ptr, and it works: a napi-rs wasm addon now sees JS-side writes and runs correctly. Two things I hit while reviewing the unsafe surface:
| } | ||
|
|
||
| /// The internal representation of an `ArrayBuffer` object. | ||
| #[derive(Debug, Clone, Trace, Finalize, JsData)] |
There was a problem hiding this comment.
ArrayBuffer derives Clone, which is now unsound for the External variant: cloning copies the raw pointer, so two ArrayBuffers alias the same region and can each hand out &mut [u8] to it simultaneously. Owned buffers are fine (the Vec is deep-copied), but an external clone breaks the aliasing invariant from_external_data documents.
Would a manual Clone that deep-copies the External variant into an Owned one (or rejects cloning external buffers) work here?
| /// Panics if `ptr` is null. | ||
| #[must_use] | ||
| pub unsafe fn from_external_ptr(ptr: *mut u8, len: usize) -> Self { | ||
| let ptr = NonNull::new(ptr.cast::<AtomicU8>()).expect("`ptr` must be non-null"); |
There was a problem hiding this comment.
For a SharedArrayBuffer, Atomics and wider views (Int32Array, Float64Array, BigInt64Array) perform aligned atomic accesses. A byte-aligned ptr isn't enough — the base address must satisfy the largest alignment those ops need. Worth asserting/documenting it here, e.g.:
| let ptr = NonNull::new(ptr.cast::<AtomicU8>()).expect("`ptr` must be non-null"); | |
| let ptr = NonNull::new(ptr.cast::<AtomicU8>()).expect("`ptr` must be non-null"); | |
| debug_assert!(ptr.as_ptr() as usize % 8 == 0, "external SharedArrayBuffer backing must be 8-byte aligned"); |
| fn as_slice(&self) -> &[u8] { | ||
| // SAFETY: The creator of an `ExternalMemory` guarantees that `ptr` is valid | ||
| // for reads and writes of `len` bytes for the whole lifetime of the buffer. | ||
| unsafe { slice::from_raw_parts(self.ptr.as_ptr(), self.len) } |
There was a problem hiding this comment.
One more precondition for slice::from_raw_parts: the region size must be <= isize::MAX bytes. It's a documented safety requirement of from_raw_parts, so it'd be worth either asserting it at construction (from_external_data/from_external_ptr) or adding it to the # Safety list the caller must uphold.
| // SAFETY: The external region is only ever accessed through atomic operations, and the | ||
| // creator of an externally-backed `SharedArrayBuffer` guarantees that the region stays | ||
| // valid for the whole lifetime of the buffer, making it safe to share between threads. | ||
| unsafe impl Send for SharedData {} |
There was a problem hiding this comment.
The SAFETY comment covers lifetime/validity, but one case worth spelling out: if the external region can be relocated (e.g. a growable WebAssembly memory that moves its base on memory.grow), the stored ptr silently dangles. The invariant the caller must uphold is stronger than "stays valid" — it's "stays valid and unmoved". Might be worth adding that word here and in the # Safety docs, since the wasm-memory use case is exactly where it bites.
This PR implements the feature requested in #5447: a public API to back an
ArrayBuffer/SharedArrayBufferwith embedder-owned memory, enabling zero-copy sharing of byte regions (WebAssembly linear memories, mmap'd files, GPU-mapped buffers) between JavaScript and native host code.What's added
Public API:
JsArrayBuffer::from_external_ptr(ptr, len, context)(unsafe) — creates anArrayBufferwhose bytes aliaslenbytes of embedder-owned memory atptr. No copy, no allocation: JS writes are immediately visible to the embedder throughptrand vice versa.JsSharedArrayBuffer::from_external_ptr(ptr, len, context)(unsafe) — same forSharedArrayBuffer.ArrayBuffer::from_external_data(ptr, len)/SharedArrayBuffer::from_external_ptr(ptr, len)(unsafe) — lower-level constructors on the buffer data types themselves.is_external()on all four types.Semantics of externally-backed buffers (the conservative answers to the design questions raised in the issue):
resize/growthrow aTypeError.detach,ArrayBuffer.prototype.transfer) — these throw aTypeErrorand leave the buffer intact. This matches the Wasm-memory use case (WebAssembly.Memory.prototype.bufferis non-detachable by user code) and keeps ownership unambiguous: Boa never allocates, grows nor frees the region.DataView,Atomics,slice, etc.) works transparently on top of the aliased memory.Implementation
ArrayBuffer's internal slot changes fromOption<AlignedVec<u8>>toOption<BufferData>withOwned(AlignedVec<u8>)/External(ExternalMemory)variants. All existing accessors (bytes,bytes_mut,bytes_with_len*,len) work on both variants, so typed arrays,DataViewandAtomicsneed no changes.SharedArrayBuffer'sInner.bufferchanges fromAlignedBox<[AtomicU8]>to aSharedDataenum withOwned/Externalvariants (withunsafe impl Send/Syncjustified by the atomic-only access contract).unsafepointer constructor (option 1 of the issue), with safety contracts spelled out on every constructor. AnArc-basedBackingStorehandle could be layered on top later without changing the internals.Tests
external_array_buffer_zero_copy— JS writes visible through the embedder's pointer and vice versa.external_array_buffer_cannot_detach_nor_resize— detach/resize throw and leave the buffer usable.external_shared_array_buffer_zero_copy— same aliasing checks forSharedArrayBuffer, plusgrowable === false.JsArrayBuffer::from_external_ptrdemonstrating the zero-copy roundtrip.Closes #5447
🤖 Generated with Claude Code