Skip to content

feat: allow backing an ArrayBuffer/SharedArrayBuffer with embedder-owned memory - #5466

Open
mansiverma897993 wants to merge 1 commit into
boa-dev:mainfrom
mansiverma897993:feat/external-backed-array-buffers
Open

feat: allow backing an ArrayBuffer/SharedArrayBuffer with embedder-owned memory#5466
mansiverma897993 wants to merge 1 commit into
boa-dev:mainfrom
mansiverma897993:feat/external-backed-array-buffers

Conversation

@mansiverma897993

Copy link
Copy Markdown
Contributor

This PR implements the feature requested in #5447: a public API to back an ArrayBuffer/SharedArrayBuffer with embedder-owned memory, enabling zero-copy sharing of byte regions (WebAssembly linear memories, mmap'd files, GPU-mapped buffers) between JavaScript and native host code.

What's added

Public API:

  • JsArrayBuffer::from_external_ptr(ptr, len, context) (unsafe) — creates an ArrayBuffer whose bytes alias len bytes of embedder-owned memory at ptr. No copy, no allocation: JS writes are immediately visible to the embedder through ptr and vice versa.
  • JsSharedArrayBuffer::from_external_ptr(ptr, len, context) (unsafe) — same for SharedArrayBuffer.
  • ArrayBuffer::from_external_data(ptr, len) / SharedArrayBuffer::from_external_ptr(ptr, len) (unsafe) — lower-level constructors on the buffer data types themselves.
  • is_external() on all four types.

Semantics of externally-backed buffers (the conservative answers to the design questions raised in the issue):

  • Always fixed-length; resize/grow throw a TypeError.
  • Cannot be detached nor transferred (detach, ArrayBuffer.prototype.transfer) — these throw a TypeError and leave the buffer intact. This matches the Wasm-memory use case (WebAssembly.Memory.prototype.buffer is non-detachable by user code) and keeps ownership unambiguous: Boa never allocates, grows nor frees the region.
  • Everything else (typed arrays, DataView, Atomics, slice, etc.) works transparently on top of the aliased memory.

Implementation

  • ArrayBuffer's internal slot changes from Option<AlignedVec<u8>> to Option<BufferData> with Owned(AlignedVec<u8>) / External(ExternalMemory) variants. All existing accessors (bytes, bytes_mut, bytes_with_len*, len) work on both variants, so typed arrays, DataView and Atomics need no changes.
  • SharedArrayBuffer's Inner.buffer changes from AlignedBox<[AtomicU8]> to a SharedData enum with Owned/External variants (with unsafe impl Send/Sync justified by the atomic-only access contract).
  • The lifetime/ownership question is answered with a documented unsafe pointer constructor (option 1 of the issue), with safety contracts spelled out on every constructor. An Arc-based BackingStore handle could be layered on top later without changing the internals.

Tests

  • external_array_buffer_zero_copy — JS writes visible through the embedder's pointer and vice versa.
  • external_array_buffer_cannot_detach_nor_resize — detach/resize throw and leave the buffer usable.
  • external_shared_array_buffer_zero_copy — same aliasing checks for SharedArrayBuffer, plus growable === false.
  • A doctest on JsArrayBuffer::from_external_ptr demonstrating the zero-copy roundtrip.

Closes #5447

🤖 Generated with Claude Code

…ned memory

Adds a public API to create `ArrayBuffer`s and `SharedArrayBuffer`s whose
bytes alias an embedder-supplied memory region instead of a Boa-owned
allocation, enabling zero-copy sharing of byte regions (Wasm linear
memories, mmap'd files, GPU-mapped buffers) between JavaScript and native
host code:

- `JsArrayBuffer::from_external_ptr` / `ArrayBuffer::from_external_data`
- `JsSharedArrayBuffer::from_external_ptr` /
  `SharedArrayBuffer::from_external_ptr`
- `is_external()` on all four types

Internally, `ArrayBuffer`'s data slot becomes a `BufferData` enum with
`Owned(AlignedVec<u8>)`/`External` variants, and `SharedArrayBuffer`'s
inner buffer becomes a `SharedData` enum with the same shape. All existing
accessors work on both variants, so typed arrays, `DataView` and `Atomics`
work transparently over external memory.

Externally-backed buffers are always fixed-length and cannot be detached,
resized, grown nor transferred; those operations throw a `TypeError`. Boa
never allocates, grows nor frees the external region.

Closes boa-dev#5447

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mansiverma897993
mansiverma897993 requested a review from a team as a code owner July 30, 2026 11:40
@github-actions github-actions Bot added C-Tests Issues and PRs related to the tests. C-Builtins PRs and Issues related to builtins/intrinsics Waiting On Review Waiting on reviews from the maintainers labels Jul 30, 2026
@github-actions github-actions Bot added this to the v1.0.0 milestone Jul 30, 2026
@github-actions

Copy link
Copy Markdown

Test262 conformance changes

Test result main count PR count difference
Total 53,125 53,125 0
Passed 51,073 51,073 0
Ignored 1,482 1,482 0
Failed 570 570 0
Panics 0 0 0
Conformance 96.14% 96.14% 0.00%

Tested main commit: 4fc75c6ae9d85f2b8065c6716f88e9b35318438c
Tested PR commit: 762e3c48c7c2ad3f76c9497cee1cc9fa69f8db1b
Compare commits: 4fc75c6...762e3c4

@codecov

codecov Bot commented Jul 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.89888% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.86%. Comparing base (6ddc2b4) to head (762e3c4).
⚠️ Report is 1012 commits behind head on main.

Files with missing lines Patch % Lines
core/engine/src/builtins/array_buffer/mod.rs 75.86% 14 Missing ⚠️
core/engine/src/builtins/array_buffer/shared.rs 84.21% 3 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##             main    #5466       +/-   ##
===========================================
+ Coverage   47.24%   62.86%   +15.61%     
===========================================
  Files         476      530       +54     
  Lines       46892    59167    +12275     
===========================================
+ Hits        22154    37195    +15041     
+ Misses      24738    21972     -2766     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Shine-neko Shine-neko left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tackling this so quickly! I tested this branch end-to-end embedding Boa, backing WebAssembly.Memory.prototype.buffer with a wasm linear memory via from_external_ptr, and it works: a napi-rs wasm addon now sees JS-side writes and runs correctly. Two things I hit while reviewing the unsafe surface:

}

/// The internal representation of an `ArrayBuffer` object.
#[derive(Debug, Clone, Trace, Finalize, JsData)]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ArrayBuffer derives Clone, which is now unsound for the External variant: cloning copies the raw pointer, so two ArrayBuffers alias the same region and can each hand out &mut [u8] to it simultaneously. Owned buffers are fine (the Vec is deep-copied), but an external clone breaks the aliasing invariant from_external_data documents.

Would a manual Clone that deep-copies the External variant into an Owned one (or rejects cloning external buffers) work here?

/// Panics if `ptr` is null.
#[must_use]
pub unsafe fn from_external_ptr(ptr: *mut u8, len: usize) -> Self {
let ptr = NonNull::new(ptr.cast::<AtomicU8>()).expect("`ptr` must be non-null");

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For a SharedArrayBuffer, Atomics and wider views (Int32Array, Float64Array, BigInt64Array) perform aligned atomic accesses. A byte-aligned ptr isn't enough — the base address must satisfy the largest alignment those ops need. Worth asserting/documenting it here, e.g.:

Suggested change
let ptr = NonNull::new(ptr.cast::<AtomicU8>()).expect("`ptr` must be non-null");
let ptr = NonNull::new(ptr.cast::<AtomicU8>()).expect("`ptr` must be non-null");
debug_assert!(ptr.as_ptr() as usize % 8 == 0, "external SharedArrayBuffer backing must be 8-byte aligned");

fn as_slice(&self) -> &[u8] {
// SAFETY: The creator of an `ExternalMemory` guarantees that `ptr` is valid
// for reads and writes of `len` bytes for the whole lifetime of the buffer.
unsafe { slice::from_raw_parts(self.ptr.as_ptr(), self.len) }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more precondition for slice::from_raw_parts: the region size must be <= isize::MAX bytes. It's a documented safety requirement of from_raw_parts, so it'd be worth either asserting it at construction (from_external_data/from_external_ptr) or adding it to the # Safety list the caller must uphold.

// SAFETY: The external region is only ever accessed through atomic operations, and the
// creator of an externally-backed `SharedArrayBuffer` guarantees that the region stays
// valid for the whole lifetime of the buffer, making it safe to share between threads.
unsafe impl Send for SharedData {}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SAFETY comment covers lifetime/validity, but one case worth spelling out: if the external region can be relocated (e.g. a growable WebAssembly memory that moves its base on memory.grow), the stored ptr silently dangles. The invariant the caller must uphold is stronger than "stays valid" — it's "stays valid and unmoved". Might be worth adding that word here and in the # Safety docs, since the wasm-memory use case is exactly where it bites.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

C-Builtins PRs and Issues related to builtins/intrinsics C-Tests Issues and PRs related to the tests. Waiting On Review Waiting on reviews from the maintainers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose a way to back an ArrayBuffer/SharedArrayBuffer with embedder-owned memory

2 participants