RFC: Externref lang item for Wasm targets

[guybedford]

This RFC proposes a new externref lang type (core::arch::wasm32::externref, feature(wasm_externref)): an opaque, unforgeable reference to a WebAssembly host value, lowering to the Wasm externref reference type in function signatures.

Following the semantics of Clang's __externref_t (shipped since Clang 15), it is a bare-position-only type, legal solely as the top-level type of function parameters, return values and local bindings and enforced at type-check time. This allows host references (e.g. JavaScript values) to be marshalled from one foreign call to another directly enabling better performance and interoperability between Wasm ecosystem crates.

Rendered

[programmerjake]

why forbid externref in closure arguments/return values? forbidding it in closure captures is reasonable since you can't store it in structs but a capture-less closure should behave the same as a function -- where externref is allowed.

View changes since the review

[guybedford]

You're correct that this could work mechanically, but it is a consequence of the blanket generics ban - FnOnce<Args> has Args = (externref, ...) which is banned as a generic externref type.

Note that my original version of this proposal only allowed externref in extern block foreign function definitions and extern "C" fn exports. That could still be a possible reframing of this proposal to avoid the complexity of arbitrary function scenarios.

[clarfonthey]

My immediate gut feeling toward this is that simply having the type "opt out" of the type system is a hammer that feels like it will lead to loads of annoying hacks, but the amount of restrictions on externref feel like they wouldn't be possible to cover via other means, which is a compelling reason to not support it.

Essentially, listing all the requirements:

• Cannot be referenced (no &-refs, or even &raw-refs)
• No defined layout (similar to extern type)
• No Type ID (!Any)
• Cannot be moved onto the stack (!Move, except…)
• Can be arguments or return values of C or Rust ABIs

Note that stuff like "never instantiated as a generic" could be useful as lints, but as type-system guarantees they seem fragile.

Note that being ABI-only have interesting consequences, like you could define repr(transparent) structs with them, but it would effectively be stuck with whatever struct you used at the ABI level, based upon what you've defined your extern fns as, which could be helpful.

Basically, I think that it's a steep bar to potentially satisfy all these requirements with explicit type-system details but it feels a lot better than just "this sidesteps the type system entirely" which is gonna lead to so many awful hacks.

View changes since the review

[Nadrieril]

Also, without a lot more opsem work:

• Cannot be the type of a struct/enum field (except maybe transparent structs)

[clarfonthey]

So, I kind of felt that would be implicit due to "no defined layout" but we can add it explicitly too. (I guess that without layout it could still technically be bolted onto the end of a struct but I'm not sure extern types allow that, especially if alignment is undefined.)

[Jules-Bertholet]

For a proposal like this one, if you want the slightest chance of acceptance, you cannot afford to skimp on this section or the Rationale and Alternatives.

First, do not assume everyone in your audience has more than passing familiarity with WASM, or any idea what an externref is. You will need to explain this in the RFC itself. RFC 3884 is a good example of providing background for a niche target feature.

Second, to convince the lang team to accept an invasive and controversial change like this one, you will need to, at minimum, prove that nothing less can meet the needs of WASM users. You cite two major motivations; you should elaborate on both:

• Interoperability and fewer custom wrappers: What are the specific patterns that break because of lack of interoperability, that would become possible with this proposal? And in the Rationale, why can't interoperability be achieved another way, e.g. standardizing on a common table system (similar to the C ABI and other tool conventions)? Ideally, give examples from specific real-world projects.
• Performance and optimization: What are the specific usage patterns you anticipate for this feature (taking into account the many limits you impose on how externref can be used)? And in the Rationale, why is this proposal the only reasonable way to meet the performance requirements of those specific patterns? Ideally, you should cite benchmarks from a real-world project.

View changes since the review

[Jules-Bertholet]

Note that whether or not this RFC ends up accepted in something like its current form, none of this work will be wasted. By fully describing the problem space, you enable everyone working on Rust to consider how what they are working on or thinking about intersects with Wasm's specific needs, and the end result will be a language that better supports them.

[Jules-Bertholet]

Please elaborate here as well.

• Clang / LLVM __externref_t: What was their motivation for doing this? Are people using and enjoying this feature despite the restrictions on where __externref_t can appear? Did they face any unexpected challenges, and how did they overcome them? Link to their proposals and documentation.
• "wasm-bindgen externref mode: post-link signature rewriting via walrus": What is this? Why is it "motivating precendent"?
• Wasm-GC languages: So, is this feature necessary to do FFI with those languages on WASM? If so, that's great motivation! Make it part of your case, don't be shy about it. How is "storing [externref] in language-level aggregates" represented in FFI?

View changes since the review

[Darksonn]

Could we change wasm to make this type less special?

You've explained that the table representation is currently bespoke, which causes interoperability problems. But why can't wasm just pick one and standardize it on the wasm side? Then it'd no longer be a bespoke representation, but just the standard ABI and we can build that into the language and support it as a completely normal type with integer size.

View changes since the review

[guybedford]

This is covered in the future possibilities section briefly but I could expand on it further.

Basically, the shared table would support &externref as the storable variant as an additional feature (__externref_t* in llvm/llvm-project#201466) This layers onto the base externref support though which itself keeps all the strong guards defined by this RFC.

So the design of this externref remains a requirement even when &externref exists - none of the work in this proposal is wasted and the layering works out naturally.

If we made externref just an alias for __externref_t* ignoring support for __externref_t then that would force all FFI values to be table indices, which is exactly the situation we are trying to get away from here.

If we made externref a magical alias that picks one or the other representation depending on usage position, that would break a whole bunch more invariants in having a far more complex Heisenberg type.

[RalfJung]

The semantics of this RFC cannot be supported by the Abstract Machine that we have developed so far to talk about Rust. That's a big setback since that Abstract Machine is needed for us to confidently answer questions like "is this MIR transform correct" or "are we correctly translating MIR to LLVM IR". In other words, as-is this RFC robs us of the ability to precisely reason about MIR, which is inacceptable. This isn't even properly discussed as a drawback, and no attempt is made at sketching what the Abstract Machine for Rust could look like with externref. You can look at MiniRust as a basis if you want to get an idea for what an Abstract Machine for current Rust could look like. (The RFC talks about the fact that this breaks type system assumptions, but that's only a small fraction of the problem. We've had RFCs that break type system assumptions before. This RFC breaks Abstract Machine assumptions. We've never had an RFC anything like this.)

Usually the approach to represent "external resources" in a language like Rust is something similar to file descriptors: a number into a table that is managed outside the realm of the language and that we can only talk about indirectly. The RFC mentions tables but only as "third-party" options. But a table does not have to be third-party, the intrinsics required for that could be in libcore. Given the gigantic cost of the proposed solution, the RFC needs to spend a comparable gigantic amount of space and effort explaining why the table solution, even if it became a first-party concept with good language and library integration, is so bad that we have to undertake the biggest change of the language since Rust 1.0 in order to avoid it.

View changes since the review

[RalfJung]

There 's a major drawback that's not discussed enough IMO: none of the existing language and library ecosystem works for this, meaning it will be a major pain to do anything with these types. You cannot return them from Result functions, you cannot use them in async blocks/functions, you cannot use == on them, you cannot newtype them... this RFC is proposing to introduce a fundamentally new kind of thing to Rust only to then make that thing a complete nightmare to use for anything.

The result of this will not feel like "Rust supports externref now". It will feel like "externref has been bolted on to Rust in a poorly integrated unergonomic way".

View changes since the review

[guybedford]

It's an FFI feature primarily, yes.

[guybedford]

Would it help if we formally restricted externref to only extern "C" and extern block functions? That would still provide the major benefit IMO.

[Jules-Bertholet]

I don't think that makes a difference. As long as values of this type are passing through Rust code, we need a full semantics to understand what is happening.

It's an FFI feature primarily, yes.

As I've suggested elsewhere, please please provide a full fleshed-out example so that those of us who don't know Wasm can fully grok your use-case. (Take all the time you need, don't rush it)

[ds84182]

You could skip all of the language changes if externref naturally decayed to an index within a table when it needs to be stored to memory. This needs runtime support. Similar to panic handlers, the compiler would depend on externally provided table allocation and free functions.

Otherwise I don't think this RFC will make any reasonable progress.

[programmerjake]

You could skip all of the language changes if externref naturally decayed to an index within a table when it needs to be stored to memory.

I think this is the third time that something like that was proposed -- I proposed it on Zulip. I do think that having the Rust type be an index into a table except when passed by value in function args/return values (maybe limited to extern "C") is a good idea, and that LLVM can help with optimizing to remove redundant table allocations/reads/writes.

[RalfJung]

Would it help if we formally restricted externref to only extern "C" and extern block functions? That would still provide the major benefit IMO.

One can define and call extern "C" functions from Rust so on its own that doesn't really help.

I think this is the third time that something like that was proposed -- I proposed it on Zulip. I do think that having the Rust type be an index into a table except when passed by value in function args/return values (maybe limited to extern "C") is a good idea, and that LLVM can help with optimizing to remove redundant table allocations/reads/writes.

Yeah that makes sense: have this behave like a normal type with a special ABI. ABIs already involve special concepts we cannot represent in Rust, like registers. Having a type that's a table index normally but a bare externref on the ABI will stretch the limits of our already overstretched ABI handling in rustc, but at least that is "just" an implementation concern.

[guybedford]

I'm not completely opposed to this framing, and see the attraction of forcing the opqaue host value semantics to the very edge of the FFI boundary. I can certainly explore the a cross-toolchain prototype further on this and share back what I find.

[TimNN]

I've thought a bunch about "externref in Rust" in the past (though it's been quite a while), including prototyping some bits.

I think my latest conclusion was also that representing externref as a table index was the cleanest way to represent it in Rust. (Because introducing it as an "abstract value" would require significant changes to / extensions of the language).

The main downside I saw was that it makes the table index conversion implicit, hiding the associated cost.

(At the time I was thinking that (post-mono) MIR (or LLVM) optimizations could even represent&externref as externref if it's address is never taken).

[RalfJung]

wasm-bindgen's JsValue does not interoperate with other custom table wrapping systems.

So let's define a standard table system for Rust that's used across the ecosystem, and let's design it in a way that it can be picked up by other languages as well (or can be adjusted to eventually achieve interop with other languages, once anyone else is interested).

View changes since the review

[Diggsey]

WASM supports multiple tables and has two different table types so far - https://developer.mozilla.org/en-US/docs/WebAssembly/Reference/Definitions/table

It's also possible that it might support doing "more things" with externref or funcref types in future, which could be hard to retrofit onto an opinionated ABI-based approach.

I wonder if this could be solved via https://doc.rust-lang.org/beta/core/arch/macro.global_asm.html instead?

Instead of trying to build a parallel set of semantics into the Rust AM, just use global ASM to turn it into operations that do make sense in the Rust AM?

[RalfJung]

The best you can do with global asm is implement the table yourself, I think? The actual values flowing through Rust can still at best be table indices.

It's also possible that it might support doing "more things" with externref or funcref types in future, which could be hard to retrofit onto an opinionated ABI-based approach.

Well, languages based on addressable memory are hard to retrofit onto an opinionated semantics with primitive reference types...

[Diggsey]

The best you can do with global asm is implement the table yourself, I think? The actual values flowing through Rust can still at best be table indices.

Yes, I'm suggesting that this might be the best approach.

Well, languages based on addressable memory are hard to retrofit onto an opinionated semantics with primitive reference types...

Well, quite. For example, these operations are already in WASM, and an opinionated ABI-based approach could support some of them via intrinsics, but many would be hard or impossible I think:

Operation / instruction	funcref	externref	Result / effect	Notes
ref.null func	✅	—	Produces a null funcref	For the function-reference hierarchy
ref.null extern	—	✅	Produces a null externref	For the external-reference hierarchy
ref.func $f	✅	—	Produces a non-null reference to $f	The function must be declared for reference use
ref.is_null	✅	✅	Produces i32	1 for null, otherwise 0
ref.as_non_null	✅	✅	Converts nullable reference to non-null reference	Traps when the operand is null
ref.eq	✅	✅	Produces i32 identity comparison	Both operands must satisfy the instruction’s reference-type constraints; it does not inspect their contents
ref.test <type>	✅	Limited	Produces i32	Runtime type test; mainly useful with typed function and GC reference hierarchies
ref.cast <type>	✅	Limited	Produces a narrowed reference	Traps if the runtime type does not match
drop	✅	✅	Discards the reference	Works on every value type
select	✅	✅	Selects one of two references	Both alternatives must have a compatible type
select (result <reftype>)	✅	✅	Typed selection	Explicit result type
local.get	✅	✅	Reads a local reference	Local must have the corresponding reference type
local.set	✅	✅	Writes a local reference
local.tee	✅	✅	Writes and retains the reference
global.get	✅	✅	Reads a global reference
global.set	✅	✅	Writes a mutable global reference	Global must be mutable
table.get	✅	✅	Reads a reference from a table	Table element type must be compatible
table.set	✅	✅	Writes a reference into a table
table.grow	✅	✅	Grows a table using a reference as the initial value	Returns the old size or -1
table.fill	✅	✅	Fills table entries with a reference
table.copy	✅	✅	Copies references between tables	Source and destination element types must be compatible
table.init	✅	✅	Initializes a table from an element segment	Element segment type must be compatible
table.size	✅	✅	Produces the table size	Does not operate on an individual reference value
elem.drop	✅	✅	Discards an element segment	Does not operate on an individual reference value
call_indirect	✅	❌	Calls the referenced function from a table	Performs a runtime signature check
return_call_indirect	✅	❌	Tail-calls a function from a table	Requires tail-call support
call_ref	✅	❌	Calls a typed function reference directly	Requires a typed function reference, not merely an unrefined funcref
return_call_ref	✅	❌	Tail-calls a typed function reference	Typed function references plus tail calls
Function parameter passing	✅	✅	Passes a reference into a function	Works for Wasm-defined and imported functions
Function result	✅	✅	Returns a reference	Works for Wasm-defined and imported functions
call / imported call	✅	✅	May consume or produce references	According to the callee’s declared signature
return	✅	✅	Returns reference-typed results	According to the current function signature
br	✅	✅	Transfers references as label arguments	According to the target block type
br_if	✅	✅	Conditionally transfers references
br_table	✅	✅	Transfers references to one of several labels	All target label types must be compatible
block, loop, if parameters/results	✅	✅	Carries references through structured control flow	Reference types can appear in block signatures
Import/export	✅	✅	Exchanges references across the host boundary	Subject to the embedding API
any.convert_extern	❌	✅ input	Converts externref to an internal anyref representation	GC/reference conversion; does not reveal the host object
extern.convert_any	❌	✅ output	Converts an internal anyref back to externref	Inverse of any.convert_extern

[guybedford]

Very explicitly the goal of this proposal is typing JavaScript interop, NOT Wasm GC interop.

That means we care about JS builtin types (wasm:js/ intrinsic imports), not arbitrary GC ops.

[juntyr]

While the proposed design would mirror externref exactly, I think a non-zero cost wrapper could work better. The wrapper would still be special in that any stores or loads to memory would be translated into stores and loads into a Rust-owned table. As an optimisation, a pass could then look for cases where an externref in fact only passed as a function parameter and remove the table calls. In that case, Rust users could use an almost-normal opaque type, and cases that are written to comply with externref's usage rules would be optimised to compile to the same code as using this restricted proposal

View changes since the review

[guybedford]

In this scenario, how would you distinguish between the opaque form, and actually getting the index of an &externref on an FFI boundary? Or would externref mean different things depending on usage site? How to categorize those cases and avoid &&externref being amgiuous?

[juntyr]

My idea would be that externref in Rust-only code is effectively a table index, but that at FFI boundaries (when defining or calling extern "C" functions), the actual WASM externref is passed, with table index conversions happening right at the boundary. A reasonably clever optimisation pass could then see that

unsafe extern "C" {
    fn create_ref() -> externref;
    fn use_ref(v: externref);
}

pub fn run() {
    unsafe {
        let r = create_ref();
        use_ref(r);
    }
}

which would lower to a externref -> table insert -> table extract -> externref sequence, could be replaced by just passing the externref directly.

[ds84182]

If anything, I don't think Clang should be a role model. __externref_t is a non-standard extension to C and C++ implemented in Clang to handle a bespoke target feature. Other compilers, like GCC and MSVC, do not have this (or any type like this).

More generally, I do not think languages should change at such a core level to support something extremely niche like this. WebAssembly has so many oddities that have to be papered over due to fundamental issues in its design. Those oddities should be dealt with by the compiler and/or runtime.

View changes since the review

[guybedford]

If anything, I don't think Clang should be a role model

Certainly that makes sense, and I agree, the proposal should be justified independently for Rust and not blindly follow any semantics based purely on precedent.

More generally, I do not think languages should change at such a core level to support something extremely niche like this.

IMO, JavaScript interop is not a complex or niche topic and this is a major quality of life improvement to that story. Rust has solved far harder and far more niche problems!

[TimNN]

Not sure if it's worth including it here, but I was somewhat interested in this topic in the past and wrote a bit about it:

• https://hackmd.io/@TimNN/SJh8MNiN6 was my explanation of the problem, describing the different implementation options I saw.
• https://hackmd.io/@TimNN/Hk1LO6iMJe was an attempt to fit this more cleanly into the type system (based on the new Sized Hierarchy RFC) , by introducing AbstractValue and AbstractCopy traits.

View changes since the review

[TimNN]

Should this (briefly) mention how this could be extended to support WASM GC types?

Also, supporting transparent wrapper types around externref seems like a relatively simple future extension.

View changes since the review

[TimNN]

I've thought a bunch about "externref in Rust" in the past (though it's been quite a while), including prototyping some bits.

I think my latest conclusion was also that representing externref as a table index was the cleanest way to represent it in Rust. (Because introducing it as an "abstract value" would require significant changes to / extensions of the language).

The main downside I saw was that it makes the table index conversion implicit, hiding the associated cost.

(At the time I was thinking that (post-mono) MIR (or LLVM) optimizations could even represent&externref as externref if it's address is never taken).