Add a DSL → JavaScript transpiler backend (jit_backend="js")#670
Merged
Conversation
Transpile a @blosc2.dsl_kernel (the bounded Python-ast subset DSLValidator accepts) to JavaScript, so kernels run at V8-optimized native speed in the browser/Pyodide. Wired into chunked_eval as a new backend alongside no-JIT and miniexpr-JIT: the kernel becomes a plain per-block UDF callable, so there are no compiled-code changes and the default path is untouched. Verified end-to-end under Pyodide (Newton 320x213, 24-frame sweep): correctness exact vs numpy, ~2x faster than the miniexpr JIT, ~8x over no-JIT. The bridge must hand the JS driver real JS Arrays (not Python lists), else the hot loop crosses the Python<->JS boundary per element (~10x slower). - src/blosc2/dsl_js.py: dsl_to_js(), build_js_module(), js_kernel() bridge - src/blosc2/lazyexpr.py: _as_js_udf() + jit_backend="js" swap in chunked_eval - tests/ndarray/test_dsl_js.py: transpiler + node numeric-equivalence tests - bench/js-transpiler/: headless Node+Pyodide bench, browser demo, README - plans/dsl-js.md: design and verified bench findings
…ack) Under WASM, a float/transpilable/non-reduction DSL kernel now auto-routes to jit_backend="js" unless the user opts out; anything JS can't do (non-float dtypes, reductions, unsupported constructs) silently falls back to miniexpr, so there's no regression. Since JS is itself a JIT, jit=True prefers it too — only jit=False, strict_miniexpr=True, or an explicit jit_backend opts out (force miniexpr with jit_backend="tcc"/"cc"). - lazyexpr.py: _maybe_js_backend prefer-js logic + _js_dtypes_ok gating; "js" documented and listed for the jit_backend param. - test_dsl_kernels.py: autouse fixture keeps this miniexpr-semantics module on miniexpr (the prefer-js default would bypass its _set_pref_expr assertions). - test_wasm_dsl_jit.py: native-CI coverage for explicit js, the prefer-js default, and the int fallback (counterpart of the node overlay harness). - bench/js-transpiler: kernel sweep (newton/poly/trans/deep/deepar) showing the js-vs-tcc win is kernel-dependent (~2x arithmetic, ~1x transcendental/light).
…dled in the windows wheels
Add P1 (index/shape symbols _i0/_n0/_flat_idx) by emitting them as trailing kernel params and reconstructing per-block global coords in the driver, and P2 (integer inputs with floating output, matching miniexpr's float promotion). Memoize transpile + js.eval so repeated evaluations don't re-parse/re-compile, which closes most of the gap on light kernels. Update tests, the node bench (new P1/P2 kernels, streamed rows), and plans/dsl-js-coverage.md.
The explicit JS path bypassed the output-dtype check, so an integer/complex output kernel would silently compute in float64 instead of failing. Raise a clear ValueError up front (matching the documented "explicit js raises on gaps" contract); integer inputs with a floating output still use JS. Keeps integer output entirely on miniexpr. Add a test and update the coverage plan.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a new execution backend for
@blosc2.dsl_kernelkernels that transpiles them to JavaScript and runs them in-browser (Pyodide/WASM), where V8 JIT-compiles the scalar loop to native code. On compute-heavy kernels it beats blosc2's WASM TinyCC JIT substantially.What's new
JS transpiler (
src/blosc2/dsl_js.py)where,if/elif/else,rangeloops,**////%, and a whitelist of math functions)._i0/_n0/_flat_idx, …): emitted as trailing kernel params; the runtime driver reconstructs each element's global coordinate per block from(offset, gshape, cshape).__runare memoized so repeated evaluations (e.g. animation loops) don't re-parse/re-eval.Backend routing (
src/blosc2/lazyexpr.py)jit_backend="js"transpiles or raises;jit=False/strict_miniexpr=True/ an explicitjit_backendopt out.Coverage (what runs on JS vs miniexpr)
_i0/_n0/_flat_idx)where,if/elif/else,range, math whitelistFull breakdown and remaining work (integer output, zero-copy block I/O) in
plans/dsl-js-coverage.md.Performance (Pyodide, ms/frame;
bench/js-transpiler/dsl-js-node.mjs)JS wins on compute-heavy kernels and is at parity/slightly behind on light, vectorizable ones (where per-block marshaling dominates):
The remaining sub-1.0 cases are marshaling cost (the bridge copies blocks in/out; miniexpr computes in place), documented as a future zero-copy
HEAPF64optimization.Tests & CI
tests/ndarray/test_dsl_js.py) and WASM integration tests (tests/ndarray/test_wasm_dsl_jit.py) covering index symbols, integer inputs, prefer-js selection, and silent fallback.wait_untilpoll instead of a singlepilot.pause()).