🛠️[v0.3.42] Release Note: More Reliable Dynamic Backend Loading, Safer MTMD Processing, and Advanced Batch Support

[JamePeng]

Release Note: 0.3.42

Better DLL Loading, Safer Batching, and Stronger Runtime Validation

It has only been a short time since the last release, but this update brings several important stability improvements and prepares the foundation for upcoming features.

The main focus of 0.3.42 is improving runtime reliability: from Windows backend loading, native error handling, MTMD validation, to more flexible batch management for future speculative decoding support.

Highlights

More Reliable Windows DLL Loading

One of the key improvements in this release is a more robust Windows DLL loading strategy.

Previously, bundled backend libraries could sometimes fail to resolve their dependencies because the DLL search order did not always prioritize the correct location.

This release:

• Removes duplicated Windows DLL directory registration logic
• Adds optional CUDA, HIP, and Vulkan runtime DLL search paths
• Ensures bundled libraries follow the correct priority order:
  /lib > /bin
• Adds detailed diagnostics for system and bundled library loading sources

This is especially important for multi-DLL backends.

For example, single-DLL backends usually work with /bin as a fallback path, but some environments may still fail because of missing dependencies such as libomp140.x86_64.dll.

For multi-DLL backends like SYCL, which relies on additional libraries (dnnl.dll, tbb12.dll, mk_*.dll, etc.), missing dependency resolution could cause ggml-sycl.dll loading failures and even trigger access violation crashes.

This update makes /lib the first lookup location and improves backend loading reliability.

Special thanks to @allanmeng for reporting and testing the SYCL backend issue.

---

Safer Native Runtime Boundaries

Several changes improve protection around Python ↔ native API interactions:

• Validate token IDs before calling native decode functions
• Prevent invalid tokens from reaching the backend where they could cause hard crashes
• Improve native exception diagnostics
• Add clearer debugging information for failed decode operations

These changes make runtime failures easier to understand and diagnose.

---

Stronger MTMD Input Validation

MTMD processing now performs more validation before entering the native tokenizer path.

Improvements include:

• Validate media markers before tokenization
• Handle missing bitmap inputs more safely
• Detect media marker and bitmap count mismatches
• Provide richer error messages with marker details and backend information

This helps prevent difficult-to-debug multimodal input failures.

---

Preparing LlamaBatch for Speculative Decoding

This release continues improving LlamaBatch as a foundation for future speculative decoding features.

New capabilities include:

• Embedding-only batch support
• Mixed token + embedding batch support
• Safer token/embedding ownership handling
• Additional validation for sequence IDs and batch writes

These changes prepare the API for future decoding paths such as EAGLE3 and MTP, where models may require both token IDs and hidden-state embeddings during decoding.

---

Other Updates

Additional improvements:

• Load ggml-base before ggml for more reliable dynamic initialization
• Make assistant message name optional in type definitions
• Sync latest llama.cpp / ggml / mtmd API changes

---

What's Next

This release mainly focuses on making the runtime foundation more stable.

With backend loading becoming more reliable and batch infrastructure becoming more flexible, the next steps will continue moving toward better speculative decoding support, including unified handling for different approaches such as MTP, EAGLE-style decoding, and other acceleration strategies.

Thanks to everyone who reported issues, tested changes, and helped improve the project.

- JamePeng