Skip to content

[CoreCLR] dlopen()-based assembly store loading#12033

Open
simonrozsival wants to merge 12 commits into
mainfrom
dev/simonrozsival/dlopen-assembly-store
Open

[CoreCLR] dlopen()-based assembly store loading#12033
simonrozsival wants to merge 12 commits into
mainfrom
dev/simonrozsival/dlopen-assembly-store

Conversation

@simonrozsival

@simonrozsival simonrozsival commented Jul 10, 2026

Copy link
Copy Markdown
Member

Summary

The CoreCLR host used to locate the assembly store by parsing the APK ZIP central directory, mmap-ing the wrapper .so, and walking its ELF section headers by hand to find the payload. Since the store is already a valid ELF shared library (#9154), this PR instead makes its payload a loadable section (SHF_ALLOC, in a PT_LOAD segment) exposed via an exported dynamic symbol, so the host just:

void *h     = dlopen("libassembly-store.so", RTLD_NOW | RTLD_LOCAL);
void *start = dlsym(h, "_assembly_store");   // -> AssemblyStoreHeader

…and lets the platform dynamic linker locate + map it out of the APK. The hash-index lookup (AssemblyStore::open_assembly) is unchanged.

This is CoreCLR-only. MonoVM is untouched and keeps the classic DSOWrapperGenerator + ZIP-scan store. The mechanism is hardcoded by runtime and cannot be overridden (_AndroidDlopenAssemblyStore = true for CoreCLR, false for MonoVM), because after this change the CoreCLR host has no ZIP fallback and MonoVM's host doesn't understand the loadable-symbol layout.

What's removed (CoreCLR)

  • src/native/clr/startup/zip.cc in its entirety (the ZIP central-directory parser) + the xamarin-startup static library.
  • Host::zip_scan_callback, AssemblyStore::map(fd/offset), and Util::get_wrapper_dso_payload_pointer_and_size usage (the ELF section-header walk).
  • The DSOApkEntry {fd,offset} native-library fast path in MonodroidDl (it was populated by the same ZIP scan). Bundled native libraries now load via dlopen-by-name, which works because they're uncompressed + page-aligned in lib/{ABI}/ (extractNativeLibs=false). The now-dead DSOApkEntry struct, its dso_apk_entries[] emission in ApplicationConfigNativeAssemblyGeneratorCLR, and the stub definition are removed too (MonoVM keeps its own copy).

Net: 889 lines deleted, 197 added (−692 net). The arm64 CoreCLR host (libnet-android.release.so) shrinks 34,784 B (−2.73%) in a clean same-tree isolation (referenced code removed; arm −3.05%, x64 −3.34%); the actual shipped main-vs-PR APK host delta is −28,640 B (see the Size table below).

Why this hasn't been done before (history)

dlopen-based loading was never used, and this is not a regression from a nicer past design — ZIP-scan + mmap is the original and only design, three layers accreted over the years:

When PR What changed Loading
2019 #2570 assemblies mmap-ed in place from the APK (never extracted) ZIP-scan + mmap
2021 #6311 assembly store introduced (assemblies.blob, O(n)→O(1)) raw blob, ZIP-scan + mmap
2024 #8478 all assemblies RID-specific; store moved to lib/{ABI}/, renamed lib*.so (not real ELF) ZIP-scan + mmap
2024 #9154 wrapped in a real ELF (objcopy --add-section, non-loadable, 16 KB-aligned) for the Android 16 KB page-size requirement ZIP-scan + mmap + ELF section-header walk
2025 #9778, #9572 CoreCLR host re-implements the same approach same

dlopen only became possible in #9154 (the store became real ELF), and even then the payload was deliberately kept non-loadable so the existing mmap loader could be reused — on the unverified assumption (see ApkSharedLibraries.md: "…without having to load the ELF image into memory") that mmap-the-section beats loading the ELF. This PR measures that assumption.

Benchmark results

Measured on a physical Samsung A16 (SM-A165F), net11.0-android MAUI sample-content app (dotnet new maui --sample-content), Release, arm64, CoreCLR, extractNativeLibs=false, 30 interleaved cold-starts (am start -W TotalTime).

Apples-to-apples (both built from the same local P7 SDK — isolates the store mechanism)

Three independent 30-interleaved rounds, main (classic ZIP-scan) vs this PR (dlopen). Rounds 1–2 reused a fixed dlopen APK; round 3 rebuilt latest fresh from the committed code (verified store exports _assembly_store, section payload is loadable, host has no DT_NEEDED):

Round main mean dlopen mean Δ (dlopen − main) paired test
1 2059.7 ms 2076.1 ms +16.4 ms (+0.80%) t=2.31, p≈0.03, CI [+1.9, +31.0]
2 2061.1 ms 2054.4 ms −6.6 ms (−0.32%) t=−1.00, p≈0.32, CI [−19.7, +6.4]
3 (fresh) 2056.0 ms 2032.3 ms −23.6 ms (−1.15%) t=−3.95, p≈0.0004, CI [−35.4, −11.9]

The three rounds span +16 → −24 ms (a ~40 ms between-session spread that exceeds any single round's effect) ⇒ the cross-session noise floor dominates. dlopen is startup-neutral — within noise, if anything trending faster, and never a regression.

Size (arm64, extractNativeLibs=false, all Stored/uncompressed in the APK; round-3 fresh build)

Component main dlopen Δ
APK total 29,670,166 29,637,398 −32,768 B
libmonodroid.so (host) 1,264,464 1,235,824 −28,640 B (−2.27%)
libassembly-store.so 7,674,048 7,674,784 +736 B (negligible)

(The host shrinks because the ZIP parser + ELF section-header walk + DSOApkEntry path are gone. Clean referenced-code isolation on the same tree: arm64 −34,784 B, arm −3.05%, x64 −3.34%.)

Is there a real difference? No — it's noise

The dlopen+dlsym call itself was measured on-device against the real 7.6 MB store:

dlopen = 0.14–0.49 ms      dlsym = ~0.006 ms      (reads magic 0x41424158 "XABA")

That's ~0.02% of a ~2050 ms cold start — far below the ±20–40 ms cross-session variance. Deleting the ZIP scan + ELF walk is provably not a startup regression.

⚠️ An earlier naive comparison (stock global .NET 11 preview 5 vs this local preview 7 build) showed +72 ms — but that was confounded by the P5→P7 delta and official-vs-local-build differences, not the dlopen change. Building main with the same local P7 toolchain isolates it to the ~0 ms above. (The residual local-P7-vs-global-P5 gap is likely partly #11730, the LZ4→Zstd store-compression switch merged after P5.)

Can we optimize the dlopen further? No — there's nothing to skip

The store .so is inspected with readelf: 0 relocations, no DT_NEEDED, no DT_INIT/init-array, two PT_LOAD segments. The linker therefore does no relocation, dependency resolution, or constructor work — it just mmaps the (demand-paged) segments + minor bookkeeping. Concretely:

  • android_dlopen_ext has no "skip relocations/init" flag — and there's nothing to skip anyway.
  • RTLD_LAZY vs RTLD_NOW is moot (no code → no PLT relocations). We pass RTLD_LOCAL (we only dlsym our own handle, so there's no need for RTLD_GLOBAL's extra global-scope bookkeeping), though the difference is immaterial at <0.5 ms.
  • The bionic apk!/lib/{abi}/lib*.so zip-path syntax (how extractNativeLibs=false is implemented) works, but measured slower on-device (~0.52–0.98 ms — the linker still opens the zip + does a CD lookup for the entry) and would re-introduce runtime APK-path / split-config handling. The by-soname dlopen is simpler and already at the floor.

Follow-ups (out of scope)

  • Pairs naturally with [experiment] On-device decompressed AssemblyStore cache (CoreCLR) #11967 (on-device decompressed store cache): the decompressed cache can itself be the dlopen-able wrapped .so, so warm starts skip both Zstd decompression and any ZIP work; android_dlopen_ext(USE_LIBRARY_FD[_OFFSET]) is the right tool to load it by fd.
  • The now-orphaned Util::get_wrapper_dso_payload_pointer_and_size definition can be deleted as pure dead-code cleanup.
  • The runtime-config blob uses the same wrapper mechanism and could adopt the same symbol-based loading.

Validation

  • CoreCLR app built with no flag auto-produces a store exporting _assembly_store (hardcode works) and cold-launches cleanly (Status: ok).
  • Store is Stored (uncompressed) + 16 KB-aligned; dlsym returns a pointer to a valid AssemblyStoreHeader (magic XABA).
  • The wrapper's payload lives in a section named payload (matching the MonoVM DSOWrapperGenerator and tools/assembly-store-reader-mk2), so the CoreClrTrimmableTypeMap* build tests that inspect the APK through the reader keep working.
  • The whole read-only store pointer chain (configure_from_payload(const void*), data_start, image_data/debug_info_data/config_data, assemblies, descriptor, assembly_store_hashes) is const — no const_cast on the store path.

Reference PRs

simonrozsival and others added 6 commits July 10, 2026 10:53
Prototype (additive, CoreCLR-only; Mono untouched) that lets the runtime
locate the assembly store via dlopen()+dlsym() instead of parsing the APK
ZIP central directory, mmapping the wrapper .so and walking its ELF section
headers by hand.

Build side (opt-in via -p:_AndroidDlopenAssemblyStore=true):
* New DlopenAssemblyStoreGenerator wraps the store blob into a shared library
  whose payload lives in a *loadable* section (SHF_ALLOC, covered by PT_LOAD)
  pointed at by an exported dynamic symbol `_assembly_store_start`. It emits a
  tiny `.incbin` stub assembled with the shipped `llvm-mc` and linked with
  `ld` (no clang needed). This differs from DSOWrapperGenerator, which uses
  `llvm-objcopy --add-section` to inject a *non-loadable* section.
* WrapAssembliesAsSharedLibraries gains a UseDlopenAssemblyStore switch,
  plumbed from the new _AndroidDlopenAssemblyStore MSBuild property. The
  existing DSOWrapperGenerator path is left intact.

Runtime side (self-detecting, no runtime flag):
* AssemblyStore::map is refactored to extract configure_from_payload(); a new
  map_from_pointer() configures the store from an in-memory payload pointer.
* Host::gather_assemblies_and_libraries first tries dlopen(libassembly-store.so)
  + dlsym(_assembly_store_start). On success it skips the ZIP scan entirely; if
  the symbol is absent it falls back to the existing scan, so a prebuilt host
  handles both old and new APKs.

Feasibility of the loadable-symbol wrapper was validated on an arm64 emulator
(dlopen+dlsym resolves the payload; objcopy --add-section does not). Startup
A/B benchmarking is still pending.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 527f04b6-c95a-4a66-b90f-4c52ba5496f3
Take a `() -> std::string` path provider (templated) instead of an eagerly
constructed std::string, so the full store path is only built when it is
actually needed - i.e. in the abort_application() branches. This restores the
original map() behavior, where the path was computed by a lambda only on
failure (or when the LOG_ASSEMBLY-guarded debug log is enabled), rather than
on every successful load.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 527f04b6-c95a-4a66-b90f-4c52ba5496f3
…ayload directly

Remove the thin map_from_pointer() wrapper and have Host call the now-public
AssemblyStore::configure_from_payload() directly, passing the path provider
lambda. configure_from_payload no longer needs to be a template: it takes a
`const std::function<std::string()>&`, which is still only invoked to build
diagnostics on the invalid-payload abort paths.

Also fix DlopenAssemblyStoreGenerator to use `List<string>` with collection
expressions (a bare `var x = [...]` has no target type -> CS9176).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 527f04b6-c95a-4a66-b90f-4c52ba5496f3
Commits to the dlopen()-based assembly store for CoreCLR and removes the code it
replaces. The store is wrapped as a real shared library whose payload is exposed
via the `_assembly_store_start` dynamic symbol; the host resolves it with
dlopen()+dlsym() and lets the dynamic linker locate + map it out of the APK.

Runtime (src/native/clr), CoreCLR only:
* Host::gather_assemblies_and_libraries now only does dlopen()+dlsym(); there is
  no ZIP-scanning fallback. It aborts if the store lacks the symbol.
* Removed zip.cc entirely (the ZIP central-directory parser), the xamarin-startup
  static library, zip_scan_callback, AssemblyStore::map(fd/offset) and the ELF
  section-header walk it used to reach the payload.
* Removed the DSOApkEntry fd/offset native-library fast path from MonodroidDl
  (it was populated by the same ZIP scan); bundled native libraries now load via
  dlopen()-by-name, which works because they are uncompressed and page-aligned in
  lib/{ABI}/ (extractNativeLibs=false).
* The scan_filesystem (fastdev) path also loads the store via dlopen().

Build tasks:
* `_AndroidDlopenAssemblyStore` is now hardcoded by runtime and MUST NOT be
  overridden: true for CoreCLR (its host only supports the dlopen store), false
  for MonoVM (its host only supports the classic scanned store).
* WrapAssembliesAsSharedLibraries selects DlopenAssemblyStoreGenerator (CoreCLR)
  vs DSOWrapperGenerator (MonoVM) accordingly.

MonoVM is untouched and keeps the classic DSO-wrapper + ZIP-scan store.

Isolated cold-start impact (both local P7, MAUI sample-content, Samsung A16, 30
interleaved runs) is within noise: dlopen +16 ms (+0.8%) vs the classic path,
while dlopen()+dlsym() of the store measures <0.5 ms on-device. The arm64 CoreCLR
host shrinks ~34 KB (-2.7%).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 527f04b6-c95a-4a66-b90f-4c52ba5496f3
We only dlsym() our own handle, so there is no need to add the store's symbols to
the global lookup scope. RTLD_LOCAL avoids the extra linker bookkeeping that
RTLD_GLOBAL would incur.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 527f04b6-c95a-4a66-b90f-4c52ba5496f3
@simonrozsival simonrozsival added the Area: CoreCLR Issues that only occur when using CoreCLR. label Jul 10, 2026
simonrozsival and others added 6 commits July 10, 2026 15:14
…ectness

Three refinements to the dlopen-based assembly store:

- Fix CI: name the wrapper's payload section `payload` (not `.payload`) so
  tools/assembly-store-reader-mk2 (Utils.FindELFPayloadSectionOffsetAndSize
  looks up "payload") and the CoreClrTrimmableTypeMap build tests that read
  the APK via it recognize the store again. Matches the section name the
  MonoVM DSOWrapperGenerator already uses. The runtime is unaffected: it
  resolves the exported dynamic symbol, not the section name.

- Rename the exported payload symbol `_assembly_store_start` -> `_assembly_store`.

- Make the whole read-only assembly-store pointer chain const so there are
  no const_casts on the store path: configure_from_payload() takes
  `const void*`, and data_start, image_data, debug_info_data, config_data,
  assemblies, descriptor and assembly_store_hashes are now const pointers.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 527f04b6-c95a-4a66-b90f-4c52ba5496f3
With the dlopen-based store the CoreCLR host no longer consumes the
DSOApkEntry {name_hash,offset,fd} table (the ZIP-scan native-library fast
path was removed earlier in this PR). Delete the now-orphaned CoreCLR-side
dead code:

- the `dso_apk_entries` LLVM IR emission + `DSOApkEntry` struct/data-provider
  and PopulateDsoApkEntries() in ApplicationConfigNativeAssemblyGeneratorCLR
- the `DSOApkEntry` struct, `dso_apk_entries[]` extern declaration, and stub
  definition in the CoreCLR xamarin-app.hh / application_dso_stub.cc

MonoVM still uses DSOApkEntry (its own ApplicationConfigNativeAssemblyGenerator
+ mono/xamarin-app-stub/xamarin-app.hh) and is left untouched.

Verified: native builds + links clean (all ABIs), and a net11.0-android MAUI
sample-content app builds, links, and cold-launches (Status: ok) with no
dso_apk_entries present.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 527f04b6-c95a-4a66-b90f-4c52ba5496f3
Note in the comment that the assembly-store payload is mapped read-only and
never modified, which is why it and every pointer derived from it are `const`.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 527f04b6-c95a-4a66-b90f-4c52ba5496f3
Accept the size changes from dlopen-based assembly store loading.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7be7d34c-aff3-4338-b15b-6061bc7a3ba9
Propagate UseAssemblyStore into the native application config and skip dlopen-based discovery when no store is packaged. This restores individual assembly loading from FastDev overrides.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7be7d34c-aff3-4338-b15b-6061bc7a3ba9
Update the assembly store and shared-library payload design docs to
describe the CoreCLR loading mechanism introduced in this PR and to
disambiguate it from the existing MonoVM behavior.

- AssemblyStores.md: add a per-runtime note explaining that MonoVM
  locates the store by scanning the APK ZIP and mmap-ing the wrapper,
  while CoreCLR resolves it via dlopen()+dlsym("_assembly_store").
- ApkSharedLibraries.md: mark the existing stub/mmap narrative and
  readelf sample as MonoVM-only, and add a detailed "Layout of the
  CoreCLR (dlopen) payload library" section with real llvm-readelf
  section-header, program-header and dynamic-symbol output showing the
  allocatable `payload` section, its PT_LOAD segment and the exported
  `_assembly_store` symbol.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b99a3be6-afab-4429-a40e-1a236ac0d98e
@simonrozsival simonrozsival marked this pull request as ready for review July 12, 2026 11:45
Copilot AI review requested due to automatic review settings July 12, 2026 11:45
@simonrozsival simonrozsival changed the title [CoreCLR] Experimental: dlopen()-based assembly store loading [CoreCLR] dlopen()-based assembly store loading Jul 12, 2026
@simonrozsival simonrozsival added the ready-to-review This PR is ready to review/merge, I think any CI failures are just flaky (ignorable). label Jul 12, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR switches the CoreCLR assembly store discovery/mapping path from a custom APK ZIP scan + mmap + ELF section-header walking to a simpler dlopen() + dlsym() flow by generating an assembly-store wrapper .so whose payload is loadable and exposed via the _assembly_store dynamic symbol. MonoVM remains on the existing ZIP-scan-based mechanism.

Changes:

  • CoreCLR native host now locates the assembly store via dlopen("libassembly-store.so") and dlsym("_assembly_store"), removing the ZIP central-directory parser and related fast paths.
  • Build tasks gained a CoreCLR-specific wrapper generator (llvm-mc + ld) and propagate a new UseDlopenAssemblyStore switch into packaging.
  • Application config and tests were updated to emit/verify a have_assembly_store flag; documentation updated to describe the two runtime-specific layouts.
Show a summary per file
File Description
src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets Adds _AndroidDlopenAssemblyStore and passes UseDlopenAssemblyStore into wrapping task.
src/Xamarin.Android.Build.Tasks/Utilities/DlopenAssemblyStoreGenerator.cs New generator producing a dlopen-able wrapper with loadable payload section + _assembly_store symbol.
src/Xamarin.Android.Build.Tasks/Utilities/ApplicationConfigNativeAssemblyGeneratorCLR.cs Removes CoreCLR DSO APK entry emission; emits have_assembly_store into application config.
src/Xamarin.Android.Build.Tasks/Utilities/ApplicationConfigCLR.cs Adds have_assembly_store field to managed representation of CoreCLR application config.
src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Resources/Base/BuildReleaseArm64SimpleDotNet.CoreCLR.apkdesc Updates expected APK file sizes for CoreCLR baseline resource.
src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Utilities/EnvironmentHelper.cs Updates CoreCLR application-config parsing to include have_assembly_store and adjusts field count.
src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Tasks/GenerateNativeApplicationConfigSourcesTests.cs New test verifying have_assembly_store is emitted correctly for CoreCLR.
src/Xamarin.Android.Build.Tasks/Tasks/WrapAssembliesAsSharedLibraries.cs Adds UseDlopenAssemblyStore task parameter and switches wrapper implementation for stores.
src/Xamarin.Android.Build.Tasks/Tasks/GenerateNativeApplicationConfigSources.cs Plumbs UseAssemblyStore into CoreCLR generator via HaveAssemblyStore.
src/native/CMakeLists.txt Stops building the removed CoreCLR startup/ZIP-scan subdirectory.
src/native/clr/xamarin-app-stub/application_dso_stub.cc Updates stub config for new have_assembly_store field; removes DSO APK entry stub.
src/native/clr/startup/zip.cc Deletes CoreCLR ZIP central-directory parser implementation.
src/native/clr/startup/CMakeLists.txt Deletes build definition for removed xamarin-startup static library.
src/native/clr/include/xamarin-app.hh Makes assembly-store pointers const; adds have_assembly_store; removes DSOApkEntry export.
src/native/clr/include/startup/zip.hh Deletes CoreCLR ZIP-scan API header.
src/native/clr/include/runtime-base/monodroid-dl.hh Removes CoreCLR embedded-DSO fd/offset fast path that depended on ZIP scan.
src/native/clr/include/host/host.hh Adds _assembly_store symbol constant and dlopen-mapping helper declaration.
src/native/clr/include/host/assembly-store.hh Replaces fd/offset mapping API with configure_from_payload() for dlopen/dlsym payload pointers.
src/native/clr/host/host.cc Implements dlopen/dlsym mapping and removes ZIP-scan path; gates store discovery on have_assembly_store.
src/native/clr/host/CMakeLists.txt Removes link dependency on deleted xa::xamarin-startup.
src/native/clr/host/assembly-store.cc Implements configure_from_payload() and switches store metadata pointers to const.
Documentation/project-docs/AssemblyStores.md Documents runtime-specific assembly store location/mapping behavior (MonoVM vs CoreCLR).
Documentation/project-docs/ApkSharedLibraries.md Adds CoreCLR dlopen-based payload library documentation and comparison details.

Copilot's findings

  • Files reviewed: 23/23 changed files
  • Comments generated: 2

{
var toolInfo = GetArchToolInfo (targetArch);

string outputDir = Path.Combine (config.BaseOutputDirectory, MonoAndroidHelper.ArchToRid (targetArch), "wrapped-dlopen");
Comment on lines 2243 to 2247
ResolvedAssemblies="@(_BuildApkAssembliesToAddToArchive)"
SupportedAbis="@(_BuildTargetAbis)"
UseAssemblyStore="$(_AndroidUseAssemblyStore)"
UseDlopenAssemblyStore="$(_AndroidDlopenAssemblyStore)"
RuntimePackLibraryDirectories="@(_RuntimePackLibraryDirectory)">
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area: CoreCLR Issues that only occur when using CoreCLR. ready-to-review This PR is ready to review/merge, I think any CI failures are just flaky (ignorable).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants