Skip to content

experiment: compile hyperion for bare metal (Hermit unikernel, virtio-net) — DRAFT, do not merge - #959

Draft
andrewgazelka wants to merge 4 commits into
mainfrom
experiment/bare-metal
Draft

experiment: compile hyperion for bare metal (Hermit unikernel, virtio-net) — DRAFT, do not merge#959
andrewgazelka wants to merge 4 commits into
mainfrom
experiment/bare-metal

Conversation

@andrewgazelka

Copy link
Copy Markdown
Member

Experimental. Do not merge. Opened as a draft so the findings are reviewable; the code is a probe, not a feature.

What works

nix build .#bare-metal produces an x86_64-unknown-hermit unikernel image. nix run .#bare-metal-vm boots it under QEMU, where it brings up virtio-net, takes a DHCP lease, and 13 ms after boot answers a Minecraft server-list ping decoded with the same valence_protocol the real server uses. The identical source also builds and runs as an ordinary Linux/macOS binary.

[    0.302772][0][INFO  network   ] DHCP config acquired!
[    0.303104][0][INFO  network   ] IP address:   192.168.76.9/24
[    0.310927][0][INFO  hermit    ] Jumping into application
[hyperion] platform: unikernel
[hyperion] capabilities: Capabilities { persistent_storage: false, unix_sockets: false, dns: false, trustworthy_wall_clock: false, adjustable_file_limit: false, subprocesses: false }
[hyperion] parallelism: 2
[hyperion] listening on 0.0.0.0:25565 after 13.166ms
[hyperion] accepted Ok(192.168.76.2:53071)
[hyperion] handshake: protocol=763 host=127.0.0.1 port=25599 next=Status
[hyperion] sent status
[hyperion] ponged 0xdeadbeefcafef00d

Two survey results made that possible and are the most transferable findings here: valence_protocol cross-compiles to Hermit unmodified, and so does bevy_ecs.

What does not work

The game server does not build. 11 of 24 workspace crates compile for the target; 13 do not, hyperion and bedwars among them. Four dependencies account for nearly all of it — socket2, openssl-sys (pulled in by reqwest's default features despite rustls-tls being requested), libz-ng-sys, and ring — and behind those sit heed/LMDB, memmap2, libdeflater and ndarray+blas, which a machine with no filesystem cannot host at all.

There is no virtio-mem anywhere in Rust guest land. Hermit has no such driver (src/drivers/ is console, fs, net, vsock); virtio-drivers has no driver, no issue, and no PR. Ballooning is an unmerged PR from June. If elastic memory is a requirement, that is Firecracker with a Linux guest, not a Rust unikernel.

Tokio on Hermit means two unmaintained forks (hermit-os/socket2, hermit-os/tokio pinned to 1.45.0, last commit 2025-05-08). mio and polling have genuine upstream support; tokio does not.

The trap worth knowing before anyone else tries this

Hyperion's pinned nightly-2025-02-22 builds a Hermit image that boots, prints, and then fails every socket call with bind: Kind(Uncategorized). That nightly's std is built against hermit-abi 0.4; kernel 0.13.2 answers 0.5. Boot and println! happen to line up across the gap; sockets do not. The identical source on nightly-2026-07-01 binds first try. bare-metal/ therefore carries its own toolchain pin.

Separately: the repo's .cargo/config.toml puts -Ctarget-cpu=native in [build] rustflags, which applies to every target, so any cross-compile dies inside core with rustc-LLVM ERROR: 64-bit code requested on a subtarget that doesn't support it!. Setting RUSTFLAGS overrides it, which is why that file is untouched here.

The durable part: a seam, not a fork

crates/hyperion-platform names the five things a hosted OS and a unikernel actually disagree about — the open-file limit, the wall clock, durable storage, AF_UNIX, and vCPU count — plus a CAPABILITIES constant so a call site asks "is there a filesystem?" rather than "am I on Unix?". The backend is one cfg arm; a third platform is a new backend/*.rs and a row in a table.

Hermit ships a real std, so this is deliberately not a portability layer. Almost nothing needs wrapping, and a wider seam would be a second standard library to maintain.

Blast radius

No existing crate was modified. Everything is new files plus two additive edits:

File Change
Cargo.toml two lines adding crates/hyperion-platform as a member
flake.nix one input, one callPackage, two packages entries, one apps entry
flake.lock the new input
Cargo.lock 7 lines for the new member

Wiring the seam into hyperion was deliberately skipped: it would buy nothing today (hyperion is blocked on four C libraries, not on cfg(unix)) and would put a diff into files other streams are editing. The seven call sites it is for are enumerated in the doc.

What it would take to close the gap

The realistic next milestone is not "the server boots". It is the proxy: it needs only the socket2 and tokio forks and has no filesystem dependency worth the name. Getting hyperion itself to boot means replacing persistence (LMDB), chunk storage (mmap'd Anvil regions), and the compression backend — that is a redesign, not a port.

Cheap wins that are worth doing regardless of this branch:

  • bvh-region has proptest in [dependencies], not [dev-dependencies]. It ships a test framework into release builds on Linux too.
  • reqwest is requested with rustls-tls but keeps default features, so openssl-sys comes along as well.

Not verified

  • nix build .#default fails on this branch — and already failed on unmodified 313503c, reproduced in a detached worktree (A hash was specified for divan-0.1.17, but there is no corresponding git dependency). Pre-existing, but it means the existing nix path could not serve as a regression check.
  • No full cargo build --workspace was run. "The normal build is unchanged" rests on the diff, not on a measurement.
  • Everything was built and booted on aarch64-darwin cross to x86_64-unknown-hermit, QEMU without KVM. Linux untested; aarch64/riscv64 Hermit untested.
  • The second half of the blocker table (heed, memmap2, libdeflater, ndarray+blas, tracing-tracy, valence_anvil) is reasoned from manifests. No build reached them.
  • No login, no gameplay, one connection at a time, compression off, nothing under load.
  • The Hermit socket2/tokio forks are named as the route to the proxy but were not built here.

Full write-up, including the prior-art comparison against Motor OS, Unikraft, virtio-drivers+smoltcp, Nanos/OSv and Firecracker with versions and dates: docs/bare-metal.md.

Names the five things a hosted OS and a unikernel disagree about -- the
open-file limit, the wall clock, durable storage, AF_UNIX, and vCPU count --
and puts one cfg arm behind them. Hosted is the default and behaves exactly
as hyperion did before, so a normal build sees no change.
A separate cargo workspace, so the Hermit git dependency and its nightly
never enter the main lockfile. The same source builds as an ordinary Linux
binary and as an x86_64-unknown-hermit machine image; on the latter it boots
under QEMU, brings up virtio-net, takes a DHCP lease and serves the
handshake/status/ping exchange with valence_protocol.
One vendor FOD covers all three cargo runs -- application, -Z build-std of
std, and the nested kernel build -- because the kernel's build script strips
every CARGO_* and RUST_* variable and can only be reached through
$HOME/.cargo/config.toml.
Leads with what runs and what does not, and ends with an explicit list of
what was never verified, including that nix build .#default was already
broken on main before this branch.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant