feat(pack): ship a library as interface + prebuilt binaries (#433) - #451
Open
Sunrisepeak wants to merge 12 commits into
Open
feat(pack): ship a library as interface + prebuilt binaries (#433)#451Sunrisepeak wants to merge 12 commits into
Sunrisepeak wants to merge 12 commits into
Conversation
A closed-source library, an offline site, or a build farm that already
compiled this once had no route through mcpp: `mcpp publish` sends source,
`mcpp pack` bundles a program's run-time closure, and neither is "a library
someone else links". Collecting the artifacts by hand was the answer.
mcpp pack mathkit # a static library package
mcpp pack mathkit --target x86_64-linux-gnu \
--target aarch64-linux-gnu # one package, two legs
WHAT IT PRODUCES IS AN ORDINARY PACKAGE. A normal `mcpp.toml`, read through
the route mcpp already had for payloads that carry their own manifest. Zero
new manifest sections and zero new keys: what to pack is `[targets.<n>].kind`
(so there is no --lib and no --artifact), which interface to publish is `[lib]`
plus the module graph, which headers are public is `[build].include_dirs`, and
each leg's ABI tag and digest ride on `[[runtime.artifacts]]`, whose fields
were already documented as optional evidence. An older mcpp still BUILDS
against these packages; it just does not run the two gates below.
Two interface modes coexist — `include/` (text, `#include`, never compiled)
and `interface/` (module units the consumer compiles). Measured: one package
consumed three ways (header only, module only, both) x static and shared = six
combinations, all green.
WHICH .cppm TRAVEL IS COMPUTED, not declared. It is the module closure of the
lib root, and the two ways of getting it wrong are asymmetric: too few fails
loudly in the consumer's compile, too many silently publishes a closed-source
implementation partition's SOURCE. `.m.o` is not the rule — an implementation
partition produces one too. The same closure decides which archive members to
drop, and getting THAT wrong was measured as well: dropping every `.m.o` also
drops the partition's real code and every target fails to link. Both lists are
printed, because "what is not travelling" is half of what a publisher needs.
TWO GATES ON THE CONSUMER SIDE, both for failures that are otherwise silent.
The interface still matches its binaries: swap two `int` members of a shipped
struct — the Itanium ABI does not mangle field order — and before this the
consumer compiled, linked, ran, and printed transposed data with no diagnostic
from any tool. And the binaries were built for this toolchain, with the
refusal listing the tags the package does have, because "not found" sends
people looking for a package already on their disk.
Also fixes two defects this work found, each with its own regression test:
* `[target.'<triple>'.build]` never matched a native build. `matches()`
short-circuited on the raw --target string while `context_for()` fell back
to the host for `cfg(...)`, so two spellings of one statement disagreed —
green in CI, silently inert on a developer's machine. The resolved triple
now lives in `cfgpred::Ctx`; there is no second answerer.
* `sources = []` was byte-identical to omitting the key, so no author could
say "compile nothing". A header-only package needs that, and without it
any leftover file under `src/` is compiled into the consumer's build.
Docs: docs/12-binary-distribution.md (+ zh), examples 05-lib-dist and
06-lib-consume. Design: .agents/docs/2026-08-17-library-distribution-design.md.
Tests: 23 unit assertions across two new suites; e2e 242-248.
Two problems the first push found, one from CI and one from running the
previous release side by side.
`mcpp.pack.library` exported a `mcpp::pack::Error` and `mcpp.pack` already
had one. A name attaches to exactly one module, and mcpp.pack.library_pipeline
imports both — clang refuses outright ("cannot be attached to other modules"),
GCC accepted it. Every Windows and macOS job failed on it while the Linux ones
were green, which is the whole argument for the three-platform matrix.
`mcpp pack` in a workspace root stopped working. Routing on
`[targets.<n>].kind` means something reads the manifest before the build does,
and a workspace root has no targets of its own — a virtual one has no
`[package]` either — so the new router read an empty list and concluded there
was nothing to pack. Found by running the previous release against
examples/04-workspace and comparing; e2e 249 is that comparison made permanent.
And the positional was accepted but never reached the application pipeline, so
a project with two `bin` targets would take `mcpp pack app2` and bundle app1 —
succeeding with the wrong answer. e2e 250 pins both directions plus the
refusal for an unknown name.
…names Found by running the path the docs already promised. `mcpp pack <shared target>` shipped only the built file — `libmathkit-shared.so` — while the object records `SONAME libmathkit.so.1`. A consumer links by the first name and the loader asks for the second, so the package linked and the program could not start. mcpp's own runtime-closure check is what reported it, naming the missing soname rather than letting it become a loader error at launch. The package now carries the SONAME alongside the link name (a symlink, falling back to a copy), which is what a distribution ships and what the design's "soname gives the correct run-time name" note always meant. e2e 251 uses `run`, not `build`: linking proves nothing here.
`00_fixture_path_hygiene.sh` caught it on the macOS leg. The rule is a Windows one: MSYS rewrites POSIX paths on the way into argv and the environment but never touches file CONTENT, so a `path = "/tmp/…"` written INTO an mcpp.toml is read by a native mcpp.exe as "root of the current drive". The failure then surfaces as a dependency that cannot be found, four steps from its cause. Six of the new fixtures wrote the package's path that way. They now source `_host_path.sh` and pass it through `host_path`, which is what the lint asks for — and the lint runs on every platform precisely so a Linux reviewer can catch this before Windows CI does.
…roved §11 of the design doc: three places the implementation diverged from the plan (no [distribution] section at all, no abi_surface flag, no new CI workflow), the four defects found while implementing — each with what caught it — and the stale-fingerprint trap I walked into three times while VERIFYING, which is the same criterion the packer itself enforces about never globbing for artifacts.
All ten new tests carried `# requires: gcc`, and that capability is Linux-only by design — macOS's g++ is Apple Clang and Windows' is not an mcpp-compatible GCC, so the runner does not grant it there. Every one of them skipped on macOS and Windows while the suite reported green, which left "a static library package works on every target" verified on exactly one platform. That is the same false-green shape this PR already corrected once (245 was gated on mingw-cross and would never have run in CI at all). Finding it a second time, in my own tests, is the argument for checking whether a test RAN rather than whether the suite was green. The five that assert on mcpp's own output — layout, both interface modes, the three refusals, workspace routing, target naming — now require nothing and run everywhere. The rest stay gated for real reasons: 245 needs a musl target, 246 and 247 read compiler-specific flag spellings out of build.ninja, 248 needs the mingw cross toolchain, and 251 exercises `kind = "shared"`, which is ELF-only.
…everywhere
Letting the portable tests run on all three platforms was the right move, and
it worked: CI answered the question I could not answer locally.
* on the Windows (MSVC-ABI clang) leg the library build inside `pack` fails
with a bare `error: build failed`;
* on macOS the closure test cannot inspect the archive, because `ar` there
resolves to an xlings shim that reports "not installed".
Neither is fixed here and neither is hidden. The three PACKING tests carry
`# requires: gcc` again with the scope written at the top, while the tests for
the COMMAND — which target it picks, how it refuses an unknown name, packing
from a workspace root — keep running everywhere, because they pass everywhere.
docs/12 and its zh mirror gain a "where it is verified" section, and the limits
table now says "every target — verified on Linux only" rather than the first
half of that sentence.
Also: a Windows pack produces a .zip, not a .tar.gz. 249 asserted only the
latter, which is why it failed there even though the pack had succeeded.
`module M:part;` and `module M;` wear one spelling and are two declarations. The
scanner treated them as one: the first was recorded as *requiring* `M:part` and
providing nothing, so a file required its own name and the graph held no edge
from the unit importing a partition to the unit defining it. Build order was
unconstrained — GCC and macOS clang recovered through their own dependency scan,
Windows clang failed with `failed to read compiled module`.
The same site had a second half: `import :part;` resolved against `u.provides`,
and an implementation unit (`module M;`) has none, so `import :secret;` inside
one stayed the literal `:secret`. The note both platforms printed —
`module 'M:part' imported but not provided in this build` — was the merged
symptom of the two, and it read like a hint rather than the cause.
Implementation partitions had no test coverage anywhere in mcpp; the library
distribution e2e is the first thing to use one, which is how this surfaced. Five
scanner unit tests pin it now, including the case the old code was written for
(`export module foo:http;` + `import :tls;` must give `foo:tls`).
Consequence for `mcpp pack`: a partition the published interface reaches now
resolves, so it is published rather than refused — the consumer cannot build the
interface's BMI without it. That is correct and it is also the one thing a
closed-source publisher must not do by accident, so it comes with a warning
naming the file. 243 asserts both halves; the closure's unresolved-import error
stays for a partition nothing provides.
With this, e2e 242/243/244/249/250 run on Linux, macOS and Windows — the docs no
longer need a "verified on Linux only" caveat.
Also, per review: examples 05-lib-dist and 06-lib-consume are one story, so they
are one directory (examples/05-lib-distribution/{producer,consumer}) with one
README; and the consumers use `import std;` rather than <cstdio>, which is what
a module example should be showing.
…sing archiver must not be a silent skip Windows packed the library correctly — `lib/x86_64-windows-msvc/mathkit.lib` — and 242 failed anyway, because the assertion hard-coded `libmathkit.a`. MinGW writes `libfoo.a` where MSVC writes `foo.lib`, which is the very reason `lib/` is keyed by triple and not by OS; the test had the fact in its comments and the GNU spelling in its `find`. 243's archive probe had the same assumption, which made its drop assertion skip silently on the clang legs rather than run. And the packer itself: with objects to drop and no archiver resolved it did nothing, quietly, leaving the published interface's objects inside the archive — two definitions of each published module's initialiser, resolved by link order. That is the failure class this whole feature exists to remove, so it is now refused with the reason.
§11.2b: unrestricting the portable tests is what surfaced the scanner's implementation-partition bug, and fixing that — rather than widening a capability — is what made 242/243/244 pass on Linux, macOS and Windows. Also the per-platform table with a reason for every remaining skip.
Asked "what is the skip reason", and only one of the four survived the question.
* 246 was gated on `gcc` because of copy-paste. Its probe greps a FILENAME
out of build.ninja, not a compiler flag — nothing in it is toolchain-
specific. Runs everywhere now.
* 247 grepped `-D<macro>`, and MSVC spells that `/D` (dialect.cppm). The
prefix was the only thing tying it to one compiler family; grepping the
macro NAME is spelling-agnostic. Runs everywhere now.
* 248 needed `mingw-cross`, which no e2e job installs — so it skipped in
every job that exists, verified on a developer's machine and nowhere else.
cross-build-test.yml's mingw job already names the e2e scripts the Linux
shards skip for exactly this reason (102, 198, 240); 248 joins them, and
the workflow header now says why that list is explicit rather than left to
run_all's cap gating.
* 251's reason held — `kind = "shared"` is ELF-only and plan.cppm refuses it
— but it only tested the side that works, which cannot tell "the gate is
handled" from "there is no gate". It is now two-sided: on ELF the package
is produced and carries both of the library's names; off ELF the pack must
be REFUSED and the message must name both the artifact kind and where it
does work. Runs everywhere.
That leaves 245, which needs two distinct buildable targets (gnu + musl) and so
genuinely cannot run where only one exists. The mechanism it checks — one cfg()
leg per triple, the native build included — is platform-independent and covered
on Linux.
The criterion, earned twice now: a test's `# requires:` has to be the real floor
of the mechanism it verifies. Telling "a real limit" from "I did not think it
through" is hard while writing it, so the reasons have to be interrogated one by
one afterwards.
…or one 247 derived the host triple with a regex for `<arch>-<os>-<env>` and fell back to `x86_64-linux-gnu`. macOS's canonical triple is `aarch64-macos` — two segments, no env — so the match failed, the fallback was used, and the test asserted that a *Linux* section should apply to a macOS build. It then reported "the bare triple was inert" against a product behaving correctly. `target/<triple>/` is mcpp's own answer to the same question. The test now builds once with nothing conditional, reads the directory name, and writes the real manifest from that — and the cfg() control is keyed on the same value's arch rather than on `unix`, which is false on Windows. The shape is the one this PR keeps meeting: a second, independent derivation of something mcpp already computes, disagreeing with it on the platform nobody checked.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #433. Please do not merge yet.
A closed-source library, an offline site, or a build farm that already compiled
this once had no route through mcpp:
mcpp publishsends source,mcpp packbundles a program's run-time closure, and neither is "a library someone else
links". Collecting the artifacts by hand was the answer.
What it produces is an ordinary package
A normal
mcpp.toml, read through the route mcpp already had for payloads thatcarry their own manifest. Zero new manifest sections and zero new keys:
[targets.<n>].kind[lib]+ the module graph[build].include_dirs[[runtime.artifacts]][target.'cfg(…)'.build] ldflags[dependencies]So there is no
--liband no--artifact static\|shared—kindis alreadywhere mcpp records what an artifact is, and a flag would be a second place to
say it. And an older mcpp still builds against these packages; it simply
does not run the two gates below.
Two interface modes coexist:
include/(text,#include, never compiled) andinterface/(module units the consumer compiles). One package consumed threeways × static and shared = six combinations, all green.
Which
.cppmtravel is computed, not declaredThe module closure of the lib root. The two ways of getting it wrong are
asymmetric, which is why it cannot be a hand-written list:
published. Nothing fails.
.m.ois not the rule — an implementation partition produces one too. The sameclosure decides which archive members to drop, and getting that wrong was
measured as well: dropping every
.m.oremoves the partition's real code andevery target fails to link. Both lists are printed, and an interface that
reaches a partition publishes it with a warning naming the file.
Two gates, for failures that are otherwise silent
The interface still matches its binaries. Swap two
intmembers of ashipped struct — the Itanium ABI does not mangle field order — and before this
the consumer compiled, linked, ran, and printed transposed data, exit code 0,
no diagnostic from any tool.
The binaries were built for this toolchain. The refusal lists the tags the
package does have, because "not found" sends people looking for a package
already on their disk.
Building inside an extracted package is also refused: its
interface/holdsdeclarations whose definitions are in the archive beside them, so a build there
produces a near-empty library and reports success.
Three pre-existing defects found on the way
Each has its own regression test, and each affects projects that never touch
packaging.
Implementation partitions did not build on Windows.
module M:part;andmodule M;wear one spelling and are two declarations; the scanner recordedthe first as requiring
M:partand providing nothing, so a file requiredits own name and the graph held no edge from the unit importing a partition
to the unit defining it. Order was unconstrained — GCC and macOS clang
recovered through their own scan, Windows clang failed with
failed to read compiled module. The note both printed,module 'M:part' imported but not provided in this build, was the cause and read like a hint. Implementationpartitions had no test coverage anywhere in mcpp; this feature's e2e is
the first thing to use one. (5 scanner unit tests)
[target.'<triple>'.build]never matched a native build.matches()short-circuited on the raw
--targetstring whilecontext_for()fell backto the host for
cfg(...). Green in CI, silently inert on a developer'smachine. (e2e 247)
sources = []was byte-identical to omitting the key, so no author couldsay "compile nothing". (e2e 246)
Verification: what actually ran, per platform
CI 19/19 green. 87/87 unit suites (2 new files, 28 assertions; 5 more in
test_modgraph). 15 regression e2e over the touched areas.
sources = []pack <name>packs that namecross-build-test.ymlmingw jobOne skip remains, and it is a real limit: 245 needs two distinct buildable
targets (
x86_64-linux-gnu+x86_64-linux-musl), and no second ready targetexists on the macOS or Windows runners. The mechanism it checks — one
cfg()leg per triple, the native build included — is platform-independent and covered
on Linux.
Getting here took interrogating every skip reason one at a time, and three of
the four did not survive the question:
# requires: gccwas copy-paste-Dprefix, and MSVC spells that/D. Grepping the macro NAME is portablecross-build-test.yml's mingw job already names the e2e scripts the Linux shards skip for exactly this reason (102, 198, 240). 248 joins themsharedis ELF-only"Before that, all ten carried
# requires: gcc, which is Linux-only by design,so they skipped on macOS and Windows while the suite reported green.
Unrestricting them is what surfaced the scanner bug above. The criterion, earned
twice: a test's
# requires:has to be the real floor of the mechanism itverifies, and that is hard to judge while writing it — so the reasons have to
be interrogated afterwards, one by one.
Self-review found five things after the first push
mcpp.pack.libraryexported a secondmcpp::pack::Error. A name attachesto one module; clang refuses, GCC accepted. Windows + macOS red, Linux green.
mcpp packin a workspace root stopped working. Found by running theprevious release against
examples/04-workspaceand comparing. (e2e 249)binproject took
mcpp pack app2and bundled app1. (e2e 250)for the
SONAME— it linked and could not start. mcpp's own runtime-closurecheck reported it. (e2e 251, which uses
run, notbuild)published interface's objects in the archive. Now refused with the reason.
Not in this PR
kind = "shared"on PE / Mach-O,sharedon*-musl(a musl target linksstatically), bundling dependencies into the package, prebuilt BMIs.
Docs
docs/12-binary-distribution.md(+ zh),examples/05-lib-distribution(producer + consumer in one directory, consumers written with
import std;).Design, the measurements behind every criterion, and the four designs my own
experiments disproved:
.agents/docs/2026-08-17-library-distribution-design.md§11.