Skip to content

Add a THIRD_PARTY_NOTICES.md generator - #1973

Draft
abrarshivani wants to merge 2 commits into
NVIDIA:mainfrom
abrarshivani:third-party-notices
Draft

Add a THIRD_PARTY_NOTICES.md generator#1973
abrarshivani wants to merge 2 commits into
NVIDIA:mainfrom
abrarshivani:third-party-notices

Conversation

@abrarshivani

@abrarshivani abrarshivani commented Aug 7, 2026

Copy link
Copy Markdown

Adds a generator for THIRD_PARTY_NOTICES.md and the generated file itself, so
the Go dependencies of the released binaries carry attribution and the full text
of their licenses in-tree. make notices regenerates it and make notices-check
regenerates and diffs it, wired into CI so a dependency change cannot land
without refreshed attribution. This mirrors what has been proposed for
gpu-operator, mig-parted, k8s-device-plugin and k8s-driver-manager.

Note for reviewers

Almost the whole diff is machine-generated. Only the first ~660 lines are
hand-written:

File Lines Hand-written?
hack/generate-notices.sh 563 yes — the only thing worth reviewing
.github/workflows/notices-check.yaml 58 yes
Makefile 27 yes
hack/create-release.sh 6 yes
.github/workflows/ci.yaml 3 yes
.gitignore 2 yes
deployments/devel/tools.go 1 yes
deployments/devel/go.mod, go.sum 174 go get + go mod tidy output
THIRD_PARTY_NOTICES.md 7827 generated (95% of the diff)

How THIRD_PARTY_NOTICES.md is generated

  1. Resolve go-licenses — prefer the repo-local ./bin/go-licenses so the
    version pinned in deployments/devel/go.mod is the one that runs.
  2. Check the platform list against DOCKER_BUILD_PLATFORM_OPTIONS in
    deployments/container/multi-arch.mk and fail on drift, so a new released
    target cannot silently produce a short file.
  3. Runtime pass, once per released platform, GOFLAGS=-mod=vendor and
    CGO_ENABLED=1: go-licenses save copies the license files and
    go-licenses csv classifies them. Union the results — go-licenses resolves
    only the host platform and build-tagged sources differ per platform, so one
    run is both incomplete and host-dependent, and there is no union mode.
  4. Toolchain pass, once per platform, from deployments/devel/go.mod via the
    module cache. The tool list is read out of deployments/devel/tools.go so it
    cannot drift from make install-tools.
  5. Collapse to one row per package, joining every distinct license rather than
    picking one — go-licenses emits a row per recognised license, so
    filepath-securejoin's MPL-2.0 would otherwise hide behind its BSD-3-Clause.
  6. Attach module@version by longest-prefix match against vendor/modules.txt.
    In vendor mode go-licenses reports a link into this repository at HEAD,
    which stops describing released content once main advances.
  7. Compose header, index table per surface, then the verbatim texts. Write to a
    temp file and move into place.

Reproducibility is a hard requirement, since the CI gate is a git diff. Every
sort and every output-affecting grep runs under LC_ALL=C; rows are collapsed
after a whole-line sort -u rather than with a keyed sort -t, -k1,1 -u, which
is not portable across BSD and GNU sort; the awk that joins licenses uses a
counter rather than an in test, which mawk and busybox awk evaluate
differently from BSD awk and gawk; and the fence-width grep passes -a so a
license containing a NUL byte does not change the measured width by host.

Two smaller notes on the collection itself:

  • Only the local module is passed to --ignore. go-licenses already omits
    the standard library via its own GOROOT check, and --ignore matches raw
    string prefixes, not path segments — so passing stdlib top-level names would
    include the bare token go and silently drop golang.org/x/*,
    google.golang.org/*, gopkg.in/* and anything else starting with those two
    letters.
  • CGO stays enabled, because make cmds does not disable it and the
    binaries are linked with -extldflags=-Wl,--export-dynamic .... With
    CGO_ENABLED=0 the build constraints exclude every file in
    github.com/NVIDIA/go-nvml/pkg/dl and internal/cuda, and go-licenses
    cannot load ./cmd/... at all. No C compiler is needed either way —
    go-licenses only lists and parses, it never compiles.

Scope

The runtime scope is the import closure of ./cmd/..., which is exactly what
make cmds builds. Both release paths run it: docker/Dockerfile.* runs
make PREFIX=... cmds to populate the deb and rpm packages, and
deployments/container/Dockerfile builds nvidia-ctk-installer from source and
then unpacks those same packages into the released image. All seven commands
under cmd/ are therefore distributed. Measured against ./... the only extra
packages are four local ones and five stdlib ones, so this scope drops no
third-party dependency; tests/ and testdata/ are separate Go modules and
cannot reach this graph.

Platforms are linux/amd64 and linux/arm64, checked against multi-arch.mk.
The packages are built per architecture rather than as a multi-arch image, but
hack/prepare-artifacts.sh publishes deb amd64/arm64 and rpm x86_64/aarch64 and
nothing else, so the two surfaces agree. The ppc64le targets in
docker/docker.mk are buildable but are not part of a release, so they are not
in the matrix.

third_party/libnvidia-container is not inventoried here, and this needs your
call.
I want to be precise about what I found rather than wave it away:

  • It is not merely vendored source that goes unbuilt. scripts/build-packages.sh
    runs git submodule update --init and scripts/build-all-components.sh drives
    its mk/docker.mk, and .github/workflows/image.yaml calls that script for
    every target. The libnvidia-container1, libnvidia-container-tools and
    libnvidia-container-dev packages end up in the same release tarballs as the
    toolkit packages — I confirmed this by unpacking the published
    nvidia-container-toolkit_1.19.0_deb_amd64.tar.gz, which contains four
    libnvidia-container* debs alongside the three toolkit ones — and the
    debpackages/rpmpackages stages of deployments/container/Dockerfile extract
    every .deb/.rpm into the released image.
  • Its build fetches C dependencies that are vendored nowhere in this repository:
    elftoolchain 0.7.1 (libelf, BSD-2-Clause), libtirpc 1.3.2 (BSD-3-Clause) and
    nvidia-modprobe-utils 550.54.14 (MIT), all downloaded over the network at build
    time by mk/elftoolchain.mk, mk/libtirpc.mk and mk/nvidia-modprobe.mk.
  • It already ships its own compliance artifacts. NOTICE, LICENSE, COPYING
    and COPYING.LESSER are in its DOC_FILES and are installed into each of its
    packages, and its NOTICE states the LGPL terms that apply when it is
    dynamically linked against libelf from elfutils (WITH_LIBELF=yes; the default
    is no, which statically links elftoolchain's BSD-2-Clause libelf instead).

So the C surface ships in the same artifacts as the Go binaries, but it carries
its own notices and belongs to a separate upstream project with its own release
process. I have written the document's Scope section to say exactly that rather
than let the header imply coverage it does not have. If you would rather this
repository also inventory the libnvidia-container dependency set, that is a
larger change — the C dependencies are fetched at build time, so it would need
network access or a new vendoring step, and it would duplicate notices that
libnvidia-container already publishes. Happy to do it if that is the call.

The document also states that it does not cover the non-Go contents of the
released image — the distroless base and the static BusyBox layout.

How it is shipped

hack/create-release.sh now uploads THIRD_PARTY_NOTICES.md as a release asset
next to the package tarballs and the checksums file. Per OSRB guidance notices
may be distributed alongside the artifacts, and this keeps the package and image
contents unchanged. Note there is no prerelease guard: an RC is still a
distributed artifact and attaching the file costs nothing.

make notices-check runs on every build rather than behind a changed-paths
filter. The inventory is an import closure, so it goes stale when ordinary .go
files change their imports, not only when go.mod or vendor/ move — a filter
keyed on dependency manifests would pass green while the file rotted, and the
failure would surface on main instead of on the PR that caused it.

Testing

  • Cross-host reproducibility. Generated on macOS (BSD awk, Go 1.26.3) and in
    golang:1.26.5 (mawk 1.3.4, the /usr/bin/awk of the build image). Identical
    sha256: e655db0d731483040479aad8c332ffd0543d024adc1babd8856bf275d8373d95.
    Two consecutive runs on the same host are also byte-identical.
  • Completeness against independent ground truth. Derived the expected set
    separately with go list -deps over ./cmd/... for both platforms, filtered
    stdlib and the local module, mapped each of the resulting 138 third-party
    packages to its license-owning directory and reduced each group to the common
    path prefix that go-licenses names a library by. Expected 33, present 33,
    missing 0, unexpected 0.
  • Zero Unknown licenses, zero "License text unavailable", zero unresolved
    module@version. Every one of the 52 index rows has a matching license-text
    section.
  • make notices-check observed passing on the committed file, failing when the
    committed file is stale, and failing when the file is untracked.
  • Error paths exercised: missing input file, drifted platform matrix,
    unparseable platform source, unsafe LICENSES_DIR override, and an
    unreadable/empty vendor/modules.txt — each fails with a specific message and
    without writing a partial file.
  • shellcheck clean on hack/generate-notices.sh. actionlint reports nothing
    for the new and changed workflows (the linux-amd64-cpu4 unknown-runner-label
    warning it emits elsewhere is a pre-existing false positive). yamllint shows
    only the same document-start/truthy warnings the existing workflows have;
    there is no yamllint config in the repo.
  • make -C deployments/devel check-modules passes with go-licenses added.

hack/generate-notices.sh builds an attribution document for the Go
dependencies of this repository: an index of every third-party package
linked into the commands under cmd/, plus the verbatim text of each
license, and the same for the build tools pinned in deployments/devel.

The runtime scope is the import closure of ./cmd/..., which is exactly
what 'make cmds' builds for both release paths: docker/Dockerfile.* runs
it to populate the deb and rpm packages, and
deployments/container/Dockerfile builds nvidia-ctk-installer from source
and unpacks those packages into the released image.

go-licenses only resolves the host platform and build-tagged sources
differ per platform, so the script runs it once per released target and
merges the results. The platform list is checked against
DOCKER_BUILD_PLATFORM_OPTIONS in deployments/container/multi-arch.mk so a
new target cannot silently produce a short file. CGO stays enabled to
match 'make cmds': with CGO_ENABLED=0 the build constraints exclude every
file in github.com/NVIDIA/go-nvml/pkg/dl and internal/cuda, and
go-licenses cannot load ./cmd/... at all.

Only the local module is passed to --ignore. go-licenses already omits
the standard library, and --ignore matches raw string prefixes rather
than path segments, so listing stdlib top-level names would drop
golang.org/x/*, google.golang.org/*, gopkg.in/* and friends.

The output has to be byte-identical between a developer's laptop and CI
or the freshness check is worthless, so every sort and every
output-affecting grep runs under LC_ALL=C, rows are collapsed after a
whole-line sort rather than with a keyed 'sort -u', and the awk that
joins multiple licenses per package uses a counter instead of an "in"
test, which mawk and busybox awk evaluate differently from BSD awk and
gawk.

'make notices' regenerates the file and 'make notices-check' regenerates
and diffs it. The check runs on every build rather than behind a
changed-paths filter: the inventory is an import closure, so it goes
stale when ordinary .go files change their imports, not only when go.mod
or vendor/ move.

The generated file is attached to the GitHub release alongside the
package tarballs, which leaves the package and image contents unchanged.

Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
Output of 'make notices'. 33 Go runtime packages linked into the commands
under cmd/ and 19 build toolchain packages, with the full license text of
each.

The file is committed so that 'make notices-check' can gate freshness
with a plain git diff, and so the attribution for a given release is
readable straight from the tag.

Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant