Add a THIRD_PARTY_NOTICES.md generator - #1973
Draft
abrarshivani wants to merge 2 commits into
Draft
Conversation
hack/generate-notices.sh builds an attribution document for the Go dependencies of this repository: an index of every third-party package linked into the commands under cmd/, plus the verbatim text of each license, and the same for the build tools pinned in deployments/devel. The runtime scope is the import closure of ./cmd/..., which is exactly what 'make cmds' builds for both release paths: docker/Dockerfile.* runs it to populate the deb and rpm packages, and deployments/container/Dockerfile builds nvidia-ctk-installer from source and unpacks those packages into the released image. go-licenses only resolves the host platform and build-tagged sources differ per platform, so the script runs it once per released target and merges the results. The platform list is checked against DOCKER_BUILD_PLATFORM_OPTIONS in deployments/container/multi-arch.mk so a new target cannot silently produce a short file. CGO stays enabled to match 'make cmds': with CGO_ENABLED=0 the build constraints exclude every file in github.com/NVIDIA/go-nvml/pkg/dl and internal/cuda, and go-licenses cannot load ./cmd/... at all. Only the local module is passed to --ignore. go-licenses already omits the standard library, and --ignore matches raw string prefixes rather than path segments, so listing stdlib top-level names would drop golang.org/x/*, google.golang.org/*, gopkg.in/* and friends. The output has to be byte-identical between a developer's laptop and CI or the freshness check is worthless, so every sort and every output-affecting grep runs under LC_ALL=C, rows are collapsed after a whole-line sort rather than with a keyed 'sort -u', and the awk that joins multiple licenses per package uses a counter instead of an "in" test, which mawk and busybox awk evaluate differently from BSD awk and gawk. 'make notices' regenerates the file and 'make notices-check' regenerates and diffs it. The check runs on every build rather than behind a changed-paths filter: the inventory is an import closure, so it goes stale when ordinary .go files change their imports, not only when go.mod or vendor/ move. The generated file is attached to the GitHub release alongside the package tarballs, which leaves the package and image contents unchanged. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
Output of 'make notices'. 33 Go runtime packages linked into the commands under cmd/ and 19 build toolchain packages, with the full license text of each. The file is committed so that 'make notices-check' can gate freshness with a plain git diff, and so the attribution for a given release is readable straight from the tag. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a generator for
THIRD_PARTY_NOTICES.mdand the generated file itself, sothe Go dependencies of the released binaries carry attribution and the full text
of their licenses in-tree.
make noticesregenerates it andmake notices-checkregenerates and diffs it, wired into CI so a dependency change cannot land
without refreshed attribution. This mirrors what has been proposed for
gpu-operator, mig-parted, k8s-device-plugin and k8s-driver-manager.
Note for reviewers
Almost the whole diff is machine-generated. Only the first ~660 lines are
hand-written:
hack/generate-notices.sh.github/workflows/notices-check.yamlMakefilehack/create-release.sh.github/workflows/ci.yaml.gitignoredeployments/devel/tools.godeployments/devel/go.mod,go.sumgo get+go mod tidyoutputTHIRD_PARTY_NOTICES.mdHow THIRD_PARTY_NOTICES.md is generated
go-licenses— prefer the repo-local./bin/go-licensesso theversion pinned in
deployments/devel/go.modis the one that runs.DOCKER_BUILD_PLATFORM_OPTIONSindeployments/container/multi-arch.mkand fail on drift, so a new releasedtarget cannot silently produce a short file.
GOFLAGS=-mod=vendorandCGO_ENABLED=1:go-licenses savecopies the license files andgo-licenses csvclassifies them. Union the results —go-licensesresolvesonly the host platform and build-tagged sources differ per platform, so one
run is both incomplete and host-dependent, and there is no union mode.
deployments/devel/go.modvia themodule cache. The tool list is read out of
deployments/devel/tools.goso itcannot drift from
make install-tools.picking one —
go-licensesemits a row per recognised license, sofilepath-securejoin's MPL-2.0 would otherwise hide behind its BSD-3-Clause.module@versionby longest-prefix match againstvendor/modules.txt.In vendor mode
go-licensesreports a link into this repository atHEAD,which stops describing released content once
mainadvances.temp file and move into place.
Reproducibility is a hard requirement, since the CI gate is a
git diff. Everysort and every output-affecting grep runs under
LC_ALL=C; rows are collapsedafter a whole-line
sort -urather than with a keyedsort -t, -k1,1 -u, whichis not portable across BSD and GNU sort; the awk that joins licenses uses a
counter rather than an
intest, which mawk and busybox awk evaluatedifferently from BSD awk and gawk; and the fence-width grep passes
-aso alicense containing a NUL byte does not change the measured width by host.
Two smaller notes on the collection itself:
--ignore.go-licensesalready omitsthe standard library via its own GOROOT check, and
--ignorematches rawstring prefixes, not path segments — so passing stdlib top-level names would
include the bare token
goand silently dropgolang.org/x/*,google.golang.org/*,gopkg.in/*and anything else starting with those twoletters.
make cmdsdoes not disable it and thebinaries are linked with
-extldflags=-Wl,--export-dynamic .... WithCGO_ENABLED=0the build constraints exclude every file ingithub.com/NVIDIA/go-nvml/pkg/dlandinternal/cuda, andgo-licensescannot load
./cmd/...at all. No C compiler is needed either way —go-licensesonly lists and parses, it never compiles.Scope
The runtime scope is the import closure of
./cmd/..., which is exactly whatmake cmdsbuilds. Both release paths run it:docker/Dockerfile.*runsmake PREFIX=... cmdsto populate the deb and rpm packages, anddeployments/container/Dockerfilebuildsnvidia-ctk-installerfrom source andthen unpacks those same packages into the released image. All seven commands
under
cmd/are therefore distributed. Measured against./...the only extrapackages are four local ones and five stdlib ones, so this scope drops no
third-party dependency;
tests/andtestdata/are separate Go modules andcannot reach this graph.
Platforms are
linux/amd64andlinux/arm64, checked againstmulti-arch.mk.The packages are built per architecture rather than as a multi-arch image, but
hack/prepare-artifacts.shpublishes deb amd64/arm64 and rpm x86_64/aarch64 andnothing else, so the two surfaces agree. The
ppc64letargets indocker/docker.mkare buildable but are not part of a release, so they are notin the matrix.
third_party/libnvidia-containeris not inventoried here, and this needs yourcall. I want to be precise about what I found rather than wave it away:
scripts/build-packages.shruns
git submodule update --initandscripts/build-all-components.shdrivesits
mk/docker.mk, and.github/workflows/image.yamlcalls that script forevery target. The
libnvidia-container1,libnvidia-container-toolsandlibnvidia-container-devpackages end up in the same release tarballs as thetoolkit packages — I confirmed this by unpacking the published
nvidia-container-toolkit_1.19.0_deb_amd64.tar.gz, which contains fourlibnvidia-container*debs alongside the three toolkit ones — and thedebpackages/rpmpackagesstages ofdeployments/container/Dockerfileextractevery
.deb/.rpminto the released image.elftoolchain 0.7.1 (libelf, BSD-2-Clause), libtirpc 1.3.2 (BSD-3-Clause) and
nvidia-modprobe-utils 550.54.14 (MIT), all downloaded over the network at build
time by
mk/elftoolchain.mk,mk/libtirpc.mkandmk/nvidia-modprobe.mk.NOTICE,LICENSE,COPYINGand
COPYING.LESSERare in itsDOC_FILESand are installed into each of itspackages, and its
NOTICEstates the LGPL terms that apply when it isdynamically linked against libelf from elfutils (
WITH_LIBELF=yes; the defaultis
no, which statically links elftoolchain's BSD-2-Clause libelf instead).So the C surface ships in the same artifacts as the Go binaries, but it carries
its own notices and belongs to a separate upstream project with its own release
process. I have written the document's Scope section to say exactly that rather
than let the header imply coverage it does not have. If you would rather this
repository also inventory the libnvidia-container dependency set, that is a
larger change — the C dependencies are fetched at build time, so it would need
network access or a new vendoring step, and it would duplicate notices that
libnvidia-container already publishes. Happy to do it if that is the call.
The document also states that it does not cover the non-Go contents of the
released image — the distroless base and the static BusyBox layout.
How it is shipped
hack/create-release.shnow uploadsTHIRD_PARTY_NOTICES.mdas a release assetnext to the package tarballs and the checksums file. Per OSRB guidance notices
may be distributed alongside the artifacts, and this keeps the package and image
contents unchanged. Note there is no prerelease guard: an RC is still a
distributed artifact and attaching the file costs nothing.
make notices-checkruns on every build rather than behind a changed-pathsfilter. The inventory is an import closure, so it goes stale when ordinary
.gofiles change their imports, not only when
go.modorvendor/move — a filterkeyed on dependency manifests would pass green while the file rotted, and the
failure would surface on
maininstead of on the PR that caused it.Testing
golang:1.26.5(mawk 1.3.4, the/usr/bin/awkof the build image). Identicalsha256:
e655db0d731483040479aad8c332ffd0543d024adc1babd8856bf275d8373d95.Two consecutive runs on the same host are also byte-identical.
separately with
go list -depsover./cmd/...for both platforms, filteredstdlib and the local module, mapped each of the resulting 138 third-party
packages to its license-owning directory and reduced each group to the common
path prefix that
go-licensesnames a library by. Expected 33, present 33,missing 0, unexpected 0.
Unknownlicenses, zero "License text unavailable", zero unresolvedmodule@version. Every one of the 52 index rows has a matching license-textsection.
make notices-checkobserved passing on the committed file, failing when thecommitted file is stale, and failing when the file is untracked.
unparseable platform source, unsafe
LICENSES_DIRoverride, and anunreadable/empty
vendor/modules.txt— each fails with a specific message andwithout writing a partial file.
shellcheckclean onhack/generate-notices.sh.actionlintreports nothingfor the new and changed workflows (the
linux-amd64-cpu4unknown-runner-labelwarning it emits elsewhere is a pre-existing false positive).
yamllintshowsonly the same
document-start/truthywarnings the existing workflows have;there is no yamllint config in the repo.
make -C deployments/devel check-modulespasses withgo-licensesadded.