A CLI tool to generate reports of the differences in build outcomes between pairs of Nixpkgs commits.
For example, the following report was generated by running this command:
npb --pr 539280That command may not give you the same result if you run it yourself, but npb's report includes a reproducible command that will:
Expand this for a reproducible command.
npb --base 26426a9ad1c5cebe35b163a94152065a7b9d1413 --head 9acc6884ece717d2d20a485da38fb2fc39d575a0 --patch 9acc6884ece717d2d20a485da38fb2fc39d575a0...17a97c09e0d9467dd191a2cfbc800c2a984b5f1b -s aarch64-darwinExpand this for a legend of all symbols below.
- ✅ = successfully built
- ❌ = failed to build
- 🚫 = dependency failed to build
- ❔ = couldn't try to build
- ⏩ = failed to evaluate
- ➖ = doesn't exist
🚫 → 🚫 · 1 package
lixStatic✅ → ✅ · 76 packages (43 unique)
lixPackageSets.git.colmenalixPackageSets.git.lixlixPackageSets.git.nillixPackageSets.git.nix-direnvlixPackageSets.git.nix-dulixPackageSets.git.nix-eval-jobslixPackageSets.git.nix-fast-buildlixPackageSets.git.nix-initlixPackageSets.git.nix-updatelixPackageSets.git.nixos-anywherelixPackageSets.git.nixos-rebuild-ng.tests.repl=lixPackageSets.latest.nixos-rebuild-ng.tests.repl=lixPackageSets.stable.nixos-rebuild-ng.tests.repl=lixPackageSets.lix_2_94.nixos-rebuild-ng.tests.repl=lixPackageSets.lix_2_95.nixos-rebuild-ng.tests.repllixPackageSets.git.nixos-rebuild-nglixPackageSets.git.nixpkgs-reviewFulllixPackageSets.git.nixpkgs-reviewlixPackageSets.git.nurllixPackageSets.latest.colmena=lixPackageSets.stable.colmena=lixPackageSets.lix_2_95.colmenalixPackageSets.latest.nil=lixPackageSets.stable.nil=lixPackageSets.lix_2_95.nillixPackageSets.latest.nix-direnv=lixPackageSets.stable.nix-direnv=lixPackageSets.lix_2_95.nix-direnvlixPackageSets.latest.nix-du=lixPackageSets.stable.nix-du=lixPackageSets.lix_2_95.nix-dulixPackageSets.latest.nix-eval-jobs=lixPackageSets.stable.nix-eval-jobs=lixPackageSets.lix_2_95.nix-eval-jobslixPackageSets.latest.nix-fast-build=lixPackageSets.stable.nix-fast-build=lixPackageSets.lix_2_95.nix-fast-buildlixPackageSets.latest.nix-init=lixPackageSets.stable.nix-init=lixPackageSets.lix_2_95.nix-initlixPackageSets.latest.nix-update=lixPackageSets.stable.nix-update=lixPackageSets.lix_2_95.nix-updatelixPackageSets.latest.nixos-anywhere=lixPackageSets.stable.nixos-anywhere=lixPackageSets.lix_2_95.nixos-anywherelixPackageSets.latest.nixos-rebuild-ng=lixPackageSets.stable.nixos-rebuild-ng=lixPackageSets.lix_2_95.nixos-rebuild-nglixPackageSets.latest.nixpkgs-reviewFull=lixPackageSets.stable.nixpkgs-reviewFull=lixPackageSets.lix_2_95.nixpkgs-reviewFulllixPackageSets.latest.nixpkgs-review=lixPackageSets.stable.nixpkgs-review=lixPackageSets.lix_2_95.nixpkgs-reviewlixPackageSets.latest.nurl=lixPackageSets.stable.nurl=lixPackageSets.lix_2_95.nurllixPackageSets.lix_2_94.colmenalixPackageSets.lix_2_94.lixlixPackageSets.lix_2_94.nillixPackageSets.lix_2_94.nix-direnvlixPackageSets.lix_2_94.nix-dulixPackageSets.lix_2_94.nix-eval-jobslixPackageSets.lix_2_94.nix-fast-buildlixPackageSets.lix_2_94.nix-initlixPackageSets.lix_2_94.nix-updatelixPackageSets.lix_2_94.nixos-anywherelixPackageSets.lix_2_94.nixos-rebuild-nglixPackageSets.lix_2_94.nixpkgs-reviewFulllixPackageSets.lix_2_94.nixpkgs-reviewlixPackageSets.lix_2_94.nurllix=lixPackageSets.latest.lix=lixPackageSets.stable.lix=lixPackageSets.lix_2_95.lix❌ → ✅ · 14 packages (4 unique)
lixPackageSets.git.nix-serve-nglixPackageSets.lix_2_94.nix-serve-ngnix-serve-ng.tests.lix=lixPackageSets.latest.nix-serve-ng=lixPackageSets.stable.nix-serve-ng=lixPackageSets.lix_2_95.nix-serve-ng=haskellPackages.nix-serve-ng.tests.lix=lixPackageSets.git.nix-serve-ng.tests.lix=lixPackageSets.latest.nix-serve-ng.tests.lix=lixPackageSets.stable.nix-serve-ng.tests.lix=lixPackageSets.lix_2_94.nix-serve-ng.tests.lix=lixPackageSets.lix_2_95.nix-serve-ng.tests.lixnix-serve-ng=haskellPackages.nix-serve-ng❔ → ❔ · 6 packages (3 unique)
lix.tests.misc=lixPackageSets.latest.lix.tests.misc=lixPackageSets.stable.lix.tests.misc=lixPackageSets.lix_2_95.lix.tests.misclixPackageSets.git.lix.tests.misclixPackageSets.lix_2_94.lix.tests.misc
npb strives to be
-
causal. Derivations are built on both sides of the change, so the report makes it clear which failures are regressions and which were preexisting.
-
thorough. By default, the tool finds and runs tests associated with the affected packages, and includes those test outcomes in the report.
-
cached. It takes a nontrivial amount of time to run Nix evaluations and failing builds, so the tool caches these; re-runs should almost always take less than one second.
-
efficient. Caching is not an excuse to be slow the first time, so on a cold run, the tool automatically chooses a point along the Pareto frontier to parallelize according to how much RAM is available.
-
reproducible. Often a change may be on a branch that gets force-pushed later, or there may be uncommitted local changes, so the report always includes a command to reproduce the exact same Git tree on both sides.
-
friendly. Users running the tool in a rich terminal should get nice colors and live progress indicators, and users running it over a pipe should get lean but informative progress lines.
npb does not
- fetch remote evaluations.
- manage Git worktrees.
- provide sandboxing.
- post to GitHub.
This repository provides a Nix flake, so if you have flakes enabled, the quickest way to use it is via the nix shell command:
nix shell github:samestep/npbThis starts a shell within your shell which has npb available on the PATH.
To list available CLI arguments:
npb --helpBy default, npb assumes the current working directory is a clone of the Nixpkgs repo. If your Nixpkgs clone is somewhere else:
npb -C ~/github/NixOS/nixpkgsNote that, just like Git's -C, this argument affects how other paths are interpreted, specifically for --patch.
By default, npb compares against master. To compare against a different base:
npb --base origin/release-26.05By default, npb evaluates on the current working tree. To evaluate on a different revision:
npb --head origin/stagingTo set --base and --head from the current state of an open pull request:
npb --pr 543008Note that this is an exception to npb's usual rule that a re-run should be fully cached. If you run npb with a --pr argument, and then that PR is updated, and then you run the same npb command again, it will fetch the current state of the PR every time.
To apply a diff on top of the --head, provide a path to a patch file. For instance, one trick that npb sometimes uses to provide reproducible commands is --patch /dev/stdin with a heredoc:
npb --patch /dev/stdin <<'PATCH'
--- a/pkgs/by-name/he/hello/package.nix
+++ b/pkgs/by-name/he/hello/package.nix
@@ -66,3 +66,3 @@ stdenv.mkDerivation (finalAttrs: {
mainProgram = "hello";
- platforms = lib.platforms.all;
+ platforms = [ ]; # bye bye
identifiers.cpeParts.vendor = "gnu";
PATCHNote that the path must include a forward-slash. That is to avoid ambiguity because --patch also accepts three-dot GitHub commit comparison expressions:
npb --patch HEAD...974b2de7dc8f59ba4ac9729c452834641f82849fBy default, npb merges the --base and the --head together, then evaluates --base versus that merged result. To instead take the merge-base of --base and --head, and evaluate that versus --head:
npb --no-mergeBy default, npb includes all passthru.tests for packages that changed between the two commits. To not include those tests:
npb --no-testsBy default, npb sets allowUnsupportedSystem to false. To set it to true instead:
npb --allow-unsupportedBy default, npb sets allowBroken to false. To set it to true instead:
npb --allow-brokenBy default, npb sets allowInsecurePredicate to a constant false. To set it to a constant true instead:
npb --allow-insecureBy default, npb caches failures and does not try to rebuild derivations that it has seen fail before. To retry failed builds anyway:
npb --retryBy default, npb only evaluates for the system of the machine it is running on. To override that and instead build for one or more other systems:
npb -s x86_64-linux -s aarch64-linux -s aarch64-darwinWhenever npb encounters a Git tree it hasn't seen before, it evaluates Nixpkgs on that tree and saves the result to disk, roughly 3 megabytes per tree per --system. To free up disk space by deleting the least recently used evaluation results:
npb --clean 100MBThe value passed to this argument must be either a YYYY-MM-DD date or an integer with a suffix, which determines its meaning as follows.
- Delete entries not used for at least some amount of time:
s/sec= secondsm/min= minutesh/hr= hoursd= daysw= weeksmo= monthsy/yr= years
- Delete least recently used entry until cache is small enough:
B= byteskB= kilobytesKiB= kibibytesMB= megabytesMiB= mebibytesGB= gigabytesGiB= gibibytesTB= terabytesTiB= tebibytes
To print the version:
npb --versionThis project owes its existence to its primary inspiration, Jörg Thalheim's nixpkgs-review, which serves a similar purpose but makes different design tradeoffs.
The development of this project was AI-assisted. There is no AI-generated text in this README, npb --help, or npb's generated reports.
npb is licensed under the MIT License.