Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixpkgs: Allow passing {host,build}Platform directly + Nixpkgs fn docs #324614

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/using-nixpkgs.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

```{=include=} chapters
using/platform-support.chapter.md
using/as-a-function.chapter.md
using/configuration.chapter.md
using/overlays.chapter.md
using/overrides.chapter.md
Expand Down
156 changes: 156 additions & 0 deletions doc/using/as-a-function.chapter.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
# Nixpkgs as a Function {#sec-nixpkgs-function}

Depending on how you use Nixpkgs, you may interact with it as a [Nix function].
The Nixpkgs function returns an attribute set containing mostly packages, but also other things, such as more package sets, and functions.
This return value is often referred to with the identifier `pkgs`, and it has a marker attribute, `_type = "pkgs";`.

The Nixpkgs function can be retrieved in multiple ways.

In a setup with [Nix channels], you may look up the Nixpkgs files with [`<nixpkgs>`][Nix lookup path]; for example

```console
$ nix repl -f '<nixpkgs>'
Added 21938 variables.
nix-repl>
```

<!-- -f: abbreviated form because it is very frequently used -->
[`nix repl`] [`-f`][`nix repl -f`] automatically imports the Nixpkgs function and invokes it for you.

Most Nix commands accept [`--arg`] and [`--argstr`] options to pass arguments to the Nixpkgs function.

In an expression, you can use the `import <nixpkgs> { }` syntax to import the Nixpkgs function, and immmediately [apply] it.

```nix
let
pkgs = import <nixpkgs> {
config = { allowUnfree = true; };
};
in
```

Note that these examples so far have relied on the implicit, [impure] [`builtins.currentSystem`] variable to configure the package set to be able to run on your system.

If you wish not to accidentally rely on the implicit [`builtins.currentSystem`] variable, and you'd like to avoid configuration in `~/.config/nixpkgs`, you may invoke `<nixpkgs/pkgs/top-level>` instead.

To make sure you can reproduce your evaluations, you may use [pinning] or locking, such as provided by [`npins`] or [Flakes].

## Arguments {#sec-nixpkgs-arguments}

For historical reasons, the platform and cross compilation parameters can be specified in multiple ways.

If you run Nix in pure mode, or if you work with expressions for multiple host platforms, you are recommended to use `hostPlatform` for native compilation (locally or on a remote machine), and both `hostPlatform` and `buildPlatform` for cross compilation.

### `hostPlatform` and `buildPlatform` {#sec-nixpkgs-arguments-platforms}

- `hostPlatform` is the platform for which the package set is built. This means that the binaries in the outputs are compatible with `hostPlatform`.

You may specify this as a cpu-os string, such as `"x86_64-linux"`, or `aarch64-darwin`, or you can pass a more details platform object produced with `lib.systems`.

- `buildPlatform` (default: `hostPlatform`) is the platform on which the derivations will run. The derivations produced by Nixpkgs will have a [`system` attribute] that matches `buildPlatform`, so that builds are sent to a machine that is capable of running the `builder` command, which is also configured to be compatible with `buildPlatform`.

Values are specified in the same formats as `hostPlatform`.

::: {.example #ex-nixpkgs-pure-native}

# Natively compiled packages

This shows how to explicitly request packages that are built for a specific platform, not relying on [`builtins.currentSystem`].
The packages will be built natively.

If you request to build this on a host that is not compatible, you need a remote builder.

```nix
let
pkgs = imports <nixpkgs> {
hostPlatform = "x86_64-linux";
};
in pkgs.hello
```

:::

::: {.example #ex-nixpkgs-pure-cross}

# Cross compiled packages

This expression will produce packages that are built on `x86_64-linux` machines, but whose outputs can run on `riscv64-linux`.

The expression is pure, so it will produce the same outcome even if run on e.g. an `aarch64-darwin` machine. For that to work, the macOS machine needs the packages to be available in a cache or it needs a remote builder that can build `x86_64-linux` derivations.

If you were to use [`builtins.currentSystem`] for `buildPlatform`, you would be able to build the packages directly on any machine, but the resulting store paths will be different, depending on the current system.
Especially if you are in a team that doesn't run a single CPU architecture and operating system or you use Nix on machines of more than one platform, you might notice that equivalent deployments would have different store paths, resulting in unnecessary rebuilds, more store path copying, and unnecessary updates to equivalent configurations with different `buildPlatform`s.
For these reason, it is recommended to use pure configurations where the platform parameters are set to fixed values for each deployment target.

```nix
let
pkgs = imports <nixpkgs> {
hostPlatform = "riscv64-linux";
buildPlatform = "x86_64-linux";
};
in pkgs.hello
```

:::

### Legacy arguments `system`, `localSystem` and `crossSystem` {#sec-nixpkgs-arguments-systems-legacy}

`crossSystem` and `system` (or `localSystem`) are ok to use on the command line, but are not recommended for pure Nix code (such as Nix flakes), or Nix code that manages the invocation of the Nixpkgs function.

These parameters existed before [`hostPlatform` and `buildPlatform`](#sec-nixpkgs-arguments-platforms) were introduced.
They are still supported, but are not recommended for new code.

The main difference is that these parameters default to [`builtins.currentSystem`], which does not work for pure code.

Furthermore, their design is optimized for impure command line use.

The term "local" is misleading, because it need not match the platform where the Nix command is run.

The term "cross" is also misleading because its value may equal `localSystem`.

Finally `crossSystem` may have to be unset as an input, complicating some code that calls the Nixpkgs function.
<!-- This example transgresses the guidelines a bit, but we have an audience here that needs answers, because who likes change. Without explanation, this comes across as a superficial, unnecessary and annoying change. Remove this example in 2026 when it is irrelevant. -->
For example, [`nixos-generate-config`] wasn't initially able to set the platform in `hardware-configuration.nix` without making assumptions about cross versus native compilation, resulting in a need for you to manually specify `system` when Flakes were introduced.

### `config` {#sec-nixpkgs-arguments-config}

Example:

```nix
import <nixpkgs> { config = { allowUnfree = true; }; }
```

See [Configuration](#sec-config-options-reference) for details.

### `overlays` {#sec-nixpkgs-arguments-overlays}

Overlays are expressions that modify the package set.

See [Overlays](#chap-overlays) for more information.

### `crossOverlays` {#sec-nixpkgs-arguments-crossOverlays}

<!-- Source: https://matthewbauer.us/slides/always-be-cross-compiling.pdf -->
Cross overlays apply only to the final package set in cross compilation.
This means that [`buildPackages`] and [`nativeBuildInputs`] are unaffected by these overlays and this increases the number of build dependencies that can be retrieved from the cache, or can be reused from a previous build.

[Nix function]: https://nix.dev/manual/nix/latest/language/constructs.html#functions
[Nix channels]: https://nix.dev/manual/nix/latest/command-ref/nix-channel.html
[Nix lookup path]: https://nix.dev/manual/nix/latest/language/constructs/lookup-path.html
[`builtins.currentSystem`]: https://nix.dev/manual/nix/latest/language/builtin-constants.html#builtins-currentSystem
[`system` attribute]: https://nix.dev/manual/nix/latest/language/derivations#attr-system
[`--arg`]: https://nix.dev/manual/nix/latest/command-ref/opt-common.html?highlight=--arg#opt-arg
[`--argstr`]: https://nix.dev/manual/nix/latest/command-ref/opt-common.html?highlight=--argstr#opt-argstr
[apply]: https://nix.dev/manual/nix/latest/language/operators
[impure]: https://nix.dev/manual/nix/latest/command-ref/conf-file.html?highlight=pure-eval#conf-pure-eval
[`npins`]: https://nix.dev/guides/recipes/dependency-management.html
[Flakes]: https://nix.dev/concepts/flakes.html#flakes
[pinning]: https://nix.dev/reference/pinning-nixpkgs.html
<!-- TODO: publish NixOS man pages -->
[`nixos-generate-config`]: https://nixos.org/manual/nixos/stable/#sec-installation
<!-- TODO: make more specific -->
[`buildPackages`]: #ssec-cross-dependency-implementation
<!-- TODO: make more specific -->
[`nativeBuildInputs`]: #ssec-stdenv-dependencies-propagated
[`nix repl`]: https://nix.dev/manual/nix/latest/command-ref/new-cli/nix3-repl.html
[`nix repl -f`]: https://nix.dev/manual/nix/latest/command-ref/new-cli/nix3-repl?highlight=--file#opt-file
174 changes: 174 additions & 0 deletions pkgs/test/top-level/default.nix
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# nix-build -A tests.top-level
{ lib, pkgs, ... }:
let
nixpkgsFun = import ../../top-level;
Expand Down Expand Up @@ -44,4 +45,177 @@ lib.recurseIntoAttrs {
assert lib.all (p: p.buildPlatform == p.hostPlatform) pkgsLocal;
assert lib.all (p: p.buildPlatform != p.hostPlatform) pkgsCross;
pkgs.emptyFile;

invocations-pure =
let
inherit (lib.systems) elaborate;
assertEqualPkgPlatforms = a: b:
# NOTE: equals should become obsolete, but currently isn't when comparing
# between two distinct Nixpkgs invocations
# `equals` should not be used in most places because it is a bit slow.
assert lib.systems.equals a.stdenv.hostPlatform b.stdenv.hostPlatform;
assert lib.systems.equals a.stdenv.buildPlatform b.stdenv.buildPlatform;
assert lib.systems.equals a.stdenv.targetPlatform b.stdenv.targetPlatform;
true;

native-s = import ../../.. { system = "i686-linux"; };
native-ls = import ../../.. { localSystem.system = "i686-linux"; };
native-le = import ../../.. { localSystem = lib.systems.elaborate "i686-linux"; };
native-h = import ../../.. { hostPlatform = "i686-linux"; };
native-he = import ../../.. { hostPlatform = lib.systems.elaborate "i686-linux"; };

cross-1-s-cs = import ../../.. { system = "i686-linux"; crossSystem = "armv7l-linux"; };
cross-1-s-css = import ../../.. { system = "i686-linux"; crossSystem.system = "armv7l-linux"; };
cross-1-s-cse = import ../../.. { system = "i686-linux"; crossSystem = elaborate "armv7l-linux"; };
cross-1-ls-cs = import ../../.. { localSystem.system = "i686-linux"; crossSystem = "armv7l-linux"; };
cross-1-ls-css = import ../../.. { localSystem.system = "i686-linux"; crossSystem.system = "armv7l-linux"; };
cross-1-ls-cse = import ../../.. { localSystem.system = "i686-linux"; crossSystem = elaborate "armv7l-linux"; };
cross-1-le-cs = import ../../.. { localSystem = elaborate "i686-linux"; crossSystem = "armv7l-linux"; };
cross-1-le-css = import ../../.. { localSystem = elaborate "i686-linux"; crossSystem.system = "armv7l-linux"; };
cross-1-le-cse = import ../../.. { localSystem = elaborate "i686-linux"; crossSystem = elaborate "armv7l-linux"; };
cross-1-b-h = import ../../.. { buildPlatform = "i686-linux"; hostPlatform = "armv7l-linux"; };
cross-1-b-hs = import ../../.. { buildPlatform = "i686-linux"; hostPlatform.system = "armv7l-linux"; };
cross-1-b-he = import ../../.. { buildPlatform = "i686-linux"; hostPlatform = elaborate "armv7l-linux"; };
cross-1-bs-h = import ../../.. { buildPlatform.system = "i686-linux"; hostPlatform = "armv7l-linux"; };
cross-1-bs-hs = import ../../.. { buildPlatform.system = "i686-linux"; hostPlatform.system = "armv7l-linux"; };
cross-1-bs-he = import ../../.. { buildPlatform.system = "i686-linux"; hostPlatform = elaborate "armv7l-linux"; };
cross-1-be-h = import ../../.. { buildPlatform = elaborate "i686-linux"; hostPlatform = "armv7l-linux"; };
cross-1-be-hs = import ../../.. { buildPlatform = elaborate "i686-linux"; hostPlatform.system = "armv7l-linux"; };
cross-1-be-he = import ../../.. { buildPlatform = elaborate "i686-linux"; hostPlatform = elaborate "armv7l-linux"; };

in

# NATIVE
# ------

assert lib.systems.equals native-s.stdenv.hostPlatform (elaborate "i686-linux");
assert lib.systems.equals native-s.stdenv.buildPlatform (elaborate "i686-linux");

# sample reflexivity (sanity check)
assert assertEqualPkgPlatforms native-s native-s;
# check the rest (assume transitivity to avoid n^2 checks)
assert assertEqualPkgPlatforms native-s native-ls;
assert assertEqualPkgPlatforms native-s native-le;
assert assertEqualPkgPlatforms native-s native-h;
assert assertEqualPkgPlatforms native-s native-he;

# These pairs must be identical
assert native-s.stdenv.hostPlatform == native-s.stdenv.buildPlatform;
assert native-ls.stdenv.hostPlatform == native-ls.stdenv.buildPlatform;
assert native-le.stdenv.hostPlatform == native-le.stdenv.buildPlatform;
assert native-h.stdenv.hostPlatform == native-h.stdenv.buildPlatform;
assert native-he.stdenv.hostPlatform == native-he.stdenv.buildPlatform;


# CROSS
# -----

assert lib.systems.equals cross-1-s-cs.stdenv.hostPlatform (elaborate "armv7l-linux");
assert lib.systems.equals cross-1-s-cs.stdenv.buildPlatform (elaborate "i686-linux");

# sample reflexivity (sanity check)
assert assertEqualPkgPlatforms cross-1-s-cs cross-1-s-cs;
# check the rest (assume transitivity to avoid n^2 checks)
assert assertEqualPkgPlatforms cross-1-s-cs cross-1-s-css;
assert assertEqualPkgPlatforms cross-1-s-cs cross-1-s-cse;
assert assertEqualPkgPlatforms cross-1-s-cs cross-1-ls-cs;
assert assertEqualPkgPlatforms cross-1-s-cs cross-1-ls-css;
assert assertEqualPkgPlatforms cross-1-s-cs cross-1-ls-cse;
assert assertEqualPkgPlatforms cross-1-s-cs cross-1-le-cs;
assert assertEqualPkgPlatforms cross-1-s-cs cross-1-le-css;
assert assertEqualPkgPlatforms cross-1-s-cs cross-1-le-cse;
assert assertEqualPkgPlatforms cross-1-s-cs cross-1-b-h;
assert assertEqualPkgPlatforms cross-1-s-cs cross-1-b-hs;
assert assertEqualPkgPlatforms cross-1-s-cs cross-1-b-he;
assert assertEqualPkgPlatforms cross-1-s-cs cross-1-bs-h;
assert assertEqualPkgPlatforms cross-1-s-cs cross-1-bs-hs;
assert assertEqualPkgPlatforms cross-1-s-cs cross-1-bs-he;
assert assertEqualPkgPlatforms cross-1-s-cs cross-1-be-h;
assert assertEqualPkgPlatforms cross-1-s-cs cross-1-be-hs;
assert assertEqualPkgPlatforms cross-1-s-cs cross-1-be-he;

assert lib.systems.equals cross-1-s-cs.stdenv.hostPlatform (elaborate "armv7l-linux");
assert lib.systems.equals cross-1-s-css.stdenv.hostPlatform (elaborate "armv7l-linux");
assert lib.systems.equals cross-1-s-cse.stdenv.hostPlatform (elaborate "armv7l-linux");
assert lib.systems.equals cross-1-ls-cs.stdenv.hostPlatform (elaborate "armv7l-linux");
assert lib.systems.equals cross-1-ls-css.stdenv.hostPlatform (elaborate "armv7l-linux");
assert lib.systems.equals cross-1-ls-cse.stdenv.hostPlatform (elaborate "armv7l-linux");
assert lib.systems.equals cross-1-le-cs.stdenv.hostPlatform (elaborate "armv7l-linux");
assert lib.systems.equals cross-1-le-css.stdenv.hostPlatform (elaborate "armv7l-linux");
assert lib.systems.equals cross-1-le-cse.stdenv.hostPlatform (elaborate "armv7l-linux");
assert lib.systems.equals cross-1-b-h.stdenv.hostPlatform (elaborate "armv7l-linux");
assert lib.systems.equals cross-1-b-hs.stdenv.hostPlatform (elaborate "armv7l-linux");
assert lib.systems.equals cross-1-b-he.stdenv.hostPlatform (elaborate "armv7l-linux");
assert lib.systems.equals cross-1-bs-h.stdenv.hostPlatform (elaborate "armv7l-linux");
assert lib.systems.equals cross-1-bs-hs.stdenv.hostPlatform (elaborate "armv7l-linux");
assert lib.systems.equals cross-1-bs-he.stdenv.hostPlatform (elaborate "armv7l-linux");
assert lib.systems.equals cross-1-be-h.stdenv.hostPlatform (elaborate "armv7l-linux");
assert lib.systems.equals cross-1-be-hs.stdenv.hostPlatform (elaborate "armv7l-linux");
assert lib.systems.equals cross-1-be-he.stdenv.hostPlatform (elaborate "armv7l-linux");

assert lib.systems.equals cross-1-s-cs.stdenv.buildPlatform (elaborate "i686-linux");
assert lib.systems.equals cross-1-s-css.stdenv.buildPlatform (elaborate "i686-linux");
assert lib.systems.equals cross-1-s-cse.stdenv.buildPlatform (elaborate "i686-linux");
assert lib.systems.equals cross-1-ls-cs.stdenv.buildPlatform (elaborate "i686-linux");
assert lib.systems.equals cross-1-ls-css.stdenv.buildPlatform (elaborate "i686-linux");
assert lib.systems.equals cross-1-ls-cse.stdenv.buildPlatform (elaborate "i686-linux");
assert lib.systems.equals cross-1-le-cs.stdenv.buildPlatform (elaborate "i686-linux");
assert lib.systems.equals cross-1-le-css.stdenv.buildPlatform (elaborate "i686-linux");
assert lib.systems.equals cross-1-le-cse.stdenv.buildPlatform (elaborate "i686-linux");
assert lib.systems.equals cross-1-b-h.stdenv.buildPlatform (elaborate "i686-linux");
assert lib.systems.equals cross-1-b-hs.stdenv.buildPlatform (elaborate "i686-linux");
assert lib.systems.equals cross-1-b-he.stdenv.buildPlatform (elaborate "i686-linux");
assert lib.systems.equals cross-1-bs-h.stdenv.buildPlatform (elaborate "i686-linux");
assert lib.systems.equals cross-1-bs-hs.stdenv.buildPlatform (elaborate "i686-linux");
assert lib.systems.equals cross-1-bs-he.stdenv.buildPlatform (elaborate "i686-linux");
assert lib.systems.equals cross-1-be-h.stdenv.buildPlatform (elaborate "i686-linux");
assert lib.systems.equals cross-1-be-hs.stdenv.buildPlatform (elaborate "i686-linux");
assert lib.systems.equals cross-1-be-he.stdenv.buildPlatform (elaborate "i686-linux");

# builtins.trace ''
# Tests passed, but note that the impure use cases are not included here.

# Impurely specified native compilation,
# currentSystem as build -> currentSystem as host:

# nix-build -A hello

# Impurely specified cross compilation for specified host,
# currentSystem as build -> crossSystem as host:

# nix-build --argstr crossSystem armv7l-linux -A hello

# ''



pkgs.emptyFile;

invocations-impure =
if !builtins?currentSystem
then
lib.trace
"Skipping invocations-impure test, because we don't have currentSystem here. Don't forget to also run the top-level tests in impure mode!"
pkgs.emptyFile
else
let
# As a mere expression, we don't get to pick currentSystem, so we have to
# pick a different one dynamically.
otherSystem = if builtins.currentSystem == "x86_64-linux" then "aarch64-linux" else "x86_64-linux";
otherSystemElaborated = lib.systems.elaborate otherSystem;
currentSystemElaborated = lib.systems.elaborate builtins.currentSystem;

impureNative = import ../../.. { };
impureCross = import ../../.. { crossSystem = otherSystem; };
in
# Obviously
assert ! lib.systems.equals currentSystemElaborated otherSystemElaborated;

assert lib.systems.equals impureNative.stdenv.hostPlatform currentSystemElaborated;
assert lib.systems.equals impureNative.stdenv.buildPlatform currentSystemElaborated;

assert lib.systems.equals impureCross.stdenv.hostPlatform otherSystemElaborated;
assert lib.systems.equals impureCross.stdenv.buildPlatform currentSystemElaborated;

pkgs.emptyFile;
}
Loading
Loading