Enable stdenv customization through (buildPython*.override { inherit stdenv; })

[ShamrockLee]

Description of changes

This PR brings the missing .override back to buildPythonPackage and buildPythonApplication, and allow specifying the stdenv used by the build helpers through buildPython*.override (as most build helpers do) instead of passing it into buildPython*.

The reason to move to buildPythonPackage.override { stdenv = ...; } from buildPythonPackage { stdenv = ...; } includes:

1. The buildPython* { inherit stdenv; ...; } will be impossible/unfeasible when taking fixed-point arguments (buildPython* (finalAttrs: { ...; })). lib.extendMkDerivation is on its way, but we need a clean ground to integrate it into buildPython*.

2. stdenv is a dependency buildPythonPackage would (directly or indirectly) get from pkgs through callPackage. It makes better sense to change it through the .override attribute provided by callPackage.

3. In the future, we could get rid of all the special, impassable arguments, and pass everything properly either into stdenv.mkDerivation or through passthru. This way, we only need overrideAttrs to override attributes, without having to defined special overriders like overridePythonAttrs and for each language and framework. stdenv could neither be passed into stdenv.mkDerivation or through passthru, since stdenv.mkDerivaiton will add a stdenv attribute to the resulting derivation.

For package definition that used to be:

{
  lib,
  stdenv,
  buildPythonPackage,
  fetchPyPi,
}:
buildPythonPackage rec {
  inherit stdenv;
  pname = "...";
  version = "...";
  src = fetchPypi {
    inherit pname source;
    hash = "...";
  };
}

can now become

{
  lib,
  stdenv,
  buildPythonPackage,
  fetchPyPi,
}:
buildPythonPackage.override { inherit stdenv; } rec {
  pname = "...";
  version = "...";
  src = fetchPypi {
    inherit pname source;
    hash = "...";
  };
}

In the future, when buildPythonPackage also takes fixed-point arguments, we could write:

{
  lib,
  stdenv,
  buildPythonPackage,
  fetchPyPi,
}:
buildPythonPackage.override { inherit stdenv; } (finalAttrs: {
  pname = "...";
  version = "...";
  src = fetchPypi {
    inherit (finalAttrs) pname source;
    hash = "...";
  };
})

This PR depends on #366593.
Initial effort for #271387.

Things done

• Built on platform(s)
  • x86_64-linux (no rebuilds other than nixos-install-tools)
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

Priorities

Add a 👍 reaction to pull requests you find important.

[ShamrockLee]

Simplify a bit.

[ShamrockLee]

Reword the release note entry.