pythonPackages: Switch to lib.makeScope for `overrideScope` support #67422

infinisil · 2019-08-25T03:14:40Z

Motivation for this change

This change allows previously ~final package sets to be overridden with
overlays , e.g. you can do pkgs.python.pkgs.overrideScope' (self: super: { ... })
whereas previously only pkgs.python.override { packageOverrides = (self: super: { ... }) was possible.

This therefore also allows packages definitions to transitively override
dependencies. So e.g. if jupyterlab_server requires jsonschema >= 3, but
the rest of the packages should still use jsonschema == 2, you can
define it as such:

jupyterlab_server = (self.overrideScope' (self: super: {
  jsonschema = self.jsonschema3;
})).callPackage ../development/python-modules/jupyterlab_server { };

This applies the overlay to the whole dependency closure, whereas

jupyterlab_server = callPackage ../development/python-modules/jupyterlab_server {
  jsonschema = self.jsonschema3;
};

would only make it apply to jupyterlab_server itself, but not its
dependencies, resulting in build-time errors. To get the desired effect without
this change, this would be needed for this specific case:

jupyterlab_server = callPackage ../development/python-modules/jupyterlab_server {
  jsonschema = self.jsonschema3;
  notebook = self.notebook.override {
    nbformat = self.nbformat.override {
      jsonschema = self.jsonschema3;
    };
    nbconvert = self.nbconvert.override {
      nbformat = self.nbformat.override {
        jsonschema = self.jsonschema3;
      };
    };
  };
};

Ping @FRidh @JonathanReeve @jonringer @veprbl

Things done

Tested that all python override strategies still work

veprbl · 2019-08-25T03:36:30Z

This provides convenient and consistent dependency overrides. However, the main problem seems to be that there will be a conflict between python packages if the jsonschema and jupyterlab (which brings jsonschema3) will appear in the same closure.

infinisil · 2019-08-25T04:36:36Z

@veprbl Yeah that is a problem (not a main one), and this PR doesn't solve this (if this can be solved at all).

The biggest reason I made this PR is to make it much more convenient to do transitive overrides, which is possible for a lot of other package sets already because of their use of makeScope.

jonringer · 2019-08-25T18:53:11Z

What about the case where I need two packages, one needs jsonschema==2.* and the other needs jsonschema>=3. Won't the buildEnv still create a PYTHONPATH that has both versions on it?

Example of a shell.nix:

...
buildInputs = [
  jupyter_server # needs jsonschema_3
  jsonschema_2
];
...

infinisil · 2019-08-25T21:14:47Z

@jonringer Yeah I think that would create such a conflicting env because of how buildInputs treats python packages, but if you do this it should work:

{
  buildInputs = [
    jupyterlab_server
    (python3.withPackages (p: [ p.jsonschema ]))
  ];
}

FRidh · 2019-08-26T05:59:40Z

What is the point of doing such an override if you cannot put multiple packages in an env anyway?

…

On Sun, Aug 25, 2019, 23:14 Silvan Mosberger ***@***.***> wrote: @jonringer <https://github.com/jonringer> Yeah I think that would create such a conflicting env because of how buildInputs treats python packages, but if you do this it should work: { buildInputs = [ jupyterlab_server (python3.withPackages (p: [ p.jsonschema ])) ]; } — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#67422?email_source=notifications&email_token=AAQHZ32COESAP6BJ3GZGVCLQGLY47A5CNFSM4IPH2SOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5C33EI#issuecomment-524664209>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAQHZ346GCJYZLQJTYNKKQ3QGLY47ANCNFSM4IPH2SOA> .

infinisil · 2019-08-26T12:41:41Z

@FRidh For jupyterlab to build at all. See the PR description for what would be needed to make it build without this change.

infinisil · 2019-08-26T12:52:36Z

I guess this is mostly useful for getting end-user applications to build correctly

jonringer · 2019-08-26T15:54:24Z

@infinisil in applications, there's a lot more freedom to change python versions, because there's a good guarantee that that the python packages will be wrapped and isolated from other python packages. A common example is the aws-sam-cli package.
However, I think your scope example is a lot more concise than the aws-sam-cli/override example :) 👍

FRidh · 2019-08-27T05:44:08Z

With notebook and jupyterlab we should separate the kernel from the application, that is, you should simply be able to build it as an app and then add a withPackages kernel. I recall this actually already being done.

…

On Mon, Aug 26, 2019, 17:54 Jon ***@***.***> wrote: @infinisil <https://github.com/Infinisil> in applications, there's a lot more freedom to change python versions, because there's a good guarantee that that the python packages will be wrapped and isolated from other python packages. A common example is the aws-sam-cli package <https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/tools/aws-sam-cli/default.nix> . However, I think your scope example is a lot more concise than the aws-sam-cli/override example :) 👍 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#67422?email_source=notifications&email_token=AAQHZ3ZNF3BQHRSJPV3I7IDQGP4DVA5CNFSM4IPH2SOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5EZPQY#issuecomment-524916675>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAQHZ3YEO4B3KCOLPGAIDE3QGP4DVANCNFSM4IPH2SOA> .

infinisil · 2019-08-31T16:07:02Z

Hm I'm not sure I'm getting what you're saying @FRidh, but maybe it's that if I define jupyterlab in top-level/all-packages.nix, I can apply an overlay to the python package set before using callPackage, therefore achieving the same deep override as shown above, but without having to change anything in the python package set. This would mean pkgs.pythonPackages.jupyterlab would be removed in favor of pkgs.jupyterlab with custom python overrides. Is that what you mean?

I still feel like this functionality could be useful though, and it's already built into the makeScope function which is commonly used for package sets. It essentially allows adding overlays to already constructed package sets, which can make it more ergonomic in some cases, e.g. when you only receive a python package set in a function but need to add an overlay to it and you don't know where it came from.

FRidh · 2019-08-31T16:09:31Z

This would mean pkgs.pythonPackages.jupyterlab would be removed in favor of pkgs.jupyterlab with custom python overrides. Is that what you mean?

Correct.

infinisil · 2019-08-31T16:22:29Z

@FRidh Alright, then let's use that to get jupyterlab to build. You onboard with merging this just for the sake of uniformity of package sets in nixpkgs?

FRidh · 2019-08-31T16:44:25Z

In general I am of the opinion that we need an RFC first to detail how the various package sets should look. I do not see any benefit feature-wise of this PR, although uniformity is absolutely desirable. It cannot cause any regressions either so that's good. However, the use case you had in mind cannot be dealt with in this way as discussed. By merging this change I expect more PR's with overrideScope inside the Python package set. That is something I would prefer to avoid.

Given that there's a lot of other ways to do bad things that's not a good enough reason, so I'd say go ahead with it. Just update the commit message to change the self.overrideScope' to e.g. a python3Packages.overrideScope'. We should also document that it should not be used within the Python package set.

This change allows previously ~final package sets to be overridden with overlays , e.g. you can do `pkgs.python.pkgs.overrideScope' (self: super: { ... })` whereas previously only `pkgs.python.override { packageOverrides = (self: super: { ... })` was possible. This therefore also allows packages definitions to transitively override dependencies. So e.g. if jupyterlab_server requires jsonschema >= 3, but the rest of the packages should still use jsonschema == 2, you can define it as such: jupyterlab_server = (python.pkgs.overrideScope' (self: super: { jsonschema = self.jsonschema3; })).callPackage ../development/python-modules/jupyterlab_server { }; This applies the overlay to the whole dependency closure, whereas jupyterlab_server = python.pkgs.callPackage ../development/python-modules/jupyterlab_server { jsonschema = self.jsonschema3; }; would only make it apply to jupyterlab_server itself, but not its dependencies, resulting in build-time errors. To get the desired effect without this change, this would be needed for this specific case: jupyterlab_server = python.pkgs.callPackage ../development/python-modules/jupyterlab_server { jsonschema = self.jsonschema3; notebook = self.notebook.override { nbformat = self.nbformat.override { jsonschema = self.jsonschema3; }; nbconvert = self.nbconvert.override { nbformat = self.nbformat.override { jsonschema = self.jsonschema3; }; }; }; };

infinisil · 2019-09-05T00:27:09Z

I updated the commit message to change self -> python.pkgs (where applicable).

I wouldn't really know where to put docs for this however, and given that this functionality might be needed in some cases (maybe even in the python package set), I wouldn't want to tell people to not use it. Unless you insist on docs I'd like this to be merged so the jupyterlab update can be merged too.

FRidh · 2019-09-06T12:43:22Z

The section How to override a Python package? in the Nixpkgs manual could be extended with an example using this and a warning it should not be used from within the Python packages set.

bb010g · 2019-10-23T02:28:34Z

Is this just waiting on documentation?

bb010g · 2019-10-27T21:25:29Z

Also, for fun (wanting to experiment with mirroring Nixpkgs's structure in a NUR repo), you can actually do this on today's Nixpkgs with a bit of backing horror:

bb010g/bb010g.nix@2ba91b8

Uncommented, abridged, and slightly modified code taken from 2ba91b8:pkgs/default.nix#L47-87 using #67423 as an example:

let
  # pkgs = self;
  pkgs' = let
    inherit (pkgs) lib;
    pkgsScope = lib.makeScope pkgs.newScope (_: pkgs);
    pythonScopeOverrides = self: super: {
      pythonInterpreters = super.pythonInterpreters.override (o: {
        pkgs = o.pkgs // {
          callPackage = fn: args: o.pkgs.callPackage fn (let
            packageOverrides = pySelf: pySuper:
              pySuper.__extends__ or
                (let pyScope = lib.makeScope self.newScope (self: pySuper // {
                  inherit (self) callPackage;
                }); in if args ? packageOverrides then
                  pyScope.overrideScope' args.packageOverrides
                  else
                    pyScope);

              f = if lib.isFunction fn then fn else import fn;
              in if (lib.functionArgs f) ? packageOverrides then
                args // { inherit packageOverrides; }
              else
                args);
        };
      });
      inherit (self.pythonInterpreters) python37;
      python37Packages = self.python37.pkgs;
      python3 = self.python37;
      python3Packages = self.python3.pkgs;
    };
  in pkgsScope.overrideScope' pythonScopeOverrides;
in { inherit (pkgs'.python3Packages.overrideScope' (self: super: {
  jsonschema = self.jsonschema3;
  jupyterlab = self.callPackage ../applications/editors/jupyterlab { };
  jupyterlab_server = self.callPackage ../servers/jupyterlab { };
})) jupyterlab jupyterlab_server; }

infinisil · 2019-11-02T14:22:29Z

Neat and hacky!

Yeah I think the only thing missing here is docs wanted by @FRidh, but I'm frightened by xml. Imo leaving out docs for this isn't too bad, because nobody would even know they could now use .overrideScope' without looking at the source code, so no need to point out to people to not use it for python-packages.nix either. We can add docs later still.

FRidh · 2019-11-02T14:28:06Z

If you don't want to point it out, why add it?

infinisil · 2019-11-02T14:42:14Z

It's still useful when you need it (such as for jupyterlab and probably more cases as well), even without docs. Docs would be nice yeah, but I don't even use python or jupyterlab, so I don't really have much motivation for this.

infinisil · 2019-12-12T22:39:03Z

If anybody else feels like doing this, feel free to use the patch in this PR

jonringer · 2019-12-12T23:43:20Z

Agreed, the azure-cli python package list would have been a good candidate

nixpkgs/pkgs/tools/admin/azure-cli/python-packages.nix

Line 42 in cf5c943

py = python.override {

mikepurvis · 2021-07-16T19:01:55Z

Am bummed that this never merged; it would have been a nice solution for composable Python overrides.

bb010g · 2021-10-25T21:40:41Z

Am bummed that this never merged; it would have been a nice solution for composable Python overrides.

@mikepurvis pythonPackages now uses lib.makeScopeWithSplicing, a variant of lib.makeScope that supports splicing properly. Override via .overrideScope. I currently use the following Nixpkgs overlay to overlay all Python interpreters at once via pkgsFinal: pkgsPrev: { pythonInterpreters = pkgsPrev.pythonInterpreters.overrideScope (pyFinal: pyPrev: …); }:

let
  inherit (builtins) hasAttr listToAttrs map;

  flow = f: g: x: g (f x);

  mapAttr = name: f: set: set // { ${name} = f set.${name}; };
  mapAttrOr = name: f: nul: set: if hasAttr name set then mapAttr name f set else nul;
  mapAttrOrElse = name: f: nulF: set: mapAttrOr name f (nulF name set) set;

  isOverridable = hasAttr "override";
  isScope = hasAttr "overrideScope";
in
pkgsFinal: pkgsPrev: let
  inherit (pkgsFinal.lib) composeExtensions makeOverridable;
  attachScope = f: genFinal:
    let final = genFinal final // {
      overrideScope = g: attachScope (composeExtensions f g) genFinal;
      overrideScopeGenFinal = g: attachScope f (g genFinal);
      scopeGenFinal = genFinal;
      scopeOverrides = f;
    }; in final;
  attachEmptyScope = attachScope (final: prev: { });
in {
  pythonInterpreters = let
    pyInterpsPrev = pkgsPrev.pythonInterpreters;
  in if isScope pyInterpsPrev then pyInterpsPrev else attachEmptyScope (final:
    pyInterpsPrev.override (mapAttr "pkgs" (pkgs':
      mapAttr "callPackage" (callPackage:
        fn: flow (mapAttrOrElse "overrides" (overrides:
          pkgs'.lib.composeExtensions final.scopeOverrides overrides
        ) (pyFinal: pyPrev: pyPrev)) (callPackage fn)
      ) pkgs'
    ))
  );
}

infinisil requested a review from FRidh as a code owner August 25, 2019 03:14

ofborg bot added the 6.topic: python label Aug 25, 2019

infinisil mentioned this pull request Aug 25, 2019

[Do Not Merge [yet]] Update jupyterlab #67423

Closed

7 tasks

ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux labels Aug 25, 2019

infinisil force-pushed the python-makeScope branch from c5f7278 to 97a7887 Compare September 5, 2019 00:22

infinisil mentioned this pull request Sep 6, 2019

Typed nixpkgs.config married to NixOS #57123

Closed

veprbl mentioned this pull request Sep 15, 2019

Can we deprecate pyyaml_3? #68587

Closed

infinisil mentioned this pull request Sep 17, 2019

Python's packageOverrides isn't composable #44426

Open

FRidh mentioned this pull request Sep 17, 2019

Enhancing lib.makeScope #68967

Closed

infinisil closed this Dec 12, 2019

FRidh mentioned this pull request May 9, 2020

Python packageOverrides improvements #87388

Draft

infinisil mentioned this pull request Oct 2, 2020

Use cases supported by the existing module system and overlays/overrides tweag/nix-ux#20

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pythonPackages: Switch to lib.makeScope for `overrideScope` support #67422

pythonPackages: Switch to lib.makeScope for `overrideScope` support #67422

infinisil commented Aug 25, 2019 •

edited

Loading

veprbl commented Aug 25, 2019

infinisil commented Aug 25, 2019

jonringer commented Aug 25, 2019 •

edited

Loading

infinisil commented Aug 25, 2019

FRidh commented Aug 26, 2019 via email

infinisil commented Aug 26, 2019

infinisil commented Aug 26, 2019

jonringer commented Aug 26, 2019

FRidh commented Aug 27, 2019 via email

infinisil commented Aug 31, 2019

FRidh commented Aug 31, 2019

infinisil commented Aug 31, 2019

FRidh commented Aug 31, 2019

infinisil commented Sep 5, 2019

FRidh commented Sep 6, 2019

bb010g commented Oct 23, 2019

bb010g commented Oct 27, 2019

infinisil commented Nov 2, 2019

FRidh commented Nov 2, 2019

infinisil commented Nov 2, 2019

infinisil commented Dec 12, 2019

jonringer commented Dec 12, 2019

mikepurvis commented Jul 16, 2021

bb010g commented Oct 25, 2021

pythonPackages: Switch to lib.makeScope for overrideScope support #67422

pythonPackages: Switch to lib.makeScope for overrideScope support #67422

Conversation

infinisil commented Aug 25, 2019 • edited Loading

Motivation for this change

Things done

veprbl commented Aug 25, 2019

infinisil commented Aug 25, 2019

jonringer commented Aug 25, 2019 • edited Loading

infinisil commented Aug 25, 2019

FRidh commented Aug 26, 2019 via email

infinisil commented Aug 26, 2019

infinisil commented Aug 26, 2019

jonringer commented Aug 26, 2019

FRidh commented Aug 27, 2019 via email

infinisil commented Aug 31, 2019

FRidh commented Aug 31, 2019

infinisil commented Aug 31, 2019

FRidh commented Aug 31, 2019

infinisil commented Sep 5, 2019

FRidh commented Sep 6, 2019

bb010g commented Oct 23, 2019

bb010g commented Oct 27, 2019

infinisil commented Nov 2, 2019

FRidh commented Nov 2, 2019

infinisil commented Nov 2, 2019

infinisil commented Dec 12, 2019

jonringer commented Dec 12, 2019

mikepurvis commented Jul 16, 2021

bb010g commented Oct 25, 2021

pythonPackages: Switch to lib.makeScope for `overrideScope` support #67422

pythonPackages: Switch to lib.makeScope for `overrideScope` support #67422

infinisil commented Aug 25, 2019 •

edited

Loading

jonringer commented Aug 25, 2019 •

edited

Loading