Better documentation for python.buildEnv / virtualenv

[jgeerds]

Thanks to @iElectric we already have some documentation about Python in Nix(OS). However, I find it still confusing when developing Python software with NixOS. There are multiple ways to get things done and it is unclear how they differentiate to each other:

with import <nixpkgs> {};

python.buildEnv.override {
  extraLibs = [ pkgs.pythonPackages.pyramid ];
  ignoreCollisions = true;
}

What are the use cases? Is it meant for packaging new software for nixpkgs or also for developing environments?

      with import <nixpkgs> {};

      buildPythonPackage {
        name = "myproject";

        buildInputs = with pkgs.pythonPackages; [ pyramid ];

        src = ./.;
      }

Is this the preferred way to build developing environments? It works very well for me, even IPython is able to import pyramid.

$ nix-shell -p pythonPackages.pyramid zlib libjpeg git

Is this the same as 2) ? (of course, zlib libjpeg and git are missing in the first example).
If not, what is the correct declarative way for doing this?

Here are some examples from our wiki:

$ cat configuration.nix
{
  packageOverrides = pkgs: with pkgs; {
    myPythonEnv = pkgs.myEnvFun {
        name = "mypython";
        buildInputs = [
          python33
          python33Packages.pyramid
          python33Packages.jinja2
        ];
    };
  };
}

This one use myEnvFun. Why do I need myEnvFun? Just for installing a "permament" environment whereas nix-shell is "on-the-fly" ? Are there more differences?

Another snippet:

 # cat default.nix
 let
   pkgs = import <nixpkgs> {};
 in
 { stdenv ? pkgs.stdenv, python ? pkgs.python, pythonIRClib ? pkgs.pythonIRClib }:

 stdenv.mkDerivation {
   name = "python-nix";
   version = "0.1.0.0";
   src = ./.;
   buildInputs = [ python pythonIRClib ];
 }

Is this obsolete? If not, what is the difference between stdenv.mkDerivation and the other examples above?

Another one:

   {
     # Define a myEnvFun environment, which makes use of pythonPlone
     packageOverrides = pkgs: rec {

       # Define a version of Python which has access to all the Plone modules
       # For 14.04 don't include "buildEnv".
       pythonPlone = pkgs.pythonFull.buildEnv.override {
         extraLibs = with pkgs.python27Packages; [ Plone ];
       };

       plone = pkgs.myEnvFun {
         name = "plone";
         buildInputs = with pkgs; [
           pythonPlone
           python27Packages.zc_buildout_nix
           stdenv # provides gcc for building python c extensions
        ];
        # Export the path to the custom version of Python with all the goodies
        # Other Python command line utilies could also be used, e.g. IPython
        extraCmds = 
          export PYTHONHOME=${pythonPlone}
          unset http_proxy # otherwise downloads will fail ("nodtd.invalid")
        ;
       };

     };
   }

I don't understand why we introduce pythonPlone. Is this really needed? What about something like:

[...]
       plone = pkgs.myEnvFun {
         name = "plone";
         buildInputs = with pkgs; [
           pythonFull
           python27Packages.Plone
           python27Packages.zc_buildout_nix
           stdenv # provides gcc for building python c extensions
        ];
[...]

?

My workflow is usually something like this one:

$ cat default.nix 
let
  pkgs = import <nixpkgs> {};
in rec {
  fooEnv = pkgs.myEnvFun {
    name = "foo";
    buildInputs = with pkgs; [
      stdenv
      python34
      python34Packages.virtualenvwrapper
    ];
  };
}

I also commit this file to the project repository. This way other developer have the same dev environment. It works for nix-shell and nix-env -f default.nix -iA env-foo. Sometimes I use it in conjunction with pip and virtualenv.

Is something wrong with my way?

PS: Is there a way to auto activate a specific virtualenv when using nix-shell ?

[domenkozar]

Yes, indeed, there are many ways with Nix. Best practices are opinionated, but I agree we could provide some guidance and differences between the approaches.

Imho myEnvFun shouldn't be used/shown since it's not documented.

[cstrahan]

At first glance, I thought myEnvFun added features not found in nix-shell (aside from producing a binary that drops you a in a shell), but now I see that nix-shell does pretty much everything as expected. I'm of the opinion that myEnvFun should be deprecated, unless there's some special feature I'm not aware of.

[7c6f434c]

At first glance, I thought myEnvFun added features not found in nix-shell (aside from producing a binary that drops you a in a shell), but now I see that nix-shell does pretty much everything as expected. I'm of the opinion that myEnvFun should be deprecated, unless there's some special feature I'm not aware of.

The very fact that myEnvFun builds an output that can be made a GC root
is already a useful feature.

I would also say that using myEnvFun seems cleaner when I try to write
down a definition of an environment where I am not going to build
anything.

[wmertens]

I agree on the myEnvFun GC root although there is work being done on
non-nixos services which would supersede myEnvFun I think.

Shall we take the Python options presented here and put them on the Wiki or
should there be a canonical way documented in the manual?

On Fri Jan 23 2015 at 6:29:17 AM Michael Raskin [email protected]
wrote:

At first glance, I thought myEnvFun added features not found in
nix-shell (aside from producing a binary that drops you a in a shell),
but now I see that nix-shell does pretty much everything as expected. I'm
of the opinion that myEnvFun should be deprecated, unless there's some
special feature I'm not aware of.

The very fact that myEnvFun builds an output that can be made a GC root
is already a useful feature.

I would also say that using myEnvFun seems cleaner when I try to write
down a definition of an environment where I am not going to build
anything.

—
Reply to this email directly or view it on GitHub
#5623 (comment).

[7c6f434c]

I agree on the myEnvFun GC root although there is work being done on
non-nixos services which would supersede myEnvFun I think.

I am also quite interested in non-NixOS services and even in
pure-functional definitions of services/environments.

Of course, pure-functional foundation under NixOS would include direct
(and better in uniformity sense) myEnvFun replacement.

I agree with you about the future, I only wanted to add some details
about present.

[cillianderoiste]

You even left out one option :D ... Since nixos-containers were added,
I've moved from using myEnvFun to using nixos-containers entirely. I
do python web development, so this suits me really well. I work on
fairly big projects which need to compile a lot of c modules and can
easily take more than 30 minutes to build from scratch, so having a
persistent environment is vital for me. Hopefully some day we'll have
the tools to conveniently package these things in nix which will make
things much easier.

NixOS: The Purely Functional Linux Distribution
http://nixos.org

[7c6f434c]

You even left out one option :D ... Since nixos-containers were added,
I've moved from using myEnvFun to using nixos-containers entirely. I

This requires running NixOS, I guess... myEnvFun is portable.

[domenkozar]

One can also create GC roots with nix-shell --out-link result

[domenkozar]

Feel free to add anything to http://nixos.org/nixpkgs/manual/#python and clean up wiki with outdated info :)

[cillianderoiste]

Is there a way to enter a persistent nix-shell, without evaluating the nix expression? Perhaps by sourcing an env-vars file?

[wmertens]

@cillianderoiste sounds like a good thing to add to nix-shell, --persistent which would create a ./nix-shell-env.sh script or something like that and a GC root for the environment, or maybe add it to the nix-env managed stuff. Imperative environment building.

[cillianderoiste]

It would be cool if there is a way to do that, or it can be added, then we could get rid of myEnvFun.

[domenkozar]

@cillianderoiste because evaluating nixexprs is slow?

[cillianderoiste]

@iElectric ah no, it's just I don't want to update a build environment just to enter it. For example, (before nixos-containers) I used myEnvFun for environments where I would run buildout to make a plone instance. This can take 30 mins plus. I don't want to change that environment once it's been created, unless I really have to ... otherwise libraries may have been garbage collected or otherwise change in a way that breaks the compiled python c-modules in eggs. I like that myEnvFun decouples the creation of the environment from the loading of the environment for this reason.

[domenkozar]

You can have something like my-nix-shell that will execute nix-shell -I /my/local/nixpkgs and then pass rest of the arguments. Then you can update /my/local/nixpkgs once a year.

[cillianderoiste]

Nice, but would you want a separate branch or clone of nixpkgs for each environment then?

[cillianderoiste]

I have a feeling it's already possible by setting TMPDIR to the local directory before running nix-shell, and then sourcing env-vars ... but I'd need to test that.

[domenkozar]

my-nix-shell can use the same nixpkgs clone for all environments. That's almost identical to a profile that doesn't auto-update

[domenkozar]

Well, when used together with --out-link. The UI of this could be better, I agree :)

[cillianderoiste]

I'll see if I can use it like this and if it works fine I can replace the examples of myEnvFun in the wiki. Perhaps at a later stage we could wrap this functionality up in a --persistent option as @wmertens suggested.

[domenkozar]

@edolstra thoughts?

[cillianderoiste]

I had a look at using nix-shell using the clang example in the wiki: https://nixos.org/wiki/Howto_develop_software_on_nixos#Example_1:_Using_nix-shell_to_Initialize_a_Development_Shell

$ mkdir myenv
$ TMPDIR=`pwd`/myenv nix-shell default.nix -A clangEnv --out-link gcroot

I can exit the nix-shell, and then source myenv/env-vars which gives me clang in my PATH again, so that kind of works. Also, after adding a shellHook to the expression, I can run that manually with $shellHook.

I was expecting "--out-link gcroot" to create a symlink called "gcroot" pointing to somewhere in /nix/var/nix/gcroots/, but no symlink was created, and I couldn't find anything in /nix/var/nix/gcroots/ that looked related.

Keeping copies of nixpkgs around, as @iElectric suggested, is a reasonable option.

Just to be clear, I don't personally have a use for myEnvFun any more, so I don't mind if it gets removed, and I'd be happy to remove the parts of the wiki that refer to it. Perhaps nix-rehash or NixUP is the way to go for a persistent development environment outside of NixOS.

[bennofs]

You can create gcroots for each dependency using the following command:

$ nix-shell ... --indirect --add-root <name>

This will create one symlink for each dependency, named <name>-<n>, where <n> is a number.

Because of this, it's best to put those in a separate directory. The command I use for haskell is similar to this:

$ mkdir .gcroots
$ nix-shell . --indirect --add-root .gcroots/dep

That command creates .gcroots/dep-1, etc, one symlink for each dependency which keeps that dep alive.

[wmertens]

Ok now wrap that all up in a script that either runs nix-shell or loads
environment variables and you get a nice imperative myEnvFun... Should be a
hit!

On Thu Feb 12 2015 at 11:22:15 AM Benno Fünfstück [email protected]
wrote:

You can create gcroots for each dependency using the following command:

$ nix-shell ... --indirect --add-root

This will create one symlink for each dependency, named -, where
is a number.

Because of this, it's best to put those in a separate directory. The
command I use for haskell is similar to this:

$ mkdir .gcroots
$ nix-shell . --indirect --add-root .gcroots/dep

That command creates .gcroots/dep-1, etc, one symlink for each dependency
which keeps that dep alive.

—
Reply to this email directly or view it on GitHub
#5623 (comment).

[cillianderoiste]

This looks fine to me. I'll remove references to myEnvFun from the wiki and add a section on how to create persistent environments with nix-shell as discussed here.

[cillianderoiste]

As @bennofs pointed out on IRC, you can use nix-instantiate to create the environment, and nix-shell to load it:

$ nix-instantiate . --indirect --add-root `pwd`/shell.drv
$ nix-shell `pwd`/shell.drv

That seems like an elegant replacement for myEnvFun. I'll work on the wiki a bit more to provide some examples and mention some useful environment variables etc.

[benmos]

It's worth noting though that the nix-shell behaviour used here relies on the ".drv" file having an absolute path (and ending in ".drv") to prevent nix-shell from trying to interpret it as Nix code.

This nix-shell behaviour is also - AFAICS - undocumented.

[samuelrivas]

It took me a while to find out that one can run this to get all dependencies rooted for GC

$ nix-shell . --indirect --add-root .gcroots/dep

It is not mentioned in the man page for nix-shell, but I think is worth doing. Unless I am very wrong the

$ nix-instantiate . --indirect --add-root `pwd`/shell.drv
$ nix-shell `pwd`/shell.drv

trick just adds a gc-root for the drv file, which is not what I would think of when asking for a "gc rooted nix shell environment." Generating the drv file is definitely not the most time consuming part of running nix-shell in a clean environment :)

[FRidh]

I'm writing a Python on Nix tutorial and obviously I want to have up to date info there.

At the time of writing I included myEnvFun, even though it is not supported. What I like about myEnvFun though is that you are able to define your environments together in your config.nix, although it is a pity you still need to install them imperatively. I guess we're all looking forward to declarative user profiles (#9250) :-) But I might remove myEnvFun or simply mention that it is not supported.

Anyway, if you like to help writing, please join. It is just a Markdown file, so there shouldn't be any barrier.

[domenkozar]

Thanks @FRidh, really appreciated. I'll go through it and leave comments when I find some time :)

[CMCDragonkai]

What's the difference between buildEnv and myEnvFun?

[domenkozar]

myEnvFun will get removed #18315 to remove the confusion between so many options.

[Profpatsch]

(triage) has the documentation improved in the meantime?

[FRidh]

Note the tutorial I mentioned has long since been merged into the Nixpkgs manual. I think it contains all relevant information. If not, people can ask for or propose specific parts.