Nix garbage collects build dependencies even when gc root is added.

[vrthra]

Steps:

• I have a development environment set under $PWD/etc/shell.nix.
• I do nix-instantiate ./etc/shell.nix --indirect --add-root $PWD/etc/shell.drv
• Then nix-shell --pure $PWD/etc/shell.drv
• Exit nix-shell, then nix-collect-garbage -d
• Try nix-shell --pure $PWD/etc/shell.drv

Result: the entire set of packages specified under shell.nix is downloaded again

• nixos-version 16.03.1083.d745044 (Emu)
• nix-env --version (Nix) 1.11.2
• nix-instantiate --eval '<nixpkgs>' -A lib.nixpkgsVersion 16.09pre-git

[vrthra]

So, this seems to require this incantation to make sure that the development environment is not gc-d:

mkdir -p .gcroots
nix-shell . --pure --indirect --add-root .gcroots/dep

[domenkozar]

@vrthra can you elaborate?

[vrthra]

After seeing that a command exists (nix-shell . --pure --indirect --add-root .gcroots/dep) for creating a non-gcd environment, I was not sure that this was a bug any more. (When I raised the bug I did not know of the above command.). Should I reopen this?

[Ericson2314]

I think this is expected behavior, but there should be an easier way to create a root for the build environment without having to open a shell.

[ashgillman]

I am having a similar issue, and I think I can shed some light. The issue crops up when you make a derivation with no src. nix-instantiate will then make a .drv, and nix-shell will happily load it, however you can't build the derivation as it has no output. I guess when the GC looks at the .drv it uses the same mechanism as the building and can't determine the dependencies.

[vcunat]

No, I don't think the mechanisms are close. Instead, I suspect that nix-shell . does some additional magic that vanilla nix-instantiate shell.nix doesn't cover.

[ashgillman]

@vcunat That's what I mean, and that he garbage collector (i.e., nix-store) also doesn't include the additional magic (I am guessing), and so won't correctly prevent collection.

Here's a MWE:

{9:06}ash@daftpunk-rbh:~/tmp ➭ cat py33.nix 
let
  pkgs = import <nixpkgs> {};
in
{ stdenv ? pkgs.stdenv }:

stdenv.mkDerivation {
  name = "python33-test";
  buildInputs = (with pkgs; [
  ]) ++ (with pkgs.python33Packages; [
    python
  ]);
}

{9:07}ash@daftpunk-rbh:~/tmp ➭ nix-store --gc --print-dead | grep python3-3
finding garbage collector roots...
determining live/dead paths...
/nix/store/3lnk8y1xn4idajsn7ajlg6fc2shm1ck7-python3-3.3.6.drv
/nix/store/5dq1nr4s0xh5iry548w09423zwpkk9w7-python3-3.3.6
{9:07}ash@daftpunk-rbh:~/tmp ➭ nix-instantiate py33.nix --indirect --add-root $PWD/py33.drv 
/home/ash/tmp/py33.drv
{9:08}ash@daftpunk-rbh:~/tmp ➭ ls -l /nix/var/nix/gcroots/auto | grep py33
lrwxrwxrwx 1 ash ash 22 Sep 19 09:08 ppnsyyqpfxh29bhprd744wd9yr1hk87w -> /home/ash/tmp/py33.drv
{9:08}ash@daftpunk-rbh:~/tmp ➭ nix-store --gc --print-dead | grep python3-3                
finding garbage collector roots...
determining live/dead paths...
/nix/store/5dq1nr4s0xh5iry548w09423zwpkk9w7-python3-3.3.6
{9:09}ash@daftpunk-rbh:~/tmp ➭ nix-shell py33.nix --indirect --add-root .gcroots/py33dep

[nix-shell:~/tmp]$ exit
{9:10}ash@daftpunk-rbh:~/tmp ➭ ls -l /nix/var/nix/gcroots/auto | grep py33
lrwxrwxrwx 1 ash ash 32 Sep 19 09:10 9nz7jcakc31gx6d1di5gh4ffb8ras07c -> /home/ash/tmp/.gcroots/py33dep-4
lrwxrwxrwx 1 ash ash 36 Sep 19 09:10 b93splkqrpj8lj216kgqjxmyj238q4pd -> /home/ash/tmp/.gcroots/py33dep-3-doc
lrwxrwxrwx 1 ash ash 32 Sep 19 09:10 bhy9zmzmgpzqalrvvi212z215wbxawaj -> /home/ash/tmp/.gcroots/py33dep-3
lrwxrwxrwx 1 ash ash 32 Sep 19 09:10 bm9ml0zamryscgs9pyqlqh16y5gxmv4r -> /home/ash/tmp/.gcroots/py33dep-2
lrwxrwxrwx 1 ash ash 30 Sep 19 09:10 m03hlcvvmpaabhhwi6vvb20idmav2mla -> /home/ash/tmp/.gcroots/py33dep
lrwxrwxrwx 1 ash ash 22 Sep 19 09:08 ppnsyyqpfxh29bhprd744wd9yr1hk87w -> /home/ash/tmp/py33.drv
{9:10}ash@daftpunk-rbh:~/tmp ➭ nix-store --gc --print-dead | grep python3-3
finding garbage collector roots...
removing stale link from ‘/nix/var/nix/gcroots/auto/q19dnb49593w3gj7bjvxrlz8239z9d3a’ to ‘/run/user/1000/nix-shell.gpLNg8/derivation’
determining live/dead paths...
{9:10}ash@daftpunk-rbh:~/tmp ➭ 

The GC still wants to delete the py33 installation when you use the traditional nix-instantiate. So the nix-shell with --indirect --add-root seems to fix (I wasn't aware of this earlier). But the nix-instantiate is still what seems to be the recommended option in the docs, which can cause a lot of pain.

[CMCDragonkai]

How come these options --indirect and --add-root are not documented inside nix-shell? It would be great if nix-shell's manual came with some mention about how to prevented GCing shell environments.

[Ralith]

The nix-shell options quoted above seem to be silently ignored today. I'm accomplishing something similar by parsing the output of nix show-derivation somewhat laboriously. It would be much nicer for nix tools to have some first-class way to reference a build-time closure.