conda 4.4.7 does not work with nb_conda_kernels in new/reinstalled envs

[michaelaye]

In base env my launch log for a jupyter notebook looks like that:

(stable) └─❱❱❱ conda activate base                                                   +9035 17:51 ❰─┘

┌─( ~/D/src )────────────────────────────────────────────────────────────────────────( @macd2860 )─┐
(base) └─❱❱❱ jn .                                                                    +9036 17:51 ❰─┘
[I 17:51:31.745 NotebookApp] [nb_conda_kernels] enabled, 8 kernels found
[I 17:51:32.046 NotebookApp] Loading IPython parallel extension
[I 17:51:32.063 NotebookApp] [jupyter_nbextensions_configurator] enabled 0.3.0
[I 17:51:32.066 NotebookApp] Serving notebooks from local directory: /Users/klay6683/Dropbox/src
[I 17:51:32.066 NotebookApp] 0 active kernels

I recently reinstalled by stable setup, which is a Python 3.6 env with all my everyday packages and no GH master stuff.
Since the reinstall, nb_conda_kernels can't call conda from within the stable env:

(base) └─❱❱❱ conda activate stable                                                   +9037 17:52 ❰─┘

┌─( ~/D/src )────────────────────────────────────────────────────────────────────────( @macd2860 )─┐
(stable) └─❱❱❱ jn .                                                                  +9038 17:52 ❰─┘
[E 17:52:50.245 NotebookApp] [nb_conda_kernels] couldn't call conda:
    [Errno 2] No such file or directory: 'conda': 'conda'
[I 17:52:50.246 NotebookApp] [nb_conda_kernels] enabled, 0 kernels found
[I 17:52:50.540 NotebookApp] Loading IPython parallel extension
[I 17:52:50.557 NotebookApp] [jupyter_nbextensions_configurator] enabled 0.3.0
[I 17:52:50.560 NotebookApp] Serving notebooks from local directory: /Users/klay6683/Dropbox/src
[I 17:52:50.560 NotebookApp] 0 active kernels

while my older envs still work fine, e.g. my Python 3.5 env.

I noticed that I can of course run my work from the base env because my kernels from the other envs are being found there, but imagine it wouldn't be working there as well, then I couldn't work at all, because I don't recall anymore how to manually install kernels, so scary indeed.

I'm running these on a macOS 10.12.6 with miniconda3 and conda-forge as default.

[michaelaye]

This is now blocking my work on another machine, please help! No env discovers the installed kernels anymore.

Update: Phew, sorry, the base env still discovers the kernels, just others not anymore.

[michaelaye]

i went back to conda 4.3.x and things are working.

[goweiting]

Hi, im experiencing the same problem after upgrading to conda 4.4.7. was working on 4.3.30 and nb_conda_kernel was able to detect all the kernels from different packages.

I had 4.3.30 on two separate system: mac OS X and a scientific linux - and nb_conda_kernel worked. upgraded to 4.4.7 on my ubuntu 16.04 and it broke.

[dartdog]

Cross referencing jupyter/jupyter_client#338

[michaelaye]

@goweiting did you directly go to 4.4.7 from 4.3.x? Because I tried 4.4.7 now again after a complete wipe of my miniconda folder, and everything seems to be working now.

On my side, this issue could be closed.

[goweiting]

@michaelaye some of my systems are still running processes so i am not ready to shut them down yet.
For the one of my Ubuntu, removing:
_nb_ext_conf
(conda remove -n <env_name> _nb_ext_conf),
fixes the problem and am able to see different environment, including IRKernel.

[benbovy]

I don't think this issue has been fixed yet.

Running from jupyter,nb_conda_kernels tries to call the conda executable to get information about all the environments.

The problem is that since conda 4.4 it appears there is no link to the conda executable added to new/reinstalled envs, which therefore cannot be found in $PATH, hence the error above.

For environments that have been created before conda 4.4 it still works because a link to the conda executable is still present in those envs. This also works with the base environment since the conda executable is installed in that env.

A workaround for new/reinstalled envs would be for example to manually create a symlink to the conda executable in their bin folders.

A cleaner solution would be that nb_conda_kernels uses the $CONDA_EXE env variable to find the conda exectuable.

xref: jupyterlab/jupyterlab#4598

[tjd2002]

@benbovy Thanks very much for the workaround and the fix. I had the same issue in conda 4.5.3 in a non-base env: nb_conda was not usable without your fix.

[mcg1969]

I'm trying to address this in #80

[mcg1969]

Actually, #82 is going to address this.

[mcg1969]

We've merged #82, so we should be good now. Of course, the packages still need to be built, and I'll work on that.

[cecilialee]

Thanks @goweiting your code works for me! In my case I installed the IRkernel and I cannot find my kernels anymore.

[michaelaye]

this is still not fixed with nb_conda_kernels 2.1.0 ?

[michaelaye]

was this fixed in 2.1.1? If yes, I'm not seeing that version on any osx-64 channel?

[tjd2002]

Not available for linux either:

$ conda search nb_conda_kernels'>=2.1[subdir=osx-64]' -c defaults -c conda-forge
Loading channels: done
# Name                  Version           Build  Channel
nb_conda_kernels           2.1.0          py27_0  pkgs/free
nb_conda_kernels           2.1.0          py27_0  conda-forge
nb_conda_kernels           2.1.0          py35_0  pkgs/free
nb_conda_kernels           2.1.0          py35_0  conda-forge
nb_conda_kernels           2.1.0          py36_0  pkgs/free
nb_conda_kernels           2.1.0          py36_0  conda-forge

$ conda search nb_conda_kernels'>=2.1[subdir=linux-64]' -c defaults -c conda-forge
Loading channels: done
# Name                  Version           Build  Channel
nb_conda_kernels           2.1.0          py27_0  pkgs/free
nb_conda_kernels           2.1.0          py27_0  conda-forge
nb_conda_kernels           2.1.0          py35_0  pkgs/free
nb_conda_kernels           2.1.0          py35_0  conda-forge
nb_conda_kernels           2.1.0          py36_0  pkgs/free
nb_conda_kernels           2.1.0          py36_0  conda-forge

@mcg1969 Any progress on shipping your latest changes?

[mcg1969]

I've submitted a conda-forge PR
conda-forge/nb_conda_kernels-feedstock#18
I don't know what is involved in getting it merged.

[tjd2002]

Thanks @mcg1969. It looks like the 2.1.1 packages are all up on conda-forge now!
https://anaconda.org/conda-forge/nb_conda_kernels

However, after installing this package on my mac, the extension is not enabled by default. I have to run:
python -m nb_conda_kernels.install --enable

This used to be done in a post-link script that was shipped in the package. Is this change intentional?

[Also, if Continuum is not going to update this package in defaults, should it be removed from there?]

[tjd2002]

So: the problem is that as of #82 the post-link.sh and pre-link.sh scripts are no longer in conda-forge's conda-recipe folder, so they will only get properly installed when the package is built locally.

E.g. when I build the 2.1.1 package locally on my mac, the scripts are included:

$ tar -tjf ~/anaconda3/conda-bld/osx-64/nb_conda_kernels-2.1.1-py36_0.tar.bz2 |sort
bin/.nb_conda_kernels-post-link.sh
bin/.nb_conda_kernels-pre-unlink.sh
info/LICENSE.txt
info/about.json
[...]

But the version shipped by conda-forge does not include these scripts:

$ tar -tjf ~/anaconda3/pkgs/nb_conda_kernels-2.1.1-py36_0.tar.bz2
info/index.json
info/LICENSE.txt
[...]

This is not caught by any tests because the package builds and tests correctly as part of this repo, and conda-forge doesn't seem to run the package's own tests (see https://github.com/conda-forge/nb_conda_kernels-feedstock/blob/master/.travis.yml)

The solution would seem to be: Find a way to coerce conda-forge to include the post-link and pre-link scripts from the source repo. (This could be as simple as copying them over to the conda-recipe folder, but I'm not sure the best way to do this).

I'll also open an issue on the feedstock page to track this.