Make pip install optional if using system Python

[JeremyRand]

Motivation

It looks like #1834 results in the backend always running pip install when it decides that a dependency isn't present. This is problematic for users who want to manage dependencies themselves (e.g. installing packages via apt).

Description

Would you take a PR that makes pip install optional in the backend if system Python is in use?

Alternatives

Can't think of any alternatives; feel free to suggest any that occur to you.

[joeyballentine]

That's the way chaiNNer has always worked. chaiNNer literally can't function without its dependencies installed so I'm not sure why you would even want that. The way I see it is if you're using system python, you're opting-in to us installing things to your system python, because we need to.

The alternative is just not letting users use system python at all, and forcing integrated python.

[JeremyRand]

Hmm, lemme see if I can figure out why this is behaving differently for me now. Something definitely changed in the last couple weeks; I assumed it was this.

[joeyballentine]

Is it trying to install things every time? Like even when they're already installed? Because it's supposed to only install things that are not installed

[RunDevelopment]

npm run dev did not install deps before and uses system python. Did you mix up development mode with system python?

[JeremyRand]

Oh. I think I see what's going on. pynvml was added as a dependency, and pip was trying to install it (and failing to do so in a loop). Can that dep be made optional? AFAICT it's only relevant for Nvidia GPU's (which I don't have), and that package has closed-source dependencies that I'd prefer not to have on my system (and also seem to have installation problems on POWER even if I was comfortable with the closed-source nature of that package).

[joeyballentine]

Hmmm... That was added as a way to let us check what GPU is installed so we can figure out what versions of certain dependencies to install (for pytorch and onnx specifically). I didn't realize it would cause issues on certain machines but i guess that's just the nature of python dependencies with natively compiled code.

If you can find me a way to check what GPU someone has in a cross platform way, I'd consider getting rid of it. But since i currently have no alternative, we kinda need it

[JeremyRand]

I see. I'll look into it and get back to you once I've evaluated options a bit.

[joeyballentine]

One thing I could do is add a flag for certain dependencies to mark them as optional, and attempt installing them separately -- and if they fail, we can just ignore them from then on. Then we'd just treat your system as if it didn't have an nvidia gpu

[JeremyRand]

Yeah I think ignoring it if it fails to automatically install is reasonable.

[joeyballentine]

Also for the record, this package doesn't actually work for getting us cross-platform GPU info, it just gives us cross-platform Nvidia GPU info and returns an error when no Nvidia GPU is found, which worked well enough for what we needed. The old way we did this was either using nvidia-smi in a subprocess or manually looking for the nvidia-smi binary on the machine and using it in a subprocess. That package is just bindings for the same library that nvidia-smi uses, meaning we can get all the same info with it without needing to search for anything or use a subprocess, so it's a pretty clean way of doing it. But it can be ignored if it fails, since we can just treat it like you have an AMD GPU from then on out.

[JeremyRand]

Yeah unfortunately I just spent 10 minutes looking through GitHub hoping there was a Python package for enumerating either GPU's or PCIe devices, but it doesn't look like there's anything (other than a couple packages that just launch lspci as a subprocess). So I think your suggestion of ignoring an installation failure is likely to be the best approach.

[joeyballentine]

@JeremyRand did it say why it was failing to install? The wheel file for it is supposedly py3-none-any which means it should be installable on any platform or python version. It also doesn't look like it has any dependencies so I'm curious what closed-source dependencies you were talking about

[JeremyRand]

Hmm, so Debian's repos mark python3-pynvml as contrib, with a dependency on libnvidia-ml1 (which is marked as non-free). Does the PyPI distribution of pynvml assume that the user will install libnvidia-ml1 themselves? If so, will it work for chaiNNer purposes without libnvidia-ml1 (assuming the user doesn't want to use an Nvidia GPU)?

[joeyballentine]

I have no idea what libnvidia-ml1 is (I'm assuming its some linux thing related to nvidia drivers). This package just binds to the NVML stuff included with the nvidia drivers. if you install the package without an nvidia GPU, it simply throws an error when you try to initialize it. I did not have to install anything other than this package for it to work.

[JeremyRand]

Hmm, I'll see if I can inspect the contents of the PyPI package and see how it differs from what Debian distributes. Will get back to you in a day or two.

[JeremyRand]

OK yes, you are right that there's no closed-source dependency. This is a bug in Debian's packaging; they are marking the Nvidia library as a dependency when it should be a recommendation or a suggestion instead. I will report that bug to Debian. Give me a day or so to make sure there are no remaining issues, and then I'll close this issue if there are no problems.

[JeremyRand]

Successfully got current chaiNNer to open after manually installing the needed deps with pip. As you predicted, no installation of an Nvidia closed-source library was needed. The automatic usage of pip in chaiNNer doesn't seem to play well with ppc64le, but that's fine, I don't mind manually running pip when the deps change. Tentatively closing this issue; I'll re-open if I discover that there was remaining breakage that I missed.