Stop importing pip, use subprocesses instead. Will close #710

[dwpaley]

As discussed in: pypa/setuptools#3297 it seems that importing pip is just not supported. Our use case was not really compelling, we were (1) checking the version and (2) using a pip._internal.main api which is a thin (and to be deprecated) wrapper over pip._internal.cli.main.main There's a clear discussion here: https://pip.pypa.io/en/stable/user_guide/#using-pip-from-your-program of why using pip via subprocess is preferred.

This is a draft because there's a second place in libtbx/auto_build/install_base_packages.py where pip._internal.main is used in a similar way.

[dwpaley]

In principle this is done, but I don't know how to exercise the build_python_module_pip method in install_base_packages.py...

[bkpoon]

The install_base_packages.py code is only really for building Python 2.7 dependencies and I doubt that it will work now since we do not specify the versions of every dependency. Probably, one dependency will be installed via pip which pulls in an incompatible dependency that we did not version.

But you can try to build a minimal set of Python 2.7 dependencies with

python bootstrap.py base --builder=cctbxlite

[dwpaley]

@JBlaschke From Billy:

maybe you will want to test the XFEL/DIALS builds at NERSC since the pkg_utils.py code is used for setting up all those entry points. The XFEL CI might be enough, but I don't know if Johannes' installation scripts do anything extra.

[JBlaschke]

Hi Folks, I am still waiting to test this on Perlmutter (longer than expected maintenance), but I can say that it works on Cori.

[JBlaschke]

This looks like a run-of-the-mill replacement of the pip module with a sub-process call. The only concern RE HPC is if you tried that with MPI then you might get into trouble because MPI (and libfabric) are not considered fork safe, while subprocess will call fork. I don't think that this will be an issue for an installer however -- so I think it's safe to proceed.

Another thing to consider is that this might be called within a workflow management tool. In that case we need to make sure that the workflow tool exposes the correct environment to all children. Most workflow tools use fork or execve so we should be safe there, and I would counsel vigilance in case we see the "wrong" pip being picked up.

[JBlaschke]

@dwpaley for cleanliness have you or @bkpoon considered: https://github.com/amoffat/sh

[JBlaschke]

@bkpoon I have a question about the most elegant way to tell bootstrap.py to use a specific branch: Right now I edit boostrap.py manually to add the -b <branch name> to the module. For someone who uses boostrap.py deploy and build in one single command, is there an argument that I can use to do straight to a specific module's branch?