make cmake use env python #196
Conversation
CustomPythonCmakeArgs forces CMake to pickup the environment python libraries and include but the interpreter is still picked using PythonInterp which picks up the system default instead of the environment python. This cause issues with vitualenv.
make cmake use env python
|
Hm, this has the potential to break everything. The whole multiple-versions-of-python-on-one-machine thing is the single most frequent reason for failed YCM installations. So we need to be really sure this is doing the right thing. I'd appreciated it if various people could try out this change; the more OS's/distros we cover, the better. |
|
The |
|
happy to give this a spin as i have 2 macs - 1 which the install always works and 1 which i always have do battle with homebrew/python/etc. However, @amchoukir could you elaborate on exactly what problem this is solving? Does it relate to an issue on the tracker? Are there some obvious test steps we should go through? Ta, |
|
Thanks for having a look and giving it a try.
But there is still an issue with
My setup is as follows: OS X 10.10.3 (Maverick) The test steps I went through are:
|
|
To my understanding |
|
|
||
| return [ | ||
| '-DPYTHON_LIBRARY={0}'.format( python_library ), | ||
| '-DPYTHON_INCLUDE_DIR={0}'.format( python_include ) | ||
| '-DPYTHON_INCLUDE_DIR={0}'.format( python_include ), | ||
| '-DPYTHON_EXECUTABLE={0}'.format(python_executable) |
There was a problem hiding this comment.
Whitespace needed around python_executable. :)
|
@vheon yes it might definitely be needed. I have access to linux servers on which I could try it out, if someone could have a look at windows. |
@amchoukir Then this PR should be modified to run on linux as well right? |
|
I think we should hold out on this change while I get the Travis OSX builds working. I've had to do some work to streamline the build which I think obsolete this change.
|
|
@puremourning Sure, np. |
|
Here are the results of my tests on Linux Scenario 1: System default sanity check -- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython2.7.so (found suitable version "2.7.6", minimum required is "2.6") Scenario 2: Vanilla virtualenv with 2.7.10 -- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython2.7.so (found suitable version "2.7.6", minimum required is "2.6") Wrong library picked up. It picked up the system default instead of the 2.7.10. Scenario 3: virtualenv with 2.7.10 and python-config copied over -- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython2.7.so (found suitable version "2.7.6", minimum required is "2.6") Wrong library picked up. It picked up the system default instead of the 2.7.10. So here python-config does not fix the issue. Here are the specifics of python-config (ycmd2.10)vagrant@vvv: Most likely this is due to the difference in paths between OS X and Linux Next Steps I will dig into build.py to check what happens. |
Yes, this is annoying. I found the same problem when making the ycmd build/test run on Travis OS X. Did you raise this with virtualenv folks? If so could we get a reference? Thanks, |
|
@puremourning yes Paul opened the following issue: |
|
Hello Everyone, I am back from vacation and I will have a look at why the build does not work on virtualenv for Linux. Meanwhile the fix to get the python-config copied in virtualenv has been merged: |
[READY] Do not rely on CMake to find Python libraries ### Problem CMake is not able to find the Python libraries from `pyenv`. We can observe this on Travis. If we look at the build output of the Python 2.6 run on Linux, we have the following line during the CMake configuration: ```sh -- Found PythonLibs: /usr/lib/libpython2.7.so (found suitable version "2.7.3", minimum required is "2.6") ``` which, in addition to be the wrong version, is the system library instead of the `pyenv` one. We confirm this by inspecting the `ycm_core.so` library from the build: ```sh $ ldd ycm_core.so | grep python libpython2.7.so.1.0 => /usr/lib/libpython2.7.so.1.0 (0x00007f0521fa1000) ``` Same issue with the Python 2.7 and 3.3 Linux runs although the library version is correct in these cases. In the Travis configuration file, there is this comment: > This is because stupid cmake 2.8.11 has a bug preventing it from finding the pyenv pythons (ostensibly; I haven't checked, but online reports say the issue is gone with cmake 3.4). I tried with CMake 3.5.1 (3.5.2 is the last version) and it is still not working. That's not all. In some situations, CMake find the wrong Python system libraries (see [this workaround on AppVeyor](https://github.com/Valloric/ycmd/blob/master/ci/appveyor/appveyor_install.bat#L36)) or cannot find them at all (see [this post on ycm-users](https://groups.google.com/forum/#!searchin/ycm-users/windows$20cmake/ycm-users/e6K-grbeUMw/0kzwtAaYCQAJ)). ### Solution From this, it is clear that we cannot rely on CMake to find the Python libraries. We need to find them in the `build.py` script and pass the `PYTHON_LIBRARY` and `PYTHON_INCLUDE_DIR` parameters to CMake, like we already do for OS X. This is implemented by creating a `FindPythonLibraries` function for each supported platform: Linux, OS X, and Windows. For OS X, we are keeping the logic from `CustomPythonCmakeArgs` except that we are dropping support for system Python 2.6. For Linux, we need to use the `ldconfig` tool to find the system Python library on some distributions (e.g. Ubuntu) because it is not installed in its standard location. For Windows, it is straightforward. In all cases, we assume that the Python running the script is the one that will be used to build the library. This PR also fixes an important issue. When building the ycmd library with Clang support and linking it to a non-system Python libray, the dynamic linker will not be able to find the Python library: ```sh $ ldd ycm_core.so | grep python libpython3.4m.so.1.0 => not found $ objdump -x ycm_core.so | grep RPATH RPATH $ORIGIN ``` This is solved by setting the `CMAKE_INSTALL_RPATH_USE_LINK_PATH` option to true (see [this wiki page](https://cmake.org/Wiki/CMake_RPATH_handling) for details on the option): ``` > ldd ycm_core.so | grep python libpython3.4m.so.1.0 => /home/micbou/.pyenv/versions/3.4.4/lib/libpython3.4m.so.1.0 (0x00007f9fb6472000) > objdump -x ycm_core.so | grep RPATH RPATH $ORIGIN:/home/micbou/.pyenv/versions/3.4.4/lib ``` Closes #196 (we don't care about the Python interpreter when building the ycmd library). Fixes #479. <!-- Reviewable:start --> --- This change is [<img src="https://reviewable.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/valloric/ycmd/481) <!-- Reviewable:end -->
[READY] Do not rely on CMake to find Python libraries ### Problem CMake is not able to find the Python libraries from `pyenv`. We can observe this on Travis. If we look at the build output of the Python 2.6 run on Linux, we have the following line during the CMake configuration: ```sh -- Found PythonLibs: /usr/lib/libpython2.7.so (found suitable version "2.7.3", minimum required is "2.6") ``` which, in addition to be the wrong version, is the system library instead of the `pyenv` one. We confirm this by inspecting the `ycm_core.so` library from the build: ```sh $ ldd ycm_core.so | grep python libpython2.7.so.1.0 => /usr/lib/libpython2.7.so.1.0 (0x00007f0521fa1000) ``` Same issue with the Python 2.7 and 3.3 Linux runs although the library version is correct in these cases. In the Travis configuration file, there is this comment: > This is because stupid cmake 2.8.11 has a bug preventing it from finding the pyenv pythons (ostensibly; I haven't checked, but online reports say the issue is gone with cmake 3.4). I tried with CMake 3.5.1 (3.5.2 is the last version) and it is still not working. That's not all. In some situations, CMake find the wrong Python system libraries (see [this workaround on AppVeyor](https://github.com/Valloric/ycmd/blob/master/ci/appveyor/appveyor_install.bat#L36)) or cannot find them at all (see [this post on ycm-users](https://groups.google.com/forum/#!searchin/ycm-users/windows$20cmake/ycm-users/e6K-grbeUMw/0kzwtAaYCQAJ)). ### Solution From this, it is clear that we cannot rely on CMake to find the Python libraries. We need to find them in the `build.py` script and pass the `PYTHON_LIBRARY` and `PYTHON_INCLUDE_DIR` parameters to CMake, like we already do for OS X. This is implemented by creating a `FindPythonLibraries` function for each supported platform: Linux, OS X, and Windows. For OS X, we are keeping the logic from `CustomPythonCmakeArgs` except that we are dropping support for system Python 2.6. For Linux, we need to use the `ldconfig` tool to find the system Python library on some distributions (e.g. Ubuntu) because it is not installed in its standard location. For Windows, it is straightforward. In all cases, we assume that the Python running the script is the one that will be used to build the library. This PR also fixes an important issue. When building the ycmd library with Clang support and linking it to a non-system Python libray, the dynamic linker will not be able to find the Python library: ```sh $ ldd ycm_core.so | grep python libpython3.4m.so.1.0 => not found $ objdump -x ycm_core.so | grep RPATH RPATH $ORIGIN ``` This is solved by setting the `CMAKE_INSTALL_RPATH_USE_LINK_PATH` option to true (see [this wiki page](https://cmake.org/Wiki/CMake_RPATH_handling) for details on the option): ``` > ldd ycm_core.so | grep python libpython3.4m.so.1.0 => /home/micbou/.pyenv/versions/3.4.4/lib/libpython3.4m.so.1.0 (0x00007f9fb6472000) > objdump -x ycm_core.so | grep RPATH RPATH $ORIGIN:/home/micbou/.pyenv/versions/3.4.4/lib ``` Closes #196 (we don't care about the Python interpreter when building the ycmd library). Fixes #479. <!-- Reviewable:start --> --- This change is [<img src="https://reviewable.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/valloric/ycmd/481) <!-- Reviewable:end -->
|
☔ The latest upstream changes (presumably #481) made this pull request unmergeable. Please resolve the merge conflicts. |
CustomPythonCmakeArgs forces CMake to pickup the environment python libraries and include but the interpreter is still picked using PythonInterp which picks up the system default instead of the environment python. This cause issues with vitualenv.