Building MXNet 1.9 from sources breaks mxnet.libinfo.find_include_path()

[Gao-HaoYuan]

Description

environment：
python 3.8
ubuntu 18.04
gcc 7.5 g++7.5
mxnet1.9(build from source)
openmpi 4.0
horovod 0.24

instructions
HOROVOD_WITHOUT_TENSORFLOW=1 HOROVOD_WITHOUT_PYTORCH=1 HOROVOD_WITH_MXNET=1 pip install dist/horovod-0.24.1.tar.gz

HOROVOD_WITH_MXNET=1 pip install --no-cache-dir horovod[MXNET]

Horovod cannot be installed or compile

Error Message

Traceback (most recent call last):
File "", line 1, in
File "/root/miniconda3/lib/python3.8/site-packages/mxnet-1.9.0-py3.8.egg/mxnet/libinfo.py", line 108, in find_include_path
raise RuntimeError('Cannot find the MXNet include path in either ' + pip_incl_path +
RuntimeError: Cannot find the MXNet include path in either /root/miniconda3/lib/python3.8/site-packages/mxnet-1.9.0-py3.8.egg/mxnet/include/ or /root/miniconda3/lib/python3.8/site-packages/mxnet-1.9.0-py3.8.egg/mxnet/../../include/

  CMake Error at /tmp/horovod-cmake-tmpaicqjfxp/cmake/data/share/cmake-3.13/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
    Could NOT find Mxnet (missing: Mxnet_LIBRARIES) (Required is at least
    version "1.4.0")
  Call Stack (most recent call first):
    /tmp/horovod-cmake-tmpaicqjfxp/cmake/data/share/cmake-3.13/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
    cmake/Modules/FindMxnet.cmake:65 (find_package_handle_standard_args)
    horovod/mxnet/CMakeLists.txt:12 (find_package)


  -- Configuring incomplete, errors occurred!
  See also "/tmp/pip-req-build-sz62zlyv/build/temp.linux-x86_64-3.8/RelWithDebInfo/CMakeFiles/CMakeOutput.log".
  Traceback (most recent call last):
    File "<string>", line 2, in <module>
    File "<pip-setuptools-caller>", line 34, in <module>
    File "/tmp/pip-req-build-sz62zlyv/setup.py", line 209, in <module>
      setup(name='horovod',
    File "/root/miniconda3/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
      return distutils.core.setup(**attrs)
    File "/root/miniconda3/lib/python3.8/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/root/miniconda3/lib/python3.8/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/root/miniconda3/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/root/miniconda3/lib/python3.8/site-packages/setuptools/command/install.py", line 61, in run
      return orig.install.run(self)
    File "/root/miniconda3/lib/python3.8/distutils/command/install.py", line 545, in run
      self.run_command('build')
    File "/root/miniconda3/lib/python3.8/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/root/miniconda3/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/root/miniconda3/lib/python3.8/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/root/miniconda3/lib/python3.8/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/root/miniconda3/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/root/miniconda3/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run
      _build_ext.run(self)
    File "/root/miniconda3/lib/python3.8/distutils/command/build_ext.py", line 340, in run
      self.build_extensions()
    File "/tmp/pip-req-build-sz62zlyv/setup.py", line 144, in build_extensions
      subprocess.check_call(command, cwd=cmake_build_dir)
    File "/root/miniconda3/lib/python3.8/subprocess.py", line 364, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/tmp/horovod-cmake-tmpaicqjfxp/bin/run_cmake', '/tmp/pip-req-build-sz62zlyv', '-DCMAKE_BUILD_TYPE=RelWithDebInfo', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY_RELWITHDEBINFO=/tmp/pip-req-build-sz62zlyv/build/lib.linux-x86_64-3.8', '-DPYTHON_EXECUTABLE:FILEPATH=/root/miniconda3/bin/python']' returned non-zero exit status 1.
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
Rolling back uninstall of horovod
Moving to /root/miniconda3/bin/horovodrun
from /tmp/pip-uninstall-ddcttzju/horovodrun
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> horovod

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

To Reproduce

HOROVOD_WITHOUT_TENSORFLOW=1 HOROVOD_WITHOUT_PYTORCH=1 HOROVOD_WITH_MXNET=1 pip install dist/horovod-0.24.1.tar.gz

HOROVOD_WITH_MXNET=1 pip install --no-cache-dir horovod[MXNET]

Steps to reproduce

(Paste the commands you ran that produced the error.)

1. compile the mxnet 1.9
   apt-get update
   sudo apt-get install -y build-essential libopenblas-dev libopencv-dev graphviz
   make -j8
   sudo python setup.py install
   export PYTHONPATH="/root/mxnet/python:$PYTHONPATH" and source ~/.bashrc
2. conda install openmpi (4.0.2)
3. HOROVOD_WITH_MXNET=1 pip install --no-cache-dir horovod[MXNET]

What have you tried to solve it?

1. compile the horovod
2. compile the openmpi

Environment

python 3.8
ubuntu 18.04
gcc 7.5 g++7.5
mxnet1.9(build from source)
openmpi 4.0
horovod 0.24

Environment Information

# Paste the diagnose.py command output here

[EnricoMi]

I could reproduce this in a ubuntu:20.04 docker container. After installing MXNet from sources via setup-utils/install-mxnet-ubuntu-python.sh, running this Python code fails:

import mxnet as mx
print(mx.libinfo.find_include_path())

The error is:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/mxnet-1.9.0-py3.8.egg/mxnet/libinfo.py", line 108, in find_include_path
    raise RuntimeError('Cannot find the MXNet include path in either ' + pip_incl_path +
RuntimeError: Cannot find the MXNet include path in either /usr/local/lib/python3.8/dist-packages/mxnet-1.9.0-py3.8.egg/mxnet/include/ or /usr/local/lib/python3.8/dist-packages/mxnet-1.9.0-py3.8.egg/mxnet/../../include/

[Gao-HaoYuan]

I have solved this problem， I made the following attempts:

i add KNOWN_CUDA_ARCH=86 to Makefile and replace c++11 with c++14 ， because my GPU is A4000.

i fix the warning as the commit #17718.

i update cmake to 3.13, i think this is the most important

add export PYTHONPATH="/root/mxnet/python:$PYTHONPATH" to .bashrc and source ~/.bashrc