Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed to install apex on Ubuntu 22.04 with 4090 GPU #1563

Closed
lihongzheng-nlp opened this issue Jan 15, 2023 · 5 comments
Closed

Failed to install apex on Ubuntu 22.04 with 4090 GPU #1563

lihongzheng-nlp opened this issue Jan 15, 2023 · 5 comments

Comments

@lihongzheng-nlp
Copy link

lihongzheng-nlp commented Jan 15, 2023 •
edited
Loading

I'm trying to install apex as one of the dependencies of training OpenNMT model with the following codes from:

git clone https://github.com/NVIDIA/apex

cd apex

pip3 install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" --global-option="--deprecated_fused_adam" --global-option="--xentropy" --global-option="--fast_multihead_attn" ./

but it always failed. Here is the installation log:

WARNING: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
Using pip 22.2.2 from /home/lihongzheng/anaconda3/lib/python3.9/site-packages/pip (python 3.9)
Processing /media/lihongzheng/data/apex
  Running command python setup.py egg_info
 
 torch.__version__  = 1.13.1

  running egg_info
  creating /tmp/pip-pip-egg-info-ej0rc4c0/apex.egg-info
  writing /tmp/pip-pip-egg-info-ej0rc4c0/apex.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-pip-egg-info-ej0rc4c0/apex.egg-info/dependency_links.txt
  writing requirements to /tmp/pip-pip-egg-info-ej0rc4c0/apex.egg-info/requires.txt
  writing top-level names to /tmp/pip-pip-egg-info-ej0rc4c0/apex.egg-info/top_level.txt
  writing manifest file '/tmp/pip-pip-egg-info-ej0rc4c0/apex.egg-info/SOURCES.txt'
  reading manifest file '/tmp/pip-pip-egg-info-ej0rc4c0/apex.egg-info/SOURCES.txt'
  adding license file 'LICENSE'
  writing manifest file '/tmp/pip-pip-egg-info-ej0rc4c0/apex.egg-info/SOURCES.txt'
  Preparing metadata (setup.py) ... done
Requirement already satisfied: packaging>20.6 in /home/lihongzheng/anaconda3/lib/python3.9/site-packages (from apex==0.1) (21.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /home/lihongzheng/anaconda3/lib/python3.9/site-packages (from packaging>20.6->apex==0.1) (3.0.9)
Skipping wheel build for apex, due to binaries being disabled for it.
Installing collected packages: apex
  Running command Running setup.py install for apex


  torch.__version__  = 1.13.1



  Compiling cuda extensions with
  nvcc: NVIDIA (R) Cuda compiler driver
  Copyright (c) 2005-2022 NVIDIA Corporation
  Built on Wed_Jun__8_16:49:14_PDT_2022
  Cuda compilation tools, release 11.7, V11.7.99
  Build cuda_11.7.r11.7/compiler.31442593_0
  from /home/lihongzheng/anaconda3/bin

  running install
  /home/lihongzheng/anaconda3/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
    warnings.warn(
  running build
  running build_py
  running build_ext
  /home/lihongzheng/anaconda3/lib/python3.9/site-packages/torch/utils/cpp_extension.py:476: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
    warnings.warn(msg.format('we could not find ninja.'))
  building 'scaled_upper_triang_masked_softmax_cuda' extension
  gcc -pthread -B /home/lihongzheng/anaconda3/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /home/lihongzheng/anaconda3/include -I/home/lihongzheng/anaconda3/include -fPIC -O2 -isystem /home/lihongzheng/anaconda3/include -fPIC -I/media/lihongzheng/data/apex/csrc -I/home/lihongzheng/anaconda3/lib/python3.9/site-packages/torch/include -I/home/lihongzheng/anaconda3/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/home/lihongzheng/anaconda3/lib/python3.9/site-packages/torch/include/TH -I/home/lihongzheng/anaconda3/lib/python3.9/site-packages/torch/include/THC -I/home/lihongzheng/anaconda3/include -I/home/lihongzheng/anaconda3/include/python3.9 -c csrc/megatron/scaled_upper_triang_masked_softmax.cpp -o build/temp.linux-x86_64-cpython-39/csrc/megatron/scaled_upper_triang_masked_softmax.o -O3 -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -DTORCH_EXTENSION_NAME=scaled_upper_triang_masked_softmax_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
  /home/lihongzheng/anaconda3/bin/nvcc -I/media/lihongzheng/data/apex/csrc -I/home/lihongzheng/anaconda3/lib/python3.9/site-packages/torch/include -I/home/lihongzheng/anaconda3/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/home/lihongzheng/anaconda3/lib/python3.9/site-packages/torch/include/TH -I/home/lihongzheng/anaconda3/lib/python3.9/site-packages/torch/include/THC -I/home/lihongzheng/anaconda3/include -I/home/lihongzheng/anaconda3/include/python3.9 -c csrc/megatron/scaled_upper_triang_masked_softmax_cuda.cu -o build/temp.linux-x86_64-cpython-39/csrc/megatron/scaled_upper_triang_masked_softmax_cuda.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -O3 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 --threads 4 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -DTORCH_EXTENSION_NAME=scaled_upper_triang_masked_softmax_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++14
  csrc/megatron/scaled_upper_triang_masked_softmax_cuda.cu:21:10: fatal error: cuda_profiler_api.h: No such file or directory
     21 | #include <cuda_profiler_api.h>
        |          ^~~~~~~~~~~~~~~~~~~~~
  compilation terminated.
  csrc/megatron/scaled_upper_triang_masked_softmax_cuda.cu:21:10: fatal error: cuda_profiler_api.h: No such file or directory
     21 | #include <cuda_profiler_api.h>
        |          ^~~~~~~~~~~~~~~~~~~~~
  compilation terminated.
  error: command '/home/lihongzheng/anaconda3/bin/nvcc' failed with exit code 255
  error: subprocess-exited-with-error

  × Running setup.py install for apex did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /home/lihongzheng/anaconda3/bin/python -u -c '
  exec(compile('"'"''"'"''"'"'
# This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
  #
  # - It imports setuptools before invoking setup.py, to enable projects that directly
  #   import from `distutils.core` to work with newer packaging standards.
  # - It provides a clear error message when setuptools is not installed.
  # - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
  #   setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
  #     manifest_maker: standard file '"'"'-c'"'"' not found".
  # - It generates a shim setup.py, for handling setup.cfg-only projects.
  import os, sys, tokenize

  try:
      import setuptools
  except ImportError as error:
      print(
          "ERROR: Can not execute `setup.py` since setuptools is not available in "
          "the build environment.",
          file=sys.stderr,
      )
      sys.exit(1)

  __file__ = %r
  sys.argv[0] = __file__

  if os.path.exists(__file__):
      filename = __file__
      with tokenize.open(__file__) as f:
          setup_py_code = f.read()
  else:
      filename = "<auto-generated setuptools caller>"
      setup_py_code = "from setuptools import setup; setup()"

  exec(compile(setup_py_code, filename, "exec"))
  '"'"''"'"''"'"' % ('"'"'/media/lihongzheng/data/apex/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' --cpp_ext --cuda_ext --deprecated_fused_adam --xentropy --fast_multihead_attn install --record /tmp/pip-record-v3ch_xts/install-record.txt --single-version-externally-managed --compile --install-headers /home/lihongzheng/anaconda3/include/python3.9/apex
  cwd: /media/lihongzheng/data/apex/
  Running setup.py install for apex ... error
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> apex

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.`

Here's the basic information of my Ubuntu server:

  • Ubuntu 22.04 with two 4090 GPU.
  • CUDA 11.7. nvidia-smi shows that NVIDIA-SMI 525.60.11 Driver Version: 525.60.11 CUDA Version: 12.0 in the original setting, but I install 11.7 later, I'm not sure whether this will have impacts on the apex?
  • Python 3.9
  • Pytorch 1.13.1

I found some similar installation problems have been reported in the issues, and I tried many times according to guides, but always failed. Anyone can help me solve the problem? Thank you guys very much!
@crcrpar
Copy link
Collaborator

crcrpar commented Jan 15, 2023

Could you confirm whether or not the header file of cuda_profiler_api.h exists in a certain directory?

@lihongzheng-nlp
Copy link
Author

lihongzheng-nlp commented Jan 15, 2023 •
edited
Loading

Could you confirm whether or not the header file of cuda_profiler_api.h exists in a certain directory?

@crcrpar thanks for your response. I searched for the file in the whole machine with find . -name cuda*.h, but there's no filecuda_profiler_api.h.

`./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cudaVDPAU.h

./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cuda_awbarrier_helpers.h

./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cuda_gl_interop.h

./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cudaEGL.h

./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cuda_fp16.h

./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cuda_awbarrier.h

./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cuda_surface_types.h

./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cudart_platform.h

./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cuda_pipeline.h

./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cuda_device_runtime_api.h

./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cuda_texture_types.h

./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cudaEGLTypedefs.h

./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cuda_pipeline_primitives.h

./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cuda_occupancy.h

./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cuda_awbarrier_primitives.h

./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cuda_bf16.h

./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cudaProfilerTypedefs.h

./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cudaGL.h

./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cuda_pipeline_helpers.h

./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cudaGLTypedefs.h
./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cudaVDPAUTypedefs.h
./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cuda.h
./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cuda_runtime.h
./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cuda_egl_interop.h
./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cuda_vdpau_interop.h
./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cuda_runtime_api.h
./anaconda3/pkgs/cuda-cudart-dev-11.7.99-0/include/cudaTypedefs.h
./anaconda3/pkgs/libcufft-dev-10.7.2.124-h98a8f43_0/include/cudalibxt.h
./anaconda3/pkgs/pytorch-1.13.1-py3.9_cuda11.7_cudnn8.5.0_0/lib/python3.9/site-packages/torch/include/c10/cuda/impl/cuda_cmake_macros.h
./anaconda3/pkgs/pytorch-1.13.1-py3.9_cuda11.7_cudnn8.5.0_0/lib/python3.9/site-packages/torch/include/torch/csrc/jit/passes/cuda_graph_fuser.h
./anaconda3/pkgs/pytorch-1.13.1-py3.9_cuda11.7_cudnn8.5.0_0/lib/python3.9/site-packages/torch/include/torch/csrc/jit/tensorexpr/cuda_random.h
./anaconda3/pkgs/pytorch-1.13.1-py3.9_cuda11.7_cudnn8.5.0_0/lib/python3.9/site-packages/torch/include/torch/csrc/jit/tensorexpr/cuda_codegen.h
./anaconda3/pkgs/pytorch-1.13.1-py3.9_cuda11.7_cudnn8.5.0_0/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/cuda.h
./anaconda3/pkgs/pytorch-1.13.1-py3.9_cuda11.7_cudnn8.5.0_0/lib/python3.9/site-packages/torch/include/torch/csrc/utils/cuda_enabled.h
./anaconda3/pkgs/pytorch-1.13.1-py3.9_cuda11.7_cudnn8.5.0_0/lib/python3.9/site-packages/torch/include/torch/csrc/utils/cuda_lazy_init.h
./anaconda3/pkgs/cuda-nvcc-11.7.99-0/include/crt/cudacc_ext.h
./anaconda3/pkgs/cuda-cupti-11.7.101-0/include/cuda_stdint.h
./anaconda3/pkgs/cuda-gdb-12.0.90-0/extras/Debugger/include/cudadebugger.h
./anaconda3/pkgs/cuda-gdb-12.0.90-0/extras/Debugger/include/cudacoredump.h
./anaconda3/pkgs/cuda-gdb-12.0.90-0/extras/Debugger/include/cuda_stdint.h
./anaconda3/include/cudaVDPAU.h
./anaconda3/include/cuda_awbarrier_helpers.h
./anaconda3/include/cuda_gl_interop.h
./anaconda3/include/cudaEGL.h
./anaconda3/include/cudalibxt.h
./anaconda3/include/cuda_fp16.h
./anaconda3/include/cuda_awbarrier.h
./anaconda3/include/cuda_surface_types.h
./anaconda3/include/cudart_platform.h
./anaconda3/include/cuda_pipeline.h
./anaconda3/include/cuda_device_runtime_api.h
./anaconda3/include/cuda_texture_types.h
./anaconda3/include/cudaEGLTypedefs.h
./anaconda3/include/cuda_pipeline_primitives.h
./anaconda3/include/cuda_occupancy.h
./anaconda3/include/cuda_awbarrier_primitives.h
./anaconda3/include/cuda_bf16.h
./anaconda3/include/cudaProfilerTypedefs.h
./anaconda3/include/cudaGL.h
./anaconda3/include/cuda_pipeline_helpers.h
./anaconda3/include/cudaGLTypedefs.h
./anaconda3/include/cudaVDPAUTypedefs.h
./anaconda3/include/cuda.h
./anaconda3/include/cuda_runtime.h
./anaconda3/include/cuda_egl_interop.h
./anaconda3/include/cuda_stdint.h
./anaconda3/include/cuda_vdpau_interop.h
./anaconda3/include/cuda_runtime_api.h
./anaconda3/include/crt/cudacc_ext.h
./anaconda3/include/cudaTypedefs.h
./anaconda3/lib/python3.9/site-packages/torch/include/c10/cuda/impl/cuda_cmake_macros.h
./anaconda3/lib/python3.9/site-packages/torch/include/torch/csrc/jit/passes/cuda_graph_fuser.h
./anaconda3/lib/python3.9/site-packages/torch/include/torch/csrc/jit/tensorexpr/cuda_random.h
./anaconda3/lib/python3.9/site-packages/torch/include/torch/csrc/jit/tensorexpr/cuda_codegen.h
./anaconda3/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/cuda.h
./anaconda3/lib/python3.9/site-packages/torch/include/torch/csrc/utils/cuda_enabled.h
./anaconda3/lib/python3.9/site-packages/torch/include/torch/csrc/utils/cuda_lazy_init.h
./anaconda3/extras/Debugger/include/cudadebugger.h
./anaconda3/extras/Debugger/include/cudacoredump.h
./anaconda3/extras/Debugger/include/cuda_stdint.h`

I found a cuda_profiler_api.h file here, should I copy it to a certain directory in my machine?

@crcrpar
Copy link
Collaborator

crcrpar commented Jan 15, 2023

Hmm, I can’t tell the compatibility of the header you found. the missing header should come when installing cuda…

@ptrblck
Copy link
Collaborator

ptrblck commented Jan 26, 2023

Based on the output it seems you are trying to use the CUDA compiler from conda:

 Compiling cuda extensions with
  nvcc: NVIDIA (R) Cuda compiler driver
  Copyright (c) 2005-2022 NVIDIA Corporation
  Built on Wed_Jun__8_16:49:14_PDT_2022
  Cuda compilation tools, release 11.7, V11.7.99
  Build cuda_11.7.r11.7/compiler.31442593_0
  from /home/lihongzheng/anaconda3/bin

Note that you would need a locally installed CUDA toolkit as I don't know if this nvcc conda binary would work, but since you are already seeing missing header files, I would assume it's not working properly.

Let us know if you still have trouble building from source after installing the CUDA toolkit locally and reopen if needed.

@ptrblck ptrblck closed this as completed Jan 26, 2023
@akmalkadi
Copy link

I have the same issue, how to install the CUDA toolkit locally

@kevingreenman kevingreenman mentioned this issue Jul 25, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@akmalkadi @ptrblck @crcrpar @lihongzheng-nlp