[Installation]: Compiling VLLM for cpu only.

[Zibri]

To ease the bug report I prepared a google colab notebook:

https://colab.research.google.com/drive/1wd29UiknYI3r-8H5Inco9IkWYcuZAVNR?usp=sharing

I followed the instructions in the documentation and this is what I got:

-- The CXX compiler identification is GNU 12.3.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Build type: RelWithDebInfo
-- Target device: cpu
-- Found Python: /usr/bin/python3 (found version "3.10.12") found components: Interpreter Development.Module 
-- Found python matching: /usr/bin/python3.
CMake Warning at /usr/local/lib/python3.10/dist-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
  static library kineto_LIBRARY-NOTFOUND not found.
Call Stack (most recent call first):
  /usr/local/lib/python3.10/dist-packages/torch/share/cmake/Torch/TorchConfig.cmake:127 (append_torchlib_if_found)
  CMakeLists.txt:67 (find_package)


-- Found Torch: /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch.so  
CMake Error at cmake/cpu_extension.cmake:57 (message):
  vLLM CPU backend requires AVX512 ISA support.
Call Stack (most recent call first):
  CMakeLists.txt:88 (include)


-- Configuring incomplete, errors occurred!
Traceback (most recent call last):
  File "/content/vllm/setup.py", line 401, in <module>
    setup(
  File "/usr/local/lib/python3.10/dist-packages/setuptools/__init__.py", line 107, in setup
    return distutils.core.setup(**attrs)
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 969, in run_commands
    self.run_command(cmd)
  File "/usr/local/lib/python3.10/dist-packages/setuptools/dist.py", line 1244, in run_command
    super().run_command(command)
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python3.10/dist-packages/setuptools/command/install.py", line 80, in run
    self.do_egg_install()
  File "/usr/local/lib/python3.10/dist-packages/setuptools/command/install.py", line 129, in do_egg_install
    self.run_command('bdist_egg')
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/usr/local/lib/python3.10/dist-packages/setuptools/dist.py", line 1244, in run_command
    super().run_command(command)
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python3.10/dist-packages/setuptools/command/bdist_egg.py", line 164, in run
    cmd = self.call_command('install_lib', warn_dir=0)
  File "/usr/local/lib/python3.10/dist-packages/setuptools/command/bdist_egg.py", line 150, in call_command
    self.run_command(cmdname)
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/usr/local/lib/python3.10/dist-packages/setuptools/dist.py", line 1244, in run_command
    super().run_command(command)
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python3.10/dist-packages/setuptools/command/install_lib.py", line 11, in run
    self.build()
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/command/install_lib.py", line 111, in build
    self.run_command('build_ext')
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/usr/local/lib/python3.10/dist-packages/setuptools/dist.py", line 1244, in run_command
    super().run_command(command)
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python3.10/dist-packages/setuptools/command/build_ext.py", line 84, in run
    _build_ext.run(self)
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
    self.build_extensions()
  File "/content/vllm/setup.py", line 193, in build_extensions
    self.configure(ext)
  File "/content/vllm/setup.py", line 175, in configure
    subprocess.check_call(
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/content/vllm', '-G', 'Ninja', '-DCMAKE_BUILD_TYPE=RelWithDebInfo', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/content/vllm/build/lib.linux-x86_64-cpython-310/vllm', '-DCMAKE_ARCHIVE_OUTPUT_DIRECTORY=build/temp.linux-x86_64-cpython-310', '-DVLLM_TARGET_DEVICE=cpu', '-DVLLM_PYTHON_EXECUTABLE=/usr/bin/python3', '-DCMAKE_JOB_POOL_COMPILE:STRING=compile', '-DCMAKE_JOB_POOLS:STRING=compile=2']' returned non-zero exit status 1.
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
[<ipython-input-3-80f4f9c86e68>](https://localhost:8080/#) in <cell line: 1>()
----> 1 get_ipython().run_cell_magic('shell', '', 'git clone [https://github.com/vllm-project/vllm\ncd](https://github.com/vllm-project/vllm/ncd) vllm\npip install --upgrade pip\npip install wheel packaging ninja setuptools>=49.4.0 numpy\npip install -v -r requirements-cpu.txt --extra-index-url [https://download.pytorch.org/whl/cpu\nVLLM_TARGET_DEVICE=cpu](https://download.pytorch.org/whl/cpu/nVLLM_TARGET_DEVICE=cpu) python setup.py install\n')

[0wwafa]

Same problem here.

[zifeitong]

vLLM CPU backend requires AVX512 ISA support.

unfortunately AVX512 is not support by the Xeon CPU used in colab. You can verify by checking /proc/cpuinfo.

[0wwafa]

can you use avx2 or other cpu features like llama.cpp does?

[zifeitong]

Please see #5452

[0wwafa]

Please see #5452

it seems it's not going anywhere..

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!