Cudatoolkit v10.1 #16

andersy005 · 2019-08-24T00:20:09Z

Checklist

Used a fork of the feedstock to propose changes
Bumped the build number (if the version is unchanged)
Reset the build number to 0 (if the version changed)
Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
Ensured the license file is being packaged.

andersy005 · 2019-08-24T00:20:26Z

Addresses #15

andersy005 · 2019-08-24T00:20:41Z

@conda-forge-admin, please rerender

conda-forge-linter · 2019-08-24T00:23:38Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

conda-forge-linter · 2019-08-24T00:24:37Z

Hi! This is the friendly automated conda-forge-webservice.

I tried to re-render for you but ran into some issues, please ping conda-forge/core for further assistance.

recipe/scripts/activate.d/cudatoolkit-dev-activate.sh

andersy005 · 2019-08-25T23:52:15Z

Update

Everything seems to be working just fine on my local machine (ubuntu 18.04). However, due an old OS (Centos 6) used by the container, there seems to be issues with old glibc version.

++ conda info

     active environment : base
    active env location : /opt/conda
            shell level : 1
       user config file : /home/conda/.condarc
 populated config files : /home/conda/.condarc
          conda version : 4.7.11
    conda-build version : 3.18.9
         python version : 3.7.3.final.0
       virtual packages : 
       base environment : /opt/conda  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/conda/pkgs
                          /home/conda/.conda/pkgs
       envs directories : /opt/conda/envs
                          /home/conda/.conda/envs
               platform : linux-64
             user-agent : conda/4.7.11 requests/2.22.0 CPython/3.7.3 Linux/4.15.0-1052-azure centos/6.10 glibc/2.12
                UID:GID : 1001:1001
             netrc file : None
           offline mode : False

stdout: Running Post installation
downloading https://developer.download.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda_10.1.243_418.87.00_linux.run to /home/conda/feedstock_root/build_artifacts/cudatoolkit-dev_1566775645864/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/pkgs/cudatoolkit-dev/cuda_10.1.243_418.87.00_linux.run
Extracting on Linux

stderr: ./cuda-installer: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by ./cuda-installer)
./cuda-installer: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by ./cuda-installer)
./cuda-installer: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by ./cuda-installer)
Traceback (most recent call last):
  File "/home/conda/feedstock_root/build_artifacts/cudatoolkit-dev_1566775645864/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/bin/cudatoolkit-dev-post-install.py", line 257, in <module>
    _main()
  File "/home/conda/feedstock_root/build_artifacts/cudatoolkit-dev_1566775645864/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/bin/cudatoolkit-dev-post-install.py", line 253, in _main
    extractor.extract()
  File "/home/conda/feedstock_root/build_artifacts/cudatoolkit-dev_1566775645864/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/bin/cudatoolkit-dev-post-install.py", line 142, in extract
    raise exc
  File "/home/conda/feedstock_root/build_artifacts/cudatoolkit-dev_1566775645864/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/bin/cudatoolkit-dev-post-install.py", line 138, in extract
    subprocess.check_call(cmd, env=dict(DISPLAY=""))
  File "/home/conda/feedstock_root/build_artifacts/cudatoolkit-dev_1566775645864/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '[PosixPath('/home/conda/feedstock_root/build_artifacts/cudatoolkit-dev_1566775645864/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/pkgs/cudatoolkit-dev/cuda_10.1.243_418.87.00_linux.run'), '--extract=/tmp/tmpd4rou7kq']' returned non-zero exit status 1.

return code: 1

pearu · 2019-08-26T07:53:18Z

recipe/meta.yaml

-    - test -f $PREFIX/bin/cudatoolkit-dev-extra-args.json  # [unix]
-    - test -d $PREFIX/scripts/activate.d  # [unix]
-    - test -d $PREFIX/scripts/deactivate.d  # [unix]
+    - numba


The run_test.py seems to be equivalent to running nvvm --version. So, instead of making the test numba (and numpy) dependent, I'd suggest:

test: commands: - nvcc --version

pearu · 2019-08-27T22:36:19Z

Thanks @andersy005 , for the access!

Regarding the remaining GLIBC issue, I found this https://devtalk.nvidia.com/default/topic/1050512/cuda-setup-and-installation/error-during-installation-version-glibc_2-14-not-found/

I'll continue looking at the solution that would not require installing glibc..

pearu · 2019-08-28T12:39:00Z

OK, it looks like we are in a dead-end with this PR: cuda installer requires glibc 2.14 but azure CI build machine has glibc 2.12.

On the other hand, the built cudatoolkit-dev conda package should be usable from users machines (that have glibc 2.14 or greater). So, one solution would be to disable testing cudatoolkit-dev installation process on azure machines. This also means that cudatoolkit-dev could not be used as a dependency for packages that are built on azure CI (like omniscidb-cuda). Although, locally, in a user machine the cudatoolkit-dev dependency should work out fine.

Another solution is to downgrade cuda to a version where its installer works with glibc 2.12. v10.0 seems to be ok. This solution would hold back upgrading cuda versions until azure CI machines will have some newer glibc version installed.

@andersy005 @scopatz Thoughts?

andersy005 · 2019-08-28T19:07:09Z

@pearu,

I am in favor of merging this as is even though it will cause other packages that use cudatoolkit-dev as a dependency to fail the CI checks for the time being. It would be nice to know whether

there's a plan of upgrading the base docker image's OS (to address the older glibc version issue) anytime soon?

and/or

there's another way of addressing the glibc version issue ourselves?

pearu · 2019-08-28T19:57:42Z

Re 1: I don't know, I suspect not soon, otherwise, the CI instances would not be so old..
Re 2: one option is to create glibc conda package or build glibc within the post-installation script. Here are the basic instructions: https://www.jertype.com/posts/upgrading-glibc/

pearu · 2019-10-01T17:40:19Z

@isuruf thanks for the rhel6 hint!
@andersy005 please review.

andersy005

Thanks @pearu, @isuruf! Glad to see this brought to completion.
Everything looks good to me. I can merge this soon unless there are any objection.

xmnlab

LGTM! thanks everyone for working on that!

xmnlab · 2019-10-01T19:42:32Z

as @andersy005 also approved that, I am merging this PR.

Remove conda dependencies

893fd53

andersy005 requested a review from xmnlab as a code owner August 24, 2019 00:20

andersy005 added 4 commits August 23, 2019 18:38

add hashsum_file

6f8690f

Fix typo

c5cf6ed

Remove conda and use wget for downloading files

d75acd2

Update cudatoolkit version

7d46519

andersy005 changed the title ~~Remove conda dependencies~~ Cudatoolkit v10.1 Aug 25, 2019

andersy005 added 3 commits August 25, 2019 05:58

Update url

940e838

Code refactoring

cd995b4

Add refactored scripts

8aecd43

pearu reviewed Aug 25, 2019

View reviewed changes

recipe/scripts/activate.d/cudatoolkit-dev-activate.sh Show resolved Hide resolved

pearu mentioned this pull request Aug 25, 2019

Add omniscidb-cuda conda-forge/staged-recipes#9225

Merged

6 tasks

andersy005 added 4 commits August 25, 2019 16:48

fix styling

c7196f9

Update OSX extractor

2caf4f7

Catch errors (if any) in the activate script

bfbbdc7

Attempt at fixing OSX permission issues

d2ad916

andersy005 added 8 commits August 25, 2019 18:56

use non-shared directory

373a1b4

Another attempt at fixing permission issue w/ shared dir

79e486f

Set numeric owner

290a7a0

Use extract instead of extractall

fc51fc9

remove tarfile and fnmatch

eddaeb4

FIx terminating ";"

4114b2d

Missing comma

8784df4

Fix OSX mount

c354ab9

pearu reviewed Aug 26, 2019

View reviewed changes

pearu added 10 commits August 28, 2019 11:12

Use subprocess.run for running the installer.

8489009

Debug the environment.

3f32223

Try running apt-get update

f9c6020

Try update glibc

f9e502e

Try c compiler dependency that should include newer glibc

d510542

Enable conda libc.so.6

35eef7b

Add LD_LIBRARY_PATH to env

d471b51

Fix typo and add CONDA_PREFIX/bin

1780a5d

Disable LD_PRELOAD

584d4a8

Clean up.

54e288c

andersy005 mentioned this pull request Oct 1, 2019

cudatoolkit-dev installation pops up a terminal window #7

Closed

pearu mentioned this pull request Oct 1, 2019

GPU Support conda-forge/pytorch-cpu-feedstock#7

Closed

isuruf and others added 5 commits October 1, 2019 04:49

Update cudatoolkit-dev-post-install.py

b3fa3b9

Debug glibc issue

92e62ad

Debug glibc issue (2)

272a6a3

Debug glibc issue (3)

fbed9a8

Fall back to using cuda-linux..run installer.

7d35f98

andersy005 commented Oct 1, 2019

View reviewed changes

xmnlab approved these changes Oct 1, 2019

View reviewed changes

xmnlab merged commit b943d10 into conda-forge:master Oct 1, 2019

This was referenced Oct 1, 2019

Update Linux installer to 10.1 #8

Closed

cudatoolkit-dev installed within a conda-build requires conda package dependency #15

Closed

LinkError: post-link script failed for package conda-forge::cudatoolkit-dev-10.0-1 #9

Closed

andersy005 deleted the remove-conda-dep branch December 11, 2019 00:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cudatoolkit v10.1 #16

Cudatoolkit v10.1 #16

andersy005 commented Aug 24, 2019 •

edited

Loading

andersy005 commented Aug 24, 2019

andersy005 commented Aug 24, 2019

conda-forge-linter commented Aug 24, 2019

conda-forge-linter commented Aug 24, 2019

andersy005 commented Aug 25, 2019 •

edited

Loading

pearu Aug 26, 2019 •

edited

Loading

pearu commented Aug 27, 2019

pearu commented Aug 28, 2019

andersy005 commented Aug 28, 2019

pearu commented Aug 28, 2019

pearu commented Oct 1, 2019

andersy005 left a comment

xmnlab left a comment

xmnlab commented Oct 1, 2019

Cudatoolkit v10.1 #16

Cudatoolkit v10.1 #16

Conversation

andersy005 commented Aug 24, 2019 • edited Loading

andersy005 commented Aug 24, 2019

andersy005 commented Aug 24, 2019

conda-forge-linter commented Aug 24, 2019

conda-forge-linter commented Aug 24, 2019

andersy005 commented Aug 25, 2019 • edited Loading

pearu Aug 26, 2019 • edited Loading

Choose a reason for hiding this comment

pearu commented Aug 27, 2019

pearu commented Aug 28, 2019

andersy005 commented Aug 28, 2019

pearu commented Aug 28, 2019

pearu commented Oct 1, 2019

andersy005 left a comment

Choose a reason for hiding this comment

xmnlab left a comment

Choose a reason for hiding this comment

xmnlab commented Oct 1, 2019

andersy005 commented Aug 24, 2019 •

edited

Loading

andersy005 commented Aug 25, 2019 •

edited

Loading

pearu Aug 26, 2019 •

edited

Loading