Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

try gpu on linux #22

Merged
merged 20 commits into from
Dec 17, 2020
Merged

try gpu on linux #22

merged 20 commits into from
Dec 17, 2020

Conversation

isuruf
Copy link
Member

@isuruf isuruf commented Dec 11, 2020

Checklist

  • Used a fork of the feedstock to propose changes
  • Bumped the build number (if the version is unchanged)
  • Reset the build number to 0 (if the version changed)
  • Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
  • Ensured the license file is being packaged.

@isuruf isuruf requested a review from sodre as a code owner December 11, 2020 18:27
@conda-forge-linter
Copy link

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@conda-forge-linter
Copy link

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe:

  • It looks like the 'pytorch-cpu' output doesn't have any tests.

- cffi
- future # [py==27]
outputs:
- name: pytorch
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@isuruf
Copy link
Member Author

isuruf commented Dec 12, 2020

@kkraus14, @jakirkham, can you test these on gpu hardware?

@hmaarrfk hmaarrfk mentioned this pull request Dec 14, 2020
@rgommers
Copy link

@isuruf what is the best way to install the packages built from this PR for offline testing?

@isuruf
Copy link
Member Author

isuruf commented Dec 15, 2020

CPU or the GPU packages? GPU packages currently time out, so you need to build it locally using build-locally.py

@rgommers
Copy link

GPU packages currently time out, so you need to build it locally using build-locally.py

GPU indeed. Okay, thanks - will try.

- magma # [cuda_compiler_version != "None"]
# other requirements
- python
- numpy

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seeing this build warning:

WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml.
Falling back to default numpy value of 1.11

That's probably not desired, should be 1.16?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think that warning is benign

@rgommers
Copy link

Running build_locally.py in an empty env ends in Cannot determine CUDA_HOME: cuda-gdb not in PATH for me. I wouldn't expect CUDA_HOME needing to be set by me, so I'm probably doing something else wrong - any idea?

...

commit b31f58de6fa8bbda5353b3c77d9be4914399724d
Author: Eli Uriegas <[email protected]>
Date:   Fri Jul 24 15:18:09 2020 -0700

    Revert "[release/1.6] .circleci: Don't use SCCACHE for windows release builds (#42024)"
    
    This reverts commit 994b37b36ef11c8e26464512e68ff5573ce81fe6.

==> /opt/conda/bin/git describe --tags --dirty <==

v1.6.0

==> /opt/conda/bin/git status <==

HEAD detached at b31f58de6fa
nothing to commit, working tree clean

Applying patch: /home/conda/recipe_root/fix_std_stdint.patch
Applying patch: /home/conda/recipe_root/fix_std_stdint.patch with args:
['-p1', '-i', '/tmp/tmphd5jjk68/fix_std_stdint.patch.native', '--binary']
checking file torch/csrc/DataLoader.cpp
patching file torch/csrc/DataLoader.cpp
Applying patch: /home/conda/recipe_root/cpp_extension.patch
Applying patch: /home/conda/recipe_root/cpp_extension.patch with args:
['-p1', '-i', '/tmp/tmpnlooig2y/cpp_extension.patch.native', '--binary']
checking file torch/utils/cpp_extension.py
patching file torch/utils/cpp_extension.py
Applying patch: /home/conda/recipe_root/remove_shared_linker_flag_override.patch
Applying patch: /home/conda/recipe_root/remove_shared_linker_flag_override.patch with args:
['-p1', '-i', '/tmp/tmpqe9ygd74/remove_shared_linker_flag_override.patch.native', '--binary']
checking file CMakeLists.txt
patching file CMakeLists.txt
Applying patch: /home/conda/recipe_root/nccl_socket.patch
Applying patch: /home/conda/recipe_root/nccl_socket.patch with args:
['-p0', '-i', '/tmp/tmpkoupb3i4/nccl_socket.patch.native', '--binary']
checking file third_party/nccl/nccl/src/include/socket.h
Hunk #1 succeeded at 22 with fuzz 2.
patching file third_party/nccl/nccl/src/include/socket.h
Hunk #1 succeeded at 22 with fuzz 2.
Applying patch: /home/conda/recipe_root/fix_dispatch_apply_auto.patch
Applying patch: /home/conda/recipe_root/fix_dispatch_apply_auto.patch with args:
['-p1', '-i', '/tmp/tmp0zzazehf/fix_dispatch_apply_auto.patch.native', '--binary']
checking file third_party/pthreadpool/src/gcd.c
patching file third_party/pthreadpool/src/gcd.c
Applying patch: /home/conda/recipe_root/fix_map_anonymous.patch
Applying patch: /home/conda/recipe_root/fix_map_anonymous.patch with args:
['-p1', '-i', '/tmp/tmp2y0mszfi/fix_map_anonymous.patch.native', '--binary']
checking file third_party/ideep/mkl-dnn/src/cpu/x64/xbyak/xbyak.h
patching file third_party/ideep/mkl-dnn/src/cpu/x64/xbyak/xbyak.h
Patch analysis gives:
[[ RA-MD1LOVE ]] - [[                                                    fix_std_stdint.patch ]]
[[ RA-MD1LOVE ]] - [[                                                     cpp_extension.patch ]]
[[ RA-MD1LOVE ]] - [[                                remove_shared_linker_flag_override.patch ]]
[[ RA-MD0LO-E ]] - [[                                                       nccl_socket.patch ]]
[[ RA-MD1LOVE ]] - [[                                           fix_dispatch_apply_auto.patch ]]
[[ RA-MD1LOVE ]] - [[                                                 fix_map_anonymous.patch ]]

Key:

R :: Reversible                       A :: Applicable
Y :: Build-prefix patch in use        M :: Minimal, non-amalgamated
D :: Dry-runnable                     N :: Patch level (1 is preferred)
L :: Patch level not-ambiguous        O :: Patch applies without offsets
V :: Patch applies without fuzz       E :: Patch applies without emitting to stderr

source tree in: /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/work
Packaging pytorch
INFO:conda_build.build:Packaging pytorch
INFO conda_build.build:build(2214): Packaging pytorch
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh


The following NEW packages will be INSTALLED:

    _libgcc_mutex:    0.1-conda_forge          conda-forge
    _openmp_mutex:    4.5-1_llvm               conda-forge
    brotlipy:         0.7.0-py38h8df0ef7_1001  conda-forge
    bzip2:            1.0.8-h7f98852_4         conda-forge
    c-ares:           1.17.1-h36c2ea0_0        conda-forge
    ca-certificates:  2020.12.5-ha878542_0     conda-forge
    certifi:          2020.12.5-py38h578d9bd_0 conda-forge
    cffi:             1.14.4-py38ha65f79e_1    conda-forge
    chardet:          3.0.4-py38h924ce5b_1008  conda-forge
    cmake:            3.19.1-h1f3970d_0        conda-forge
    cryptography:     3.3.1-py38h2b97feb_0     conda-forge
    cudatoolkit:      10.2.89-h8f6ccaa_6       conda-forge
    cudnn:            7.6.5-cuda10.2_0         defaults   
    curl:             7.71.1-he644dc0_8        conda-forge
    expat:            2.2.9-he1b5a44_2         conda-forge
    future:           0.18.2-py38h578d9bd_2    conda-forge
    gettext:          0.19.8.1-h0b5b191_1005   conda-forge
    git:              2.29.2-pl5320h78be0e9_1  conda-forge
    idna:             2.10-pyh9f0ad1d_0        conda-forge
    krb5:             1.17.2-h926e7f8_0        conda-forge
    ld_impl_linux-64: 2.35.1-hea4e1c9_1        conda-forge
    libblas:          3.8.0-21_mkl             conda-forge
    libcblas:         3.8.0-21_mkl             conda-forge
    libcurl:          7.71.1-hcdd3856_8        conda-forge
    libedit:          3.1.20191231-he28a2e2_2  conda-forge
    libev:            4.33-h516909a_1          conda-forge
    libffi:           3.3-h58526e2_2           conda-forge
    libgcc-ng:        9.3.0-h5dbcf3e_17        conda-forge
    libiconv:         1.16-h516909a_0          conda-forge
    liblapack:        3.8.0-21_mkl             conda-forge
    libnghttp2:       1.41.0-h8cfc5f6_2        conda-forge
    libssh2:          1.9.0-hab1572f_5         conda-forge
    libstdcxx-ng:     9.3.0-h2ae2ef3_17        conda-forge
    libuv:            1.40.0-hd18ef5c_0        conda-forge
    llvm-openmp:      11.0.0-hfc4b9b4_1        conda-forge
    magma:            2.5.2-h7384022_0         conda-forge
    mkl:              2020.4-h726a3e6_304      conda-forge
    mkl-devel:        2020.4-ha770c72_305      conda-forge
    mkl-include:      2020.4-h726a3e6_304      conda-forge
    nccl:             2.7.8.1-hc6a2c23_1       conda-forge
    ncurses:          6.2-h58526e2_4           conda-forge
    ninja:            1.10.2-h4bd325d_0        conda-forge
    numpy:            1.16.6-py38h18fd61f_0    conda-forge
    openssl:          1.1.1i-h7f98852_0        conda-forge
    pcre:             8.44-he1b5a44_0          conda-forge
    perl:             5.32.0-h36c2ea0_0        conda-forge
    pip:              20.3.3-pyhd8ed1ab_0      conda-forge
    pkg-config:       0.29.2-h36c2ea0_1008     conda-forge
    pycparser:        2.20-pyh9f0ad1d_2        conda-forge
    pyopenssl:        20.0.0-pyhd8ed1ab_0      conda-forge
    pysocks:          1.7.1-py38h924ce5b_2     conda-forge
    python:           3.8.6-hffdb5ce_1_cpython conda-forge
    python_abi:       3.8-1_cp38               conda-forge
    pyyaml:           5.3.1-py38h8df0ef7_1     conda-forge
    readline:         8.0-he28a2e2_2           conda-forge
    requests:         2.25.0-pyhd3deb0d_0      conda-forge
    rhash:            1.3.6-h516909a_1001      conda-forge
    setuptools:       49.6.0-py38h924ce5b_2    conda-forge
    six:              1.15.0-pyh9f0ad1d_0      conda-forge
    sqlite:           3.34.0-h74cdb3f_0        conda-forge
    tk:               8.6.10-h21135ba_1        conda-forge
    typing:           3.7.4.3-py38h32f6830_1   conda-forge
    urllib3:          1.25.11-py_0             conda-forge
    wheel:            0.36.2-pyhd3deb0d_0      conda-forge
    xz:               5.2.5-h516909a_1         conda-forge
    yaml:             0.2.5-h516909a_0         conda-forge
    zlib:             1.2.11-h516909a_1010     conda-forge

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... By downloading and using the CUDA Toolkit conda packages, you accept the terms and conditions of the CUDA End User License Agreement (EULA): https://docs.nvidia.com/cuda/eula/index.html

done
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env


The following NEW packages will be INSTALLED:

    _libgcc_mutex:            0.1-conda_forge         conda-forge
    _openmp_mutex:            4.5-1_gnu               conda-forge
    binutils_impl_linux-64:   2.35.1-h193b22a_1       conda-forge
    binutils_linux-64:        2.35-hc3fd857_29        conda-forge
    c-ares:                   1.17.1-h36c2ea0_0       conda-forge
    ca-certificates:          2020.12.5-ha878542_0    conda-forge
    curl:                     7.71.1-he644dc0_8       conda-forge
    expat:                    2.2.9-he1b5a44_2        conda-forge
    gcc_impl_linux-64:        7.5.0-hd9e1a51_17       conda-forge
    gcc_linux-64:             7.5.0-he2a3fca_29       conda-forge
    gettext:                  0.19.8.1-h0b5b191_1005  conda-forge
    git:                      2.29.2-pl5320h78be0e9_1 conda-forge
    gxx_impl_linux-64:        7.5.0-h7ea4de1_17       conda-forge
    gxx_linux-64:             7.5.0-h547f3ba_29       conda-forge
    kernel-headers_linux-64:  2.6.32-h77966d4_13      conda-forge
    krb5:                     1.17.2-h926e7f8_0       conda-forge
    ld_impl_linux-64:         2.35.1-hea4e1c9_1       conda-forge
    libcurl:                  7.71.1-hcdd3856_8       conda-forge
    libedit:                  3.1.20191231-he28a2e2_2 conda-forge
    libev:                    4.33-h516909a_1         conda-forge
    libffi:                   3.3-h58526e2_2          conda-forge
    libgcc-devel_linux-64:    7.5.0-h42c25f5_17       conda-forge
    libgcc-ng:                9.3.0-h5dbcf3e_17       conda-forge
    libgomp:                  9.3.0-h5dbcf3e_17       conda-forge
    libiconv:                 1.16-h516909a_0         conda-forge
    libnghttp2:               1.41.0-h8cfc5f6_2       conda-forge
    libssh2:                  1.9.0-hab1572f_5        conda-forge
    libstdcxx-devel_linux-64: 7.5.0-h4084dd6_17       conda-forge
    libstdcxx-ng:             9.3.0-h2ae2ef3_17       conda-forge
    ncurses:                  6.2-h58526e2_4          conda-forge
    nvcc_linux-64:            10.2-h1a5f58c_10        conda-forge
    openssl:                  1.1.1i-h7f98852_0       conda-forge
    patch:                    2.7.6-h516909a_1001     conda-forge
    pcre:                     8.44-he1b5a44_0         conda-forge
    perl:                     5.32.0-h36c2ea0_0       conda-forge
    sysroot_linux-64:         2.12-h77966d4_13        conda-forge
    tk:                       8.6.10-h21135ba_1       conda-forge
    zlib:                     1.2.11-h516909a_1010    conda-forge

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Packaging pytorch-1.6.0-cuda102py38h9f8c3ab_0
INFO:conda_build.build:Packaging pytorch-1.6.0-cuda102py38h9f8c3ab_0
INFO conda_build.build:bundle_conda(1454): Packaging pytorch-1.6.0-cuda102py38h9f8c3ab_0
INFO: activate-binutils_linux-64.sh made the following environmental changes:
+ADDR2LINE=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-addr2line
+AR=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-ar
+AS=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-as
+CXXFILT=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-c++filt
+ELFEDIT=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-elfedit
+GPROF=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-gprof
+LD_GOLD=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-ld.gold
+LD=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-ld
+NM=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-nm
+OBJCOPY=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-objcopy
+OBJDUMP=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-objdump
+RANLIB=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-ranlib
+READELF=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-readelf
+SIZE=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-size
+STRINGS=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-strings
+STRIP=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-strip
INFO: activate-gcc_linux-64.sh made the following environmental changes:
+build_alias=x86_64-conda-linux-gnu
-BUILD=x86_64-conda_cos6-linux-gnu
+BUILD=x86_64-conda-linux-gnu
+CC_FOR_BUILD=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-cc
+CC=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-cc
+CFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/work=/usr/local/src/conda/pytorch-1.6.0 -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh=/usr/local/src/conda-prefix
+CMAKE_ARGS=-DCMAKE_LINKER=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-ld -DCMAKE_STRIP=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-strip -DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM=NEVER -DCMAKE_FIND_ROOT_PATH_MODE_LIBRARY=ONLY -DCMAKE_FIND_ROOT_PATH_MODE_INCLUDE=ONLY -DCMAKE_FIND_ROOT_PATH=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh;/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/x86_64-conda-linux-gnu/sysroot -DCMAKE_INSTALL_PREFIX=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh -DCMAKE_INSTALL_LIBDIR=lib
+CMAKE_PREFIX_PATH=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh:/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/x86_64-conda-linux-gnu/sysroot/usr
+CONDA_BUILD_SYSROOT=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/x86_64-conda-linux-gnu/sysroot
+_CONDA_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_x86_64_conda_cos6_linux_gnu
+CPPFLAGS=-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/include
+CPP=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-cpp
+DEBUG_CFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-all -fno-plt -Og -g -Wall -Wextra -fvar-tracking-assignments -ffunction-sections -pipe -isystem /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/work=/usr/local/src/conda/pytorch-1.6.0 -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh=/usr/local/src/conda-prefix
+DEBUG_CPPFLAGS=-D_DEBUG -D_FORTIFY_SOURCE=2 -Og -isystem /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/include
+GCC_AR=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-gcc-ar
+GCC=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-gcc
+GCC_NM=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-gcc-nm
+GCC_RANLIB=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-gcc-ranlib
+host_alias=x86_64-conda-linux-gnu
+HOST=x86_64-conda-linux-gnu
+LDFLAGS=-Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib -Wl,-rpath-link,/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib -L/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib
INFO: activate-gxx_linux-64.sh made the following environmental changes:
+CXXFLAGS=-fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/work=/usr/local/src/conda/pytorch-1.6.0 -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh=/usr/local/src/conda-prefix
+CXX_FOR_BUILD=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-c++
+CXX=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-c++
+DEBUG_CXXFLAGS=-fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-all -fno-plt -Og -g -Wall -Wextra -fvar-tracking-assignments -ffunction-sections -pipe -isystem /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/work=/usr/local/src/conda/pytorch-1.6.0 -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh=/usr/local/src/conda-prefix
+GXX=/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin/x86_64-conda-linux-gnu-g++
which: no cuda-gdb in (/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_build_env/bin:/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/bin:/opt/conda/condabin:/opt/conda/bin:/opt/conda/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/conda/bin)
Cannot determine CUDA_HOME: cuda-gdb not in PATH
Traceback (most recent call last):
  File "/opt/conda/bin/conda-build", line 11, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.8/site-packages/conda_build/cli/main_build.py", line 474, in main
    execute(sys.argv[1:])
  File "/opt/conda/lib/python3.8/site-packages/conda_build/cli/main_build.py", line 463, in execute
    outputs = api.build(args.recipe, post=args.post, test_run_post=args.test_run_post,
  File "/opt/conda/lib/python3.8/site-packages/conda_build/api.py", line 186, in build
    return build_tree(
  File "/opt/conda/lib/python3.8/site-packages/conda_build/build.py", line 3008, in build_tree
    packages_from_this = build(metadata, stats,
  File "/opt/conda/lib/python3.8/site-packages/conda_build/build.py", line 2291, in build
    newly_built_packages = bundlers[pkg_type](output_d, m, env, stats)
  File "/opt/conda/lib/python3.8/site-packages/conda_build/build.py", line 1536, in bundle_conda
    utils.check_call_env(interpreter_and_args + [dest_file],
  File "/opt/conda/lib/python3.8/site-packages/conda_build/utils.py", line 410, in check_call_env
    return _func_defaulting_env_to_os_environ('call', *popenargs, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/conda_build/utils.py", line 390, in _func_defaulting_env_to_os_environ
    raise subprocess.CalledProcessError(proc.returncode, _args)
subprocess.CalledProcessError: Command '['/bin/bash', '-e', '/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1608053267045/work/build_pytorch.sh']' returned non-zero exit status 1.
Traceback (most recent call last):
  File "build-locally.py", line 75, in <module>
    main()
  File "build-locally.py", line 71, in main
    run_docker_build(ns)
  File "build-locally.py", line 23, in run_docker_build
    subprocess.check_call([script])
  File "/home/rgommers/anaconda3/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['.scripts/run_docker_build.sh']' returned non-zero exit status 1.

@isuruf
Copy link
Member Author

isuruf commented Dec 15, 2020

Ah, you probably need shyaml installed or you need to set DOCKERIMAGE env variable.

@rgommers
Copy link

Ah, you probably need shyaml installed

Yep that was it. It's building now, but it looks like it only uses one CPU core. That may be my default Docker config, not sure (or is there a conda-forge specific config for this?). Usually when I build PyTorch on a machine with 12 physical cores, I set MAX_JOBS=9, that's about optimal.

@isuruf
Copy link
Member Author

isuruf commented Dec 15, 2020

You can set CPU_COUNT env variable which will be passed through to the docker container

@rgommers
Copy link

This is looking promising. I tested the CUDA 10.2, Python 3.8 build. The majority of the test suite runs fine, it stops on the first encountered failure at:

FAIL: test_unused (__main__.TestTensorExprFuser)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_tensorexpr.py", line 1249, in test_unused
    assert cx.elapsed_value() == 1
AssertionError

Which is a known issue, the test itself is wrong: pytorch/pytorch#41358. So so far so good.

@rgommers
Copy link

Since it took a bit of time to figure out the local build/install/test cycle, summarizing here:

conda create -n pytorch-cforge
conda activate pytorch-cforge
conda config --env --add channels conda-forge
conda install python=3.8 shyaml compilers hypothesis   # first two for building; last two for running tests
export CPU_COUNT=9
python build_locally.py
conda install pytorch -c file:///full/path/to/pytorch-cpu-feedstock/build_artifacts/
# Check out tag v1.6.0 in a local clone of the pytorch repo, then:
python /path/to/pytorch/test/run_test.py

@isuruf
Copy link
Member Author

isuruf commented Dec 16, 2020

A simpler way would be

conda create -n pytorch-cforge
conda activate pytorch-cforge
conda config --env --add channels conda-forge
conda install python=3.8 shyaml conda-build   # first two for building; last one for running tests
export CPU_COUNT=9
python build_locally.py
conda build --test  file:///full/path/to/pytorch-cpu-feedstock/build_artifacts/pytorch-1.6.0-<build-string>.tar.bz2

@isuruf
Copy link
Member Author

isuruf commented Dec 17, 2020

@hmaarrfk, merge? @chrisburr offered to build the GPU variants locally and upload.

@hmaarrfk hmaarrfk merged commit 43735ed into conda-forge:master Dec 17, 2020
@hmaarrfk
Copy link
Contributor

Thanks for all the hard work!

@h-vetinari
Copy link
Member

@isuruf
Just checking what the thought was for the build strings here, since I stumbled over them and tracked them down to this PR - I had been thinking about adding the cuda version to the faiss-buildstrings as well: conda-forge/faiss-split-feedstock@53422a1

Perhaps - like for the mutex naming (cf conda-forge/conda-forge.github.io#1059) - there should be a standard way to do this? Speaking of mutexes - if I understand correctly there's now no "proc" package for pytorch to select the type of installation? Does conda install pytorch=*=cpu still work with the qualifier in front? (wanted to try, but there are no CF packages for windows yet...)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants