[CI] Test CMake builds instead of Makefile builds #17645

leezu · 2020-02-21T02:12:40Z

Description

Use cmake build for most CI builds in preparation of dropping Makefile build

Fixes #17548

Comments

The following Makefile based builds are preserved

staticbuild scripts. Changes to CD pipeline will happen in Switch CD to cmake build #17646
Docs builds. Language binding specific build logic requires further changes
Jetson build. Jetpack 3.3 toolchain based on Cuda 9.0 causes 'Internal
Compiler Error (codegen): "there was an error in verifying the lgenfe
output!"' errors with cmake. This seems to be a known issue in Cuda 9.0 and
we need to update Jetpack toolchain to work around it.
MKL builds. Waiting for fix of OpenMP Error #17641

Improvements to CMake build

Enable -Werror for RelWithDebugInfo build in analogy to "make DEV=1" build
Add USE_LIBJPEG_TURBO to CMake build
Improve finding Python 3 executable

Changes to CI setup

Install protobuf in ubuntu_core.sh and centos_core.sh
Install up-to-date CMake on Centos 7
Don't use RelWithDebInfo on Android builds, as gcc 4.9 throws
-Wdelete-non-virtual-dtor

Code changes

Locally disable warnings introduced by GCC7

ci/docker/runtime_functions.sh

leezu · 2020-02-21T07:10:46Z

@apeforest I agree the TVM Op testing can be improved. I'll first attempt to make this PR pass CI, without changing any of the build configurations tested (but only switch the build system).
Once CI passes, we can consider merging and have a separate PR for changing configurations or iterate on this PR.

apeforest · 2020-02-21T19:07:25Z

Could you first test this in the mxnet-ci-dev environment? If there is anything wrong, we don't want it to break the CI production right?

In particular the following Makefile based builds are preserved 1) staticbuild scripts 2) Docs builds. Language binding specific build logic requires further changes 3) Jetson build. Jetpack 3.3 toolchain based on Cuda 9.0 causes 'Internal Compiler Error (codegen): "there was an error in verifying the lgenfe output!"' errors with cmake. This seems to be a known issue in Cuda 9.0 and we need to update Jetpack toolchain to work around it. All Makefile based builds are marked with a "Makefile" postfix in the title. Improvements to CMake build - Enable -Werror for RelWithDebugInfo build in analogy to "make DEV=1" build - Add USE_LIBJPEG_TURBO to CMake build - Improve finding Python 3 executable Changes to CI setup - Install protobuf in ubuntu_core.sh and centos_core.sh - Install up-to-date CMake on Centos 7 - Don't use RelWithDebInfo on Android builds, as gcc 4.9 throws -Wdelete-non-virtual-dtor Code changes - Locally disable warnings introduced by GCC7

Fixes a regression in CD introduced by apache#17645

* Fix cd/mxnet_lib/dynamic/Jenkins_pipeline.groovy Fixes a regression in CD introduced by #17645 * Fix whitespace * Add NATIVE_ADDITIONAL.md Fixes a regression in CD introduced by #16899 * Update other $TYPE_ADDITIONAL.MD * Fix cd/python/docker Fixes regression introduced by #15990

The following Makefile based builds are preserved 1) staticbuild scripts 2) Docs builds. Language binding specific build logic requires further changes 3) Jetson build. Jetpack 3.3 toolchain based on Cuda 9.0 causes 'Internal Compiler Error (codegen): "there was an error in verifying the lgenfe output!"' errors with cmake. This seems to be a known issue in Cuda 9.0 and we need to update Jetpack toolchain to work around it. 4) MKL builds. Waiting for fix of apache#17641 All Makefile based builds are marked with a "Makefile" postfix in the title. Improvements to CMake build - Enable -Werror for RelWithDebugInfo build in analogy to "make DEV=1" build - Add USE_LIBJPEG_TURBO to CMake build - Improve finding Python 3 executable Changes to CI setup - Install protobuf and zmq where missing - Install up-to-date CMake on Centos 7 - Don't use RelWithDebInfo on Android builds, as gcc 4.9 throws -Wdelete-non-virtual-dtor Code changes - Disable warnings introduced by GCC7 at via #pragma GCC diagnostic

* Fix cd/mxnet_lib/dynamic/Jenkins_pipeline.groovy Fixes a regression in CD introduced by apache#17645 * Fix whitespace * Add NATIVE_ADDITIONAL.md Fixes a regression in CD introduced by apache#16899 * Update other $TYPE_ADDITIONAL.MD * Fix cd/python/docker Fixes regression introduced by apache#15990

The following Makefile based builds are preserved 1) staticbuild scripts 2) Docs builds. Language binding specific build logic requires further changes 3) Jetson build. Jetpack 3.3 toolchain based on Cuda 9.0 causes 'Internal Compiler Error (codegen): "there was an error in verifying the lgenfe output!"' errors with cmake. This seems to be a known issue in Cuda 9.0 and we need to update Jetpack toolchain to work around it. 4) MKL builds. Waiting for fix of apache#17641 All Makefile based builds are marked with a "Makefile" postfix in the title. Improvements to CMake build - Enable -Werror for RelWithDebugInfo build in analogy to "make DEV=1" build - Add USE_LIBJPEG_TURBO to CMake build - Improve finding Python 3 executable Changes to CI setup - Install protobuf and zmq where missing - Install up-to-date CMake on Centos 7 - Don't use RelWithDebInfo on Android builds, as gcc 4.9 throws -Wdelete-non-virtual-dtor Code changes - Disable warnings introduced by GCC7 at via #pragma GCC diagnostic

* Fix cd/mxnet_lib/dynamic/Jenkins_pipeline.groovy Fixes a regression in CD introduced by apache#17645 * Fix whitespace * Add NATIVE_ADDITIONAL.md Fixes a regression in CD introduced by apache#16899 * Update other $TYPE_ADDITIONAL.MD * Fix cd/python/docker Fixes regression introduced by apache#15990

* Fix CD (#17776) * Fix cd/mxnet_lib/dynamic/Jenkins_pipeline.groovy Fixes a regression in CD introduced by #17645 * Fix whitespace * Add NATIVE_ADDITIONAL.md Fixes a regression in CD introduced by #16899 * Update other $TYPE_ADDITIONAL.MD * Fix cd/python/docker Fixes regression introduced by #15990 * [CD] update pypi description, setup.py (#17681) * update pypi description, setup.py, use manylinux2010, use unified dist link for nightly * Use manylinux2014 Co-authored-by: Leonard Lausen <[email protected]> * reverting .so path as per MAKE flow Co-authored-by: Leonard Lausen <[email protected]> Co-authored-by: Sheng Zha <[email protected]> Co-authored-by: Leonard Lausen <[email protected]>

sxjscience · 2020-07-01T06:20:07Z

@ciyongch Would this go to v1.7? I come up to this commit when I'm cherry-picking the related commits one by one

ciyongch · 2020-07-03T06:10:31Z

Hi @sxjscience , sorry for overlooking your message. I don't see there's a strong requirement for this and likely not to include this patch in v1.7.

* Fix CD (apache#17776) * Fix cd/mxnet_lib/dynamic/Jenkins_pipeline.groovy Fixes a regression in CD introduced by apache#17645 * Fix whitespace * Add NATIVE_ADDITIONAL.md Fixes a regression in CD introduced by apache#16899 * Update other $TYPE_ADDITIONAL.MD * Fix cd/python/docker Fixes regression introduced by apache#15990 * [CD] update pypi description, setup.py (apache#17681) * update pypi description, setup.py, use manylinux2010, use unified dist link for nightly * Use manylinux2014 Co-authored-by: Leonard Lausen <[email protected]> * reverting .so path as per MAKE flow Co-authored-by: Leonard Lausen <[email protected]> Co-authored-by: Sheng Zha <[email protected]> Co-authored-by: Leonard Lausen <[email protected]>

leezu requested review from aaronmarkham, marcoabreu and szha as code owners February 21, 2020 02:12

leezu force-pushed the citestcmake branch 6 times, most recently from aebaae0 to 46adbfb Compare February 21, 2020 06:36