Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-34921: [C++][Python][Java] Require CMake 3.16 or later #35921

Merged
merged 8 commits into from
Jun 27, 2023

Conversation

kou
Copy link
Member

@kou kou commented Jun 6, 2023

Rationale for this change

We need to support Ubuntu 20.04 (CMake 3.16) or later, CentOS 7 (CMake 3.17) or later, and Debian GNU/Linux bullseye (CMake 3.18) or later.

So we can require CMake 3.16 or later.

What changes are included in this PR?

Require CMake 3.16 or later.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

@github-actions
Copy link

github-actions bot commented Jun 6, 2023

⚠️ GitHub issue #34921 has been automatically assigned in GitHub to PR creator.

@kou
Copy link
Member Author

kou commented Jun 6, 2023

I'll rebase on main after #35923 is merged.

@kou kou force-pushed the cpp-cmake-3.16 branch from b9911f9 to 5d8bbea Compare June 6, 2023 01:28
@assignUser
Copy link
Member

🎉 very nice!

@pitrou
Copy link
Member

pitrou commented Jun 6, 2023

The CI failure on Windows looks related:
https://github.com/apache/arrow/actions/runs/5183549768/jobs/9354274524#step:11:1307

It seems that bundled dependencies are built with the wrong flags? (release instead of debug perhaps)

orc.lib(TypeImpl.cc.obj) : error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '0' doesn't match value '2' in unity_26_cxx.cxx.obj
orc.lib(TypeImpl.cc.obj) : error LNK2038: mismatch detected for 'RuntimeLibrary': value 'MT_StaticRelease' doesn't match value 'MDd_DynamicDebug' in unity_26_cxx.cxx.obj

This might be because of https://cmake.org/cmake/help/latest/policy/CMP0091.html : """MSVC runtime library flags are selected by an abstraction""".

@kou kou force-pushed the cpp-cmake-3.16 branch from 5d8bbea to 775cd0a Compare June 9, 2023 03:24
@kou
Copy link
Member Author

kou commented Jun 9, 2023

Thanks.
CMP0091 was related.

@kou
Copy link
Member Author

kou commented Jun 9, 2023

FYI: I noticed that Amazon Linux 2 uses CMake 3.13. Amazon Linux 2's EOL was 2023-06-30 but extended to 2025-06-03: https://aws.amazon.com/amazon-linux-2/faqs/?nc1=h_ls
If we add support for Amazon Linux 2023 in Apache Arrow 13.0.0, can we drop support for Amazon Linux 2 in Apache Arrow 13.0.0?
Or should we support CMake 3.13 or later for now?

@kou kou force-pushed the cpp-cmake-3.16 branch from 6f44254 to badabb5 Compare June 10, 2023 02:11
@kou
Copy link
Member Author

kou commented Jun 10, 2023

@github-actions crossbow submit -g linux

@github-actions

This comment was marked as outdated.

@assignUser
Copy link
Member

@kou I am unable to find 3.13 on ama linux 2 (with epel and extras installed and active)

bash-4.2# cmake --version
cmake version 2.8.12.2

In any case I think it is fine to only support the 2023 version as that is the version recommended by aws.
Also 3.13 is missing FetchContent_MakeAvailable and other things which would make moving from ep to fc kind of futile.

@kou
Copy link
Member Author

kou commented Jun 10, 2023

You can find CMake 3 by yum install -y cmake3 and cmak3 --version.

@assignUser
Copy link
Member

Ah I see thanks! Still +1 on supporting 2023 over 2

kou added 6 commits June 26, 2023 15:00
We need to support Ubuntu 20.04 (CMake 3.16) or later, CentOS 7 (CMake
3.17) or later, and Debian GNU/Linux buster (CMake 3.13 but backports
repository has CMake 3.18) or later.

So we can require CMake 3.16 or later.
@kou kou force-pushed the cpp-cmake-3.16 branch from badabb5 to 0e889a5 Compare June 26, 2023 06:01
@kou
Copy link
Member Author

kou commented Jun 26, 2023

@github-actions crossbow submit -g nightly-tests -g nightly-packaging -g nightly-release

@github-actions
Copy link

Revision: 0e889a5

Submitted crossbow builds: ursacomputing/crossbow @ actions-eada57a01a

Task Status
almalinux-8-amd64 Github Actions
almalinux-8-arm64 Github Actions
almalinux-9-amd64 Github Actions
almalinux-9-arm64 Github Actions
amazon-linux-2023-amd64 Github Actions
amazon-linux-2023-arm64 Github Actions
centos-7-amd64 Github Actions
centos-8-stream-amd64 Github Actions
centos-8-stream-arm64 Github Actions
centos-9-stream-amd64 Github Actions
centos-9-stream-arm64 Github Actions
conan-maximum Github Actions
conan-minimum Github Actions
conda-clean Azure
conda-linux-aarch64-cpu-py3 Azure
conda-linux-aarch64-cpu-r42 Azure
conda-linux-ppc64le-cpu-py3 Azure
conda-linux-x64-cpu-py3 Azure
conda-linux-x64-cpu-r42 Azure
conda-linux-x64-cuda-py3 Azure
conda-osx-arm64-cpu-py3 Azure
conda-osx-arm64-cpu-r42 Azure
conda-osx-x64-cpu-py3 Azure
conda-osx-x64-cpu-r42 Azure
conda-win-x64-cpu-py3 Azure
conda-win-x64-cpu-r41 Azure
conda-win-x64-cuda-py3 Azure
debian-bookworm-amd64 Github Actions
debian-bookworm-arm64 Github Actions
debian-bullseye-amd64 Github Actions
debian-bullseye-arm64 Github Actions
example-cpp-minimal-build-static Github Actions
example-cpp-minimal-build-static-system-dependency Github Actions
example-python-minimal-build-fedora-conda Github Actions
example-python-minimal-build-ubuntu-venv Github Actions
homebrew-cpp Github Actions
homebrew-r-autobrew Github Actions
homebrew-r-brew Github Actions
java-jars Github Actions
nuget Github Actions
python-sdist Github Actions
r-binary-packages Github Actions
test-alpine-linux-cpp Github Actions
test-build-cpp-fuzz Github Actions
test-build-vcpkg-win Github Actions
test-conda-cpp Github Actions
test-conda-cpp-valgrind Azure
test-conda-python-3.10 Github Actions
test-conda-python-3.10-hdfs-2.9.2 Github Actions
test-conda-python-3.10-hdfs-3.2.1 Github Actions
test-conda-python-3.10-pandas-latest Github Actions
test-conda-python-3.10-pandas-nightly Github Actions
test-conda-python-3.10-spark-master Github Actions
test-conda-python-3.10-substrait Github Actions
test-conda-python-3.11 Github Actions
test-conda-python-3.11-dask-latest Github Actions
test-conda-python-3.11-dask-upstream_devel Github Actions
test-conda-python-3.11-hypothesis Github Actions
test-conda-python-3.11-pandas-upstream_devel Github Actions
test-conda-python-3.8 Github Actions
test-conda-python-3.8-pandas-1.0 Github Actions
test-conda-python-3.8-spark-v3.1.2 Github Actions
test-conda-python-3.9 Github Actions
test-conda-python-3.9-pandas-latest Github Actions
test-conda-python-3.9-spark-v3.2.0 Github Actions
test-cuda-cpp Github Actions
test-cuda-python Github Actions
test-debian-11-cpp-amd64 Github Actions
test-debian-11-cpp-i386 Github Actions
test-debian-11-go-1.17 Azure
test-debian-11-go-1.20 Azure
test-debian-11-python-3 Azure
test-debian-c-glib Github Actions
test-debian-ruby Github Actions
test-fedora-35-cpp Github Actions
test-fedora-35-python-3 Azure
test-fedora-r-clang-sanitizer Azure
test-r-arrow-backwards-compatibility Github Actions
test-r-depsource-bundled Azure
test-r-depsource-system Github Actions
test-r-dev-duckdb Github Actions
test-r-devdocs Github Actions
test-r-gcc-11 Github Actions
test-r-gcc-12 Github Actions
test-r-install-local Github Actions
test-r-install-local-minsizerel Github Actions
test-r-library-r-base-latest Azure
test-r-linux-as-cran Github Actions
test-r-linux-rchk Github Actions
test-r-linux-valgrind Azure
test-r-minimal-build Azure
test-r-offline-maximal Github Actions
test-r-offline-minimal Azure
test-r-rhub-debian-gcc-devel-lto-latest Azure
test-r-rhub-debian-gcc-release-custom-ccache Azure
test-r-rhub-ubuntu-gcc-release-latest Azure
test-r-rstudio-r-base-4.1-opensuse153 Azure
test-r-rstudio-r-base-4.2-centos7-devtoolset-8 Azure
test-r-rstudio-r-base-4.2-focal Azure
test-r-ubuntu-22.04 Github Actions
test-r-versions Github Actions
test-skyhook-integration Github Actions
test-ubuntu-20.04-cpp Github Actions
test-ubuntu-20.04-cpp-20 Github Actions
test-ubuntu-20.04-cpp-bundled Github Actions
test-ubuntu-20.04-cpp-minimal-with-formats Github Actions
test-ubuntu-20.04-cpp-thread-sanitizer Github Actions
test-ubuntu-20.04-python-3 Azure
test-ubuntu-22.04-cpp Github Actions
test-ubuntu-c-glib Github Actions
test-ubuntu-default-docs Azure
test-ubuntu-r-sanitizer Azure
test-ubuntu-ruby Github Actions
ubuntu-focal-amd64 Github Actions
ubuntu-focal-arm64 Github Actions
ubuntu-jammy-amd64 Github Actions
ubuntu-jammy-arm64 Github Actions
ubuntu-lunar-amd64 Github Actions
ubuntu-lunar-arm64 Github Actions
verify-rc-source-cpp-linux-almalinux-8-amd64 Github Actions
verify-rc-source-cpp-linux-conda-latest-amd64 Github Actions
verify-rc-source-cpp-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-source-cpp-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-source-cpp-macos-amd64 Github Actions
verify-rc-source-cpp-macos-arm64 Github Actions
verify-rc-source-cpp-macos-conda-amd64 Github Actions
verify-rc-source-csharp-linux-almalinux-8-amd64 Github Actions
verify-rc-source-csharp-linux-conda-latest-amd64 Github Actions
verify-rc-source-csharp-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-source-csharp-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-source-csharp-macos-amd64 Github Actions
verify-rc-source-csharp-macos-arm64 Github Actions
verify-rc-source-go-linux-almalinux-8-amd64 Github Actions
verify-rc-source-go-linux-conda-latest-amd64 Github Actions
verify-rc-source-go-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-source-go-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-source-go-macos-amd64 Github Actions
verify-rc-source-go-macos-arm64 Github Actions
verify-rc-source-integration-linux-almalinux-8-amd64 Github Actions
verify-rc-source-integration-linux-conda-latest-amd64 Github Actions
verify-rc-source-integration-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-source-integration-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-source-integration-macos-amd64 Github Actions
verify-rc-source-integration-macos-arm64 Github Actions
verify-rc-source-integration-macos-conda-amd64 Github Actions
verify-rc-source-java-linux-almalinux-8-amd64 Github Actions
verify-rc-source-java-linux-conda-latest-amd64 Github Actions
verify-rc-source-java-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-source-java-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-source-java-macos-amd64 Github Actions
verify-rc-source-js-linux-almalinux-8-amd64 Github Actions
verify-rc-source-js-linux-conda-latest-amd64 Github Actions
verify-rc-source-js-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-source-js-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-source-js-macos-amd64 Github Actions
verify-rc-source-js-macos-arm64 Github Actions
verify-rc-source-python-linux-almalinux-8-amd64 Github Actions
verify-rc-source-python-linux-conda-latest-amd64 Github Actions
verify-rc-source-python-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-source-python-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-source-python-macos-amd64 Github Actions
verify-rc-source-python-macos-arm64 Github Actions
verify-rc-source-python-macos-conda-amd64 Github Actions
verify-rc-source-ruby-linux-almalinux-8-amd64 Github Actions
verify-rc-source-ruby-linux-conda-latest-amd64 Github Actions
verify-rc-source-ruby-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-source-ruby-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-source-ruby-macos-amd64 Github Actions
verify-rc-source-ruby-macos-arm64 Github Actions
verify-rc-source-windows Github Actions
wheel-clean Github Actions
wheel-macos-big-sur-cp310-arm64 Github Actions
wheel-macos-big-sur-cp311-arm64 Github Actions
wheel-macos-big-sur-cp38-arm64 Github Actions
wheel-macos-big-sur-cp39-arm64 Github Actions
wheel-macos-mojave-cp310-amd64 Github Actions
wheel-macos-mojave-cp311-amd64 Github Actions
wheel-macos-mojave-cp38-amd64 Github Actions
wheel-macos-mojave-cp39-amd64 Github Actions
wheel-manylinux-2-28-cp310-amd64 Github Actions
wheel-manylinux-2-28-cp310-arm64 Github Actions
wheel-manylinux-2-28-cp311-amd64 Github Actions
wheel-manylinux-2-28-cp311-arm64 Github Actions
wheel-manylinux-2-28-cp38-amd64 Github Actions
wheel-manylinux-2-28-cp38-arm64 Github Actions
wheel-manylinux-2-28-cp39-amd64 Github Actions
wheel-manylinux-2-28-cp39-arm64 Github Actions
wheel-manylinux-2014-cp310-amd64 Github Actions
wheel-manylinux-2014-cp310-arm64 Github Actions
wheel-manylinux-2014-cp311-amd64 Github Actions
wheel-manylinux-2014-cp311-arm64 Github Actions
wheel-manylinux-2014-cp38-amd64 Github Actions
wheel-manylinux-2014-cp38-arm64 Github Actions
wheel-manylinux-2014-cp39-amd64 Github Actions
wheel-manylinux-2014-cp39-arm64 Github Actions
wheel-windows-cp310-amd64 Github Actions
wheel-windows-cp311-amd64 Github Actions
wheel-windows-cp38-amd64 Github Actions
wheel-windows-cp39-amd64 Github Actions

@kou
Copy link
Member Author

kou commented Jun 26, 2023

No objection on the mailing thread: https://lists.apache.org/thread/1gfpg0pxhk10n10lo0snx68t3pxfs2sl

I'll merge this once we confirm that this doesn't break our CI.

@kou kou merged commit 11b140a into apache:main Jun 27, 2023
@kou kou deleted the cpp-cmake-3.16 branch June 27, 2023 05:53
@kou kou removed the awaiting committer review Awaiting committer review label Jun 27, 2023
kou pushed a commit that referenced this pull request Jun 27, 2023
Followup to #35921. The nixlibs.R build script checks for suitable cmake on the system, so since we require newer cmake for C++ now, we need to update the check here. 

I refactored slightly so that there now is a variable in the script called `cmake_minimum_required`, same as in the CMakeLists.txt files throughout the project, so hopefully the next time we update the cmake version, we'll find this file better.

I also bumped the version of cmake that we download (if cmake 3.16 is not found) to the latest release.

cc @ kou 

Authored-by: Neal Richardson <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
@conbench-apache-arrow
Copy link

Conbench analyzed the 6 benchmark runs on commit 11b140a7.

There were 8 benchmark results indicating a performance regression:

The full Conbench report has more details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[C++] Require CMake 3.16 or later
3 participants