Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[python3] Build error #33068

Open
byxrolland opened this issue Aug 9, 2023 · 27 comments
Open

[python3] Build error #33068

byxrolland opened this issue Aug 9, 2023 · 27 comments
Assignees
Labels
requires:more-information This Issue requires more information to solve

Comments

@byxrolland
Copy link

Package: python3:x64-linux -> 3.11.4

Host Environment

  • Host: x64-linux
  • Compiler: GNU 11.4.0
  • vcpkg-tool version: 2023-08-02-6d13efa755f9b5e101712d210199e4139b4c29f6
    vcpkg-scripts version: 247662e 2023-08-08 (12 hours ago)

To Reproduce

vcpkg install boost[mpi]

Failure logs

-- Using cached python-cpython-v3.11.4.tar.gz.
-- Cleaning sources at /home/byxrolland/vcpkg/buildtrees/python3/src/v3.11.4-c76ea88c37.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source /home/byxrolland/vcpkg/downloads/python-cpython-v3.11.4.tar.gz
-- Applying patch 0001-only-build-required-projects.patch
-- Applying patch 0003-use-vcpkg-zlib.patch
-- Applying patch 0004-devendor-external-dependencies.patch
-- Applying patch 0005-dont-copy-vcruntime.patch
-- Applying patch 0008-python.pc.patch
-- Applying patch 0010-dont-skip-rpath.patch
-- Applying patch 0012-force-disable-curses.patch
-- Applying patch 0014-fix-get-python-inc-output.patch
-- Applying patch 0002-static-library.patch
-- Applying patch 0011-gcc-ldflags-fix.patch
-- Using source at /home/byxrolland/vcpkg/buildtrees/python3/src/v3.11.4-c76ea88c37.clean
-- Getting CMake variables for x64-linux-dbg
-- Getting CMake variables for x64-linux-rel
-- Generating configure for x64-linux
-- Finished generating configure for x64-linux
-- Configuring x64-linux-dbg
-- Configuring x64-linux-rel
-- Building x64-linux-dbg
-- Installing x64-linux-dbg
-- Building x64-linux-rel
-- Installing x64-linux-rel
-- Installing: /home/byxrolland/vcpkg/packages/python3_x64-linux/share/python3/copyright
-- Fixing pkgconfig file: /home/byxrolland/vcpkg/packages/python3_x64-linux/lib/pkgconfig/python-3.11-embed.pc
-- Fixing pkgconfig file: /home/byxrolland/vcpkg/packages/python3_x64-linux/lib/pkgconfig/python-3.11.pc
-- Fixing pkgconfig file: /home/byxrolland/vcpkg/packages/python3_x64-linux/debug/lib/pkgconfig/python-3.11-embed.pc
-- Fixing pkgconfig file: /home/byxrolland/vcpkg/packages/python3_x64-linux/debug/lib/pkgconfig/python-3.11.pc
CMake Error at ports/python3/portfile.cmake:294 (message):
  There should be no modules with "_failed" suffix:


  /home/byxrolland/vcpkg/packages/python3_x64-linux/debug/lib/python3.11/lib-dynload/_crypt.cpython-311d-x86_64-linux-gnu_failed.so



  /home/byxrolland/vcpkg/packages/python3_x64-linux/lib/python3.11/lib-dynload/_crypt.cpython-311-x86_64-linux-gnu_failed.so
Call Stack (most recent call first):
  scripts/ports.cmake:147 (include)



Additional context

@FrankXie05 FrankXie05 added the category:port-bug The issue is with a library, which is something the port should already support label Aug 9, 2023
@FrankXie05
Copy link
Contributor

Duplicate of #29447

@FrankXie05 FrankXie05 marked this as a duplicate of #29447 Aug 9, 2023
@FrankXie05 FrankXie05 closed this as not planned Won't fix, can't repro, duplicate, stale Aug 9, 2023
@byxrolland
Copy link
Author

I didn't have this problem when building Python 3.10, only 3.11. At first it required autoconf. After I installed autoconf it said there was a macro not defined. After I installed automake and libtool, it showed the error as above.

Duplicate of #29447

@Osyotr
Copy link
Contributor

Osyotr commented Aug 11, 2023

@FrankXie05 this does not look like a duplicate of #29447. Please reopen the ticket.

@byxrolland please upload more logs.

@FrankXie05 FrankXie05 reopened this Aug 11, 2023
@FrankXie05
Copy link
Contributor

@byxrolland Could you please run command sudo apt-get install libcrypt-dev to re-install crypt?

@FrankXie05 FrankXie05 added requires:more-information This Issue requires more information to solve and removed category:port-bug The issue is with a library, which is something the port should already support labels Aug 15, 2023
@byxrolland
Copy link
Author

@FrankXie05 this does not look like a duplicate of #29447. Please reopen the ticket.

@byxrolland please upload more logs.

Where can I find more logs?

@byxrolland
Copy link
Author

@byxrolland Could you please run command sudo apt-get install libcrypt-dev to re-install crypt?

It is already installed.

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
libcrypt-dev is already the newest version (1:4.4.27-1).
libcrypt-dev set to manually installed.

@Osyotr
Copy link
Contributor

Osyotr commented Aug 15, 2023

Where can I find more logs?

/home/byxrolland/vcpkg/buildtrees/python3

@byxrolland
Copy link
Author

Where can I find more logs?

/home/byxrolland/vcpkg/buildtrees/python3

In autoconf-x64-linux-err.log:

autoreconf: export WARNINGS=
autoreconf: Entering directory '.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal --force 
autoreconf: configure.ac: tracing
autoreconf: configure.ac: not using Libtool
autoreconf: configure.ac: not using Intltool
autoreconf: configure.ac: not using Gtkdoc
autoreconf: running: /home/linuxbrew/.linuxbrew/Cellar/autoconf/2.71/bin/autoconf --force
autoreconf: running: /home/linuxbrew/.linuxbrew/Cellar/autoconf/2.71/bin/autoheader --force
autoreconf: configure.ac: not using Automake
autoreconf: Leaving directory '.'

The following messages occurred frequently in several err.logs:

*** WARNING: renaming "_crypt" since importing it failed: libcrypt.so.2: cannot open shared object file: No such file or directory
renaming build/scripts-3.11/pydoc3 to build/scripts-3.11/pydoc3.11
renaming build/scripts-3.11/idle3 to build/scripts-3.11/idle3.11
renaming build/scripts-3.11/2to3 to build/scripts-3.11/2to3-3.11
configure: WARNING: unrecognized options: --disable-silent-rules, --enable-static
configure: WARNING: unrecognized options: --disable-silent-rules, --enable-static
CMake Warning:
  Manually-specified variables were not used by the project:

    BUILD_SHARED_LIBS
    CMAKE_INSTALL_BINDIR
    CMAKE_INSTALL_LIBDIR
    VCPKG_PLATFORM_TOOLSET
    VCPKG_SET_CHARSET_FLAG
    _VCPKG_ROOT_DIR

Hope these could be helpful.

@Osyotr
Copy link
Contributor

Osyotr commented Aug 16, 2023

Would you kindly upload config-x64-linux-rel-config.log and build-x64-linux-rel-out.log?

@byxrolland
Copy link
Author

@Osyotr
Copy link
Contributor

Osyotr commented Aug 17, 2023

Ok, here are some hints:

PKG_CONFIG='/home/linuxbrew/.linuxbrew/bin/pkg-config'
pkg_cv_LIBCRYPT_CFLAGS=-I/home/linuxbrew/.linuxbrew/Cellar/libxcrypt/4.4.36/include
pkg_cv_LIBCRYPT_LIBS='-L/home/linuxbrew/.linuxbrew/Cellar/libxcrypt/4.4.36/lib -lcrypt'

You use pkg-config from linuxbrew which then finds libxcrypt in /home/linuxbrew/.linuxbrew/Cellar/libxcrypt/4.4.36/lib, but this path is not in your library path. Consider setting LD_LIBRARY_PATH prior to running vcpkg.

@byxrolland
Copy link
Author

Ok, here are some hints:

PKG_CONFIG='/home/linuxbrew/.linuxbrew/bin/pkg-config'
pkg_cv_LIBCRYPT_CFLAGS=-I/home/linuxbrew/.linuxbrew/Cellar/libxcrypt/4.4.36/include
pkg_cv_LIBCRYPT_LIBS='-L/home/linuxbrew/.linuxbrew/Cellar/libxcrypt/4.4.36/lib -lcrypt'

You use pkg-config from linuxbrew which then finds libxcrypt in /home/linuxbrew/.linuxbrew/Cellar/libxcrypt/4.4.36/lib, but this path is not in your library path. Consider setting LD_LIBRARY_PATH prior to running vcpkg.

But the linuxbrew has libxcrypt installed. Why is that a problem?

Also, could you tell how to set LD_LIBRARY_PATH?

@byxrolland
Copy link
Author

Also, I noticed other problems:

The linuxbrew has gcc-13 installed but vcpkg only detected the system gcc.

Also, there is an error that appears frequently in config-x64-linux-rel-config.log, namely

cc1: error: argument to '-O' should be a non-negative integer, 'g', 's' or 'fast'

Is that going to be a problem?

@Osyotr
Copy link
Contributor

Osyotr commented Aug 20, 2023

The linuxbrew has gcc-13 installed but vcpkg only detected the system gcc.

How do you invoke gcc that is installed from linuxbrew? I don't have any experience with linuxbrew but I guess there's some kind of environment activation like in SCL or venv (UPD: looks like that's true https://docs.brew.sh/Homebrew-on-Linux). In that case you need to run vcpkg from the same environment.

But the linuxbrew has libxcrypt installed. Why is that a problem?

The problem is that the path to the library is not in your LD_LIBRARY_PATH so any program that links to libxcrypt.so will not be able to start. It might be the same issue I explained above.

Also, there is an error that appears frequently in config-x64-linux-rel-config.log

I'd say this is python/cpython#105062, not really critical to build process.

@byxrolland
Copy link
Author

byxrolland commented Aug 20, 2023

How do you invoke gcc that is installed from linuxbrew? I don't have any experience with linuxbrew but I guess there's some kind of environment activation like in SCL or venv (UPD: looks like that's true https://docs.brew.sh/Homebrew-on-Linux). In that case you need to run vcpkg from the same environment.

It is in LINE 201 of bootstrap.sh. Your script doesn't accept gcc-13. It stops at g++-12.

@Osyotr
Copy link
Contributor

Osyotr commented Aug 20, 2023

It is in LINE 201 of bootstrap.sh. Your script doesn't accept gcc-13. It stops at g++-12.

This code https://github.com/microsoft/vcpkg/blob/master/scripts/bootstrap.sh#L201 is used only if you build vcpkg tool from source and does not affect building of packages.

@github-actions
Copy link

This is an automated message. Per our repo policy, stale issues get closed if there has been no activity in the past 28 days. The issue will be automatically closed in 14 days. If you wish to keep this issue open, please add a new comment.

@github-actions github-actions bot added Stale and removed Stale labels Sep 18, 2023
@byxrolland
Copy link
Author

I'm waiting for g++-13 support in the script.

@dg0yt
Copy link
Contributor

dg0yt commented Sep 21, 2023

I'm waiting for g++-13 support in the script.

The support is already there: Set CXX to the desired compiler.
The if stuff is to be removed, #33306 (comment).

@byxrolland
Copy link
Author

I'm waiting for g++-13 support in the script.

The support is already there: Set CXX to the desired compiler. The if stuff is to be removed, #33306 (comment).

Could you tell me how to set it?

@byxrolland
Copy link
Author

OK. I set CXX and CC to the linuxbrew location (which will use its own autoconf and automake) and the problem is solved. But I really would like vcpkg to auto-detect the latest C and C++ compiler available on my machine. Otherwise, the mismatch of compilers and autoconf is really hard to find.

@dg0yt
Copy link
Contributor

dg0yt commented Sep 23, 2023

I think you are mixing up things.

  • The bootstrap script is for building the vcpkg tool (when it isn't downloaded as a binary). It doesn't change what is used to build ports like python3.
    As written in [python3] Build error #33068 (comment)
  • Port builds are controlled by triplet and toolchain files. If you don't want to touch these files you can simply setup your PATH to start with th directories which provide the desired g++, gcc, autoconf etc. You can create a single directory with symlinks for convenience. (I use that for arm64-linux on x64-linux.)
    This is a standard pattern.
    Setting CC/CXX won't work well for cross-builds.
  • I don't think there is a general mismatch of compilers and autoconf.
    But with different package providers, it is easy to end up with mixing incompatible libraries at runtime.

@PhoebeHui
Copy link
Contributor

I don't think there is a general mismatch of compilers and autoconf.
But with different package providers, it is easy to end up with mixing incompatible libraries at runtime.

I totally agree.

@byxrolland, it looks the problem was solved now?

@byxrolland
Copy link
Author

It works for me know but I wouldn't consider it as solved as you changed the behavior of vcpkg and don't regard that as your problem now.

@PhoebeHui
Copy link
Contributor

@byxrolland, good to know it doesn't block you now。

For autoconf automake failures, we have another issue #33080 to track.

For your suggestions ' I really would like vcpkg to auto-detect the latest C and C++ compiler available on my machine', I would say it's not a python3 issue, AFAIK vcpkg assumes that a useable compiler exists in your system https://github.com/microsoft/vcpkg/blob/master/scripts/detect_compiler/CMakeLists.txt, which from CMake CMAKE_CXX_COMPILER and CMAKE_C_COMPILER(linux.cmake), that means vcpkg doesn't auto detect or change enviroment variable about compiler, but rely on these values from build system or user manually passed. Anyway, in my opinion, it's out of a scope for a specific port issue, but could be improved in the long term, it would be great if you could open a discussion issue with details suggestions.

@asclearuc
Copy link

Looks like I have similar problem, and don't know how to solve it...

  • Host: Ubuntu 20/WSL (under Windows 11)
  • Compiler: clang-9
  • vcpkg: 2023.06.20 (however the same is true for the master branch of vcpkg)

ATTN: the same build runs OK on Ubuntu 18/WSL (under the same Windows 11)

Trying to build python using command vcpkg install python3, and here is the output:

-- Fixing pkgconfig file: /home/asclearuc/development/engine/vcpkg/packages/python3_x64-linux-clang9-aparavi/debug/lib/pkgconfig/python-3.10.pc
CMake Error at /home/asclearuc/development/engine/apLib/cmake/ports/python3/portfile.cmake:299 (message):
  There should be no modules with "_failed" suffix:
 /home/asclearuc/development/engine/vcpkg/packages/python3_x64-linux-clang9-aparavi/debug/lib/python3.10/lib-dynload/_crypt.cpython-310d-x86_64-linux-gnu_failed.so
  /home/asclearuc/development/engine/vcpkg/packages/python3_x64-linux-clang9-aparavi/debug/lib/python3.10/lib-dynload/nis.cpython-310d-x86_64-linux-gnu_failed.so
  /home/asclearuc/development/engine/vcpkg/packages/python3_x64-linux-clang9-aparavi/lib/python3.10/lib-dynload/_crypt.cpython-310-x86_64-linux-gnu_failed.so
  /home/asclearuc/development/engine/vcpkg/packages/python3_x64-linux-clang9-aparavi/lib/python3.10/lib-dynload/nis.cpython-310-x86_64-linux-gnu_failed.so
Call Stack (most recent call first):
  scripts/ports.cmake:170 (include)

error: building python3:x64-linux-clang9-aparavi failed with: BUILD_FAILED

Investigating a little bit log files:

  • from build-x64-linux-clang9-aparavi-dbg-err.log - there are 2 undefined symbols, one of them is crypt_r
*** WARNING: renaming "_crypt" since importing it failed: /home/asclearuc/development/engine.latest-vcpkg/vcpkg/buildtrees/python3/x64-linux-clang9-aparavi-dbg/build/lib.linux-x86_64-3.10-pydebug/_crypt.cpython-310d-x86_64-linux-gnu.so: undefined symbol: crypt_r
*** WARNING: renaming "nis" since importing it failed: /home/asclearuc/development/engine.latest-vcpkg/vcpkg/buildtrees/python3/x64-linux-clang9-aparavi-dbg/build/lib.linux-x86_64-3.10-pydebug/nis.cpython-310d-x86_64-linux-gnu.so: undefined symbol: yp_get_default_domain
renaming build/scripts-3.10/pydoc3 to build/scripts-3.10/pydoc3.10
renaming build/scripts-3.10/idle3 to build/scripts-3.10/idle3.10
renaming build/scripts-3.10/2to3 to build/scripts-3.10/2to3-3.10
  • looking at the binaries:

On Ubuntu 18 (please notice presence of crypt in the output):

vcpkg/buildtrees/python3$ ldd ./x64-linux-clang9-aparavi-dbg/build/lib.linux-x86_64-3.10-pydebug/_crypt.cpython-310d-x86_64-linux-gnu.so
linux-vdso.so.1 (0x00007ffdb9df5000)
libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f0b2390a000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0b236eb000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0b232fa000)
/lib64/ld-linux-x86-64.so.2 (0x00007f0b23b42000)

On Ubuntu 20 (libcrypt is not present :( ):

vcpkg/buildtrees/python3$ ldd ./x64-linux-clang9-aparavi-rel/build/lib.linux-x86_64-3.10/_crypt.cpython-310-x86_64-linux-gnu_failed.so
linux-vdso.so.1 (0x00007ffdcf273000)
libpthread.so.0 => /usr/lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f660b4c8000)
libc.so.6 => /usr/lib/x86_64-linux-gnu/libc.so.6 (0x00007f660b2d6000)
/lib64/ld-linux-x86-64.so.2 (0x00007f660b4f1000)
  • command line to link _crypt

on Ubuntu 20:

/usr/bin/clang-9 -pthread -shared -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib -stdlib=libc++ -Wl,--no-export-dynamic /usr/lib/llvm-9/lib/libc++.a /usr/lib/llvm-9/lib/libc++abi.a -fuse-ld=lld-9 -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib/manual-link -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib/manual-link -fPIC -g -DCATCH_CONFIG_NO_POSIX_SIGNALS build/temp.linux-x86_64-3.10-pydebug/home/asclearuc/development/engine/vcpkg/buildtrees/python3/src/v3.10.7-e072c2af43.clean/Modules/_cryptmodule.o -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib/manual-link -L/usr/local/lib -Wl,--enable-new-dtags,-R/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib -Wl,--enable-new-dtags,-R/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib/manual-link -Wl,--enable-new-dtags,-R/usr/local/lib -Wl,--enable-new-dtags,-R/lib64 -Wl,--enable-new-dtags,-R/usr/lib64 -Wl,--enable-new-dtags,-R/lib -Wl,--enable-new-dtags,-R/usr/lib -o build/lib.linux-x86_64-3.10-pydebug/_crypt.cpython-310d-x86_64-linux-gnu.so

and corresponding line on Ubuntu 18 (-lcrypt is present):

/usr/bin/clang-9 -pthread -shared -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib -stdlib=libc++ -Wl,--no-export-dynamic /usr/lib/llvm-9/lib/libc++.a /usr/lib/llvm-9/lib/libc++abi.a -fuse-ld=lld-9 -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib/manual-link -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib/manual-link -fPIC -g -DCATCH_CONFIG_NO_POSIX_SIGNALS build/temp.linux-x86_64-3.10-pydebug/home/asclearuc/development/engine/vcpkg/buildtrees/python3/src/v3.10.7-e072c2af43.clean/Modules/_cryptmodule.o -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib/manual-link -L/usr/lib/x86_64-linux-gnu -L/usr/local/lib -Wl,--enable-new-dtags,-R/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib -Wl,--enable-new-dtags,-R/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib/manual-link -Wl,--enable-new-dtags,-R/usr/lib/x86_64-linux-gnu -Wl,--enable-new-dtags,-R/usr/local/lib -Wl,--enable-new-dtags,-R/lib64 -Wl,--enable-new-dtags,-R/usr/lib64 -Wl,--enable-new-dtags,-R/lib -Wl,--enable-new-dtags,-R/usr/lib -lcrypt -o build/lib.linux-x86_64-3.10-pydebug/_crypt.cpython-310d-x86_64-linux-gnu.so
  • config-x64-linux-clang9-aparavi-dbg-config.log has the same information for both U18/U20
ac_cv_search_crypt=-lcrypt
ac_cv_search_crypt_r='none required'

So, for me - it looks like configuration is generally the same.
However, it results in different compile/link options :(

Any advice?

PS. Different scenarios were tried for U20, like setting CC/CXX to the corresponding values, setting export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH - no matter result was the same

Happy Winter Holidays!

@Osyotr
Copy link
Contributor

Osyotr commented Dec 24, 2023

Any advice?

If you don't need _crypt and nis modules, you can create an overlay port and remove the post-build check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
requires:more-information This Issue requires more information to solve
Projects
None yet
Development

No branches or pull requests

6 participants