Cannot compile [email protected]+1 #751

tbenst · 2020-11-27T23:00:35Z

Hoping that someone can help me understand what changed on the HDF5.jl side between 0.13.6 and 0.14.1--perhaps this issue may originate with BinaryBuilder.jl?

Compiles fine:

[[HDF5]]
deps = ["Blosc", "HDF5_jll", "Libdl", "Mmap", "Random"]
git-tree-sha1 = "0713cbabdf855852dfab3ce6447c87145f3d9ea8"
uuid = "f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f"
version = "0.13.6"

[[HDF5_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Zlib_jll"]
git-tree-sha1 = "bc9c3d43ffd4d8988bfa372b86d4bdbd26860e95"
uuid = "0234f1f7-429e-5d53-9886-15a909be8d59"
version = "1.10.5+7"

Fails to compile:

[[HDF5]]
deps = ["Blosc", "Compat", "HDF5_jll", "Libdl", "Mmap", "Random", "Requires"]
git-tree-sha1 = "96d77533eb46e208e801b939db8a27626c166565"
uuid = "f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f"
version = "0.14.1"

[[HDF5_jll]]
deps = ["Artifacts", "JLLWrappers", "LibCURL_jll", "Libdl", "OpenSSL_jll", "Pkg", "Zlib_jll"]
git-tree-sha1 = "fd83fa0bde42e01952757f01149dd968c06c4dba"
uuid = "0234f1f7-429e-5d53-9886-15a909be8d59"
version = "1.12.0+1"

The error:

julia> using HDF5
[ Info: Precompiling HDF5 [f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f]
ERROR: LoadError: LoadError: InitError: could not load library "/home/tyler/.julia/artifacts/519367e9365948074c1fcc9f4365597f147a5ab7/lib/libmbedtls.so"
/nix/store/7xkbxjxaryrz8ka1padvgh61ljfn85cr-mbedtls-2.16.3/lib/libmbedx509.so.0: undefined symbol: mbedtls_mutex_unlock
Stacktrace:
 [1] dlopen(::String, ::UInt32; throw_error::Bool) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [2] dlopen(::String, ::UInt32) at /build/source/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109
 [3] macro expansion at /home/tyler/.julia/packages/JLLWrappers/KuIwt/src/products/library_generators.jl:61 [inlined]
 [4] __init__() at /home/tyler/.julia/packages/MbedTLS_jll/qMb7d/src/wrappers/x86_64-linux-gnu.jl:16
 [5] _include_from_serialized(::String, ::Array{Any,1}) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [6] _require_search_from_serialized(::Base.PkgId, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [7] _require(::Base.PkgId) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [8] require(::Base.PkgId) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:? (repeats 2 times)
 [9] include(::Function, ::Module, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [10] include at ./Base.jl:368 [inlined]
 [11] include(::String) at /home/tyler/.julia/packages/LibSSH2_jll/T20dp/src/LibSSH2_jll.jl:1
 [12] top-level scope at /home/tyler/.julia/packages/LibSSH2_jll/T20dp/src/LibSSH2_jll.jl:54
 [13] include(::Function, ::Module, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:? (repeats 2 times)

 [14] top-level scope at none:2
 [15] eval at ./boot.jl:331 [inlined]
 [16] eval(::Expr) at ./client.jl:467
 [17] top-level scope at ./none:3
during initialization of module MbedTLS_jll
in expression starting at /home/tyler/.julia/packages/LibSSH2_jll/T20dp/src/wrappers/x86_64-linux-gnu.jl:4
in expression starting at /home/tyler/.julia/packages/LibSSH2_jll/T20dp/src/LibSSH2_jll.jl:47
ERROR: LoadError: LoadError: Failed to precompile LibSSH2_jll [29816b5a-b9ab-546f-933c-edad1886dfa8] to /home/tyler/.julia/compiled/v1.5/LibSSH2_jll/K6mup_kELNb.ji.
Stacktrace:
 [1] error(::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [2] compilecache(::Base.PkgId, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [3] _require(::Base.PkgId) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [4] require(::Base.PkgId) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:? (repeats 2 times)
 [5] include(::Function, ::Module, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [6] include at ./Base.jl:368 [inlined]
 [7] include(::String) at /home/tyler/.julia/packages/LibCURL_jll/9bFfE/src/LibCURL_jll.jl:1
 [8] top-level scope at /home/tyler/.julia/packages/LibCURL_jll/9bFfE/src/LibCURL_jll.jl:54
 [9] include(::Function, ::Module, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:? (repeats 2 times)
 [10] top-level scope at none:2
 [11] eval at ./boot.jl:331 [inlined]
 [12] eval(::Expr) at ./client.jl:467
 [13] top-level scope at ./none:3
in expression starting at /home/tyler/.julia/packages/LibCURL_jll/9bFfE/src/wrappers/x86_64-linux-gnu.jl:4
in expression starting at /home/tyler/.julia/packages/LibCURL_jll/9bFfE/src/LibCURL_jll.jl:47
ERROR: LoadError: LoadError: Failed to precompile LibCURL_jll [deac9b47-8bc7-5906-a0fe-35ac56dc84c0] to /home/tyler/.julia/compiled/v1.5/LibCURL_jll/9JWaY_kELNb.ji.
Stacktrace:
 [1] error(::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [2] compilecache(::Base.PkgId, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [3] _require(::Base.PkgId) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [4] require(::Base.PkgId) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:? (repeats 2 times)
 [5] include(::Function, ::Module, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:? (repeats 2 times)
 [6] top-level scope at /home/tyler/.julia/packages/JLLWrappers/KuIwt/src/toplevel_generators.jl:170
 [7] include(::Function, ::Module, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:? (repeats 2 times)
 [8] top-level scope at none:2
 [9] eval at ./boot.jl:331 [inlined]
 [10] eval(::Expr) at ./client.jl:467
 [11] top-level scope at ./none:3
in expression starting at /home/tyler/.julia/packages/HDF5_jll/BGk9m/src/wrappers/x86_64-linux-gnu.jl:6
in expression starting at /home/tyler/.julia/packages/HDF5_jll/BGk9m/src/HDF5_jll.jl:8
ERROR: LoadError: LoadError: Failed to precompile HDF5_jll [0234f1f7-429e-5d53-9886-15a909be8d59] to /home/tyler/.julia/compiled/v1.5/HDF5_jll/w9xrx_kELNb.ji.
Stacktrace:
 [1] error(::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [2] compilecache(::Base.PkgId, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [3] _require(::Base.PkgId) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [4] require(::Base.PkgId) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:? (repeats 2 times)
 [5] include(::Function, ::Module, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [6] include at ./Base.jl:368 [inlined]
 [7] include(::String) at /home/tyler/.julia/packages/HDF5/LW5kO/src/HDF5.jl:1
 [8] top-level scope at /home/tyler/.julia/packages/HDF5/LW5kO/src/HDF5.jl:43
 [9] include(::Function, ::Module, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:? (repeats 2 times)
 [10] top-level scope at none:2
 [11] eval at ./boot.jl:331 [inlined]
 [12] eval(::Expr) at ./client.jl:467
 [13] top-level scope at ./none:3
in expression starting at /home/tyler/.julia/packages/HDF5/LW5kO/deps/deps.jl:3
in expression starting at /home/tyler/.julia/packages/HDF5/LW5kO/src/HDF5.jl:42
ERROR: Failed to precompile HDF5 [f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f] to /home/tyler/.julia/compiled/v1.5/HDF5/L7Dga_kELNb.ji.
Stacktrace:
 [1] error(::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [2] compilecache(::Base.PkgId, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [3] _require(::Base.PkgId) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [4] require(::Base.PkgId) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:? (repeats 2 times)

Further testing reveals that the issue arises from HDF5_jll at 1.12.0+1, and not HDF5.jl.

I believe this may have to do with the addition of LibCURL_jll and OpenSSL_jll...? It's not immediately clear how/why these libraries are needed? Many thanks for any thoughts!

Edit: possibly related: https://discourse.julialang.org/t/mbedtls-1-0-2-failed-to-load-on-julia-1-5-2-solved-by-downgrading-to-mbedtls-0-7-0/47430/7

The text was updated successfully, but these errors were encountered:

musm · 2020-11-27T23:15:48Z

Could you post your versioninfo() ?

tbenst · 2020-11-27T23:41:34Z

sure,

julia> versioninfo()
Julia Version 1.5.3
Commit 788b2c77c1* (2020-11-09 13:37 UTC)
Platform Info:
  OS: Linux (x86_64-unknown-linux-gnu)
  CPU: Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, cascadelake)
Environment:
  JULIA_PKGDIR =

giordano · 2020-11-28T00:44:05Z

What distribution is that?

tbenst · 2020-11-28T09:04:06Z

It’s NixOS, a fully declarative and reproducible distro of Linux based on the Nix package manager. In order to be reproducible, all dependencies must be declared (but that’s our job ;). If you're curious and want to get a flavor of it, here's how we package and compile julia.

My system mbedtls is at 2.16.3. julia is using:

[[MbedTLS]]
deps = ["Dates", "MbedTLS_jll", "Random", "Sockets"]
git-tree-sha1 = "1c38e51c3d08ef2278062ebceade0e46cefc96fe"
uuid = "739be429-bea8-5141-9913-cc70e7f3736d"
version = "1.0.3"

[[MbedTLS_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "0eef589dd1c26a3ac9d753fe1a8bcad63f956fa6"
uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1"
version = "2.16.8+1"

The mbedtls_mutex_lock symbol seems to be listed in system libmbedcrypto:

$ nm -D /nix/store/7xkbxjxaryrz8ka1padvgh61ljfn85cr-mbedtls-2.16.3/lib/libmbedcrypto.so | grep mbedtls_mutex_lock
0000000000061078 D mbedtls_mutex_lock

musm · 2020-12-03T18:08:54Z

Isn't that an MbedTLS_jll issue not HDF5_jl?

jmert · 2020-12-03T18:13:43Z

Yeah, I was just looking at this as well. I think it's an issue upstream somewhere since as far as I can tell, the MbedTLS_jll version of libmbedx509.so doesn't have an mbedtls_mutex_unlock that it needs resolved:

$ nm -D /home/justin/.julia/artifacts/519367e9365948074c1fcc9f4365597f147a5ab7/lib/libmbedx509.so | grep unlock

versus what I see in my system's copy (Arch linux):

$ nm -D /usr/lib/libmbedx509.so | grep unlock
                 U mbedtls_mutex_unlock

My guess is that something in the dependency chain is accidentally picking up a system library rather than the artifact libraries. That's not an issue in HDF5/HDF5_jll, so this issue should get moved somewhere else in the chain. (I'm unfortunately not super well versed in jll packages yet...)

giordano · 2020-12-03T18:17:46Z

You can see with

using Libdl
dllist()

what libraries are loaded

musm · 2020-12-03T18:18:11Z

This might be due to JuliaLang/julia#38409 (comment) and fixed by JuliaLang/julia#38409 ?

giordano · 2020-12-03T18:18:39Z

This might be due to JuliaLang/julia#38409 (comment) and fixed by JuliaLang/julia#38409 ?

I don't think it's related in any way

musm · 2020-12-03T18:20:19Z

😄

So MbedTLS_jll is built incorrectly and missing mbedtls_mutex_unlock ?

giordano · 2020-12-03T18:24:13Z

My guess is that something in the dependency chain is accidentally picking up a system library rather than the artifact libraries.

musm · 2020-12-03T18:30:53Z

Makes sense now. Ok I'm closing this. This looks like it might be a bug in the build system dependency chain locater itself then and should be moved to https://github.com/JuliaPackaging/BinaryBuilder.jl/issues

giordano · 2020-12-03T18:33:19Z

No, it's some other packages loading system libraries instead those served through artifacts

giordano · 2020-12-03T18:34:48Z

This is why I asked for dllist. Usually are Python packages that pull external dependencies

jmert · 2020-12-03T18:49:12Z

Storing and then grepping through the output of LD_DEBUG=bindings,libs julia -e 'using MbedTLS_jll' might also be illuminating to narrow down on what is triggering the load of the system library; presumably it should be happening before the artifact libraries are loaded by MbedTLS_jll.__init__().

musm · 2020-12-03T18:57:44Z

@tbenst, @zsteve do you mind posting the output of

$ LD_DEBUG=bindings,libs julia -e 'using MbedTLS_jll' 2>&1 |  grep libmbedtls_*

and/or

using Libdl
dllist()

So we can determine what's causing this issue?

giordano · 2020-12-03T18:58:59Z

dllist() is usually long and the output will be truncated, something like println.(dllist()) or similar is better if you want to share the output

jmert · 2020-12-03T18:59:56Z

I was using filter(contains("mbed"), dllist()) while playing around with what happens on my system, and that works pretty well.

giordano · 2020-12-03T19:01:05Z

I'm a bit sceptical LD_DEBUG=bindings,libs julia -e 'using MbedTLS_jll' 2>&1 | grep libmbedtls_* can be any useful, usually other packages pull in other libraries (most of the time Python packages called by PyCall), and that's the problem.

jmert · 2020-12-03T19:02:17Z

Yeah, I think you'd need all of the context of LD_DEBUG for interpretation to be useful, so it'd have to be uploaded as a very large gist or to be pre-interpreted and have only the relevant context posted.

musm closed this as completed Dec 3, 2020

musm mentioned this issue Dec 3, 2020

MbedTLS dependency chain is accidentally picking up a system library rather than artifact libraries JuliaPackaging/BinaryBuilder.jl#973

Closed

zsteve mentioned this issue Dec 3, 2020

Earth Mover Distance with MathOptInterface JuliaOptimalTransport/OptimalTransport.jl#19

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot compile [email protected]+1 #751

Cannot compile [email protected]+1 #751

tbenst commented Nov 27, 2020 •

edited

Loading

musm commented Nov 27, 2020 •

edited

Loading

tbenst commented Nov 27, 2020

giordano commented Nov 28, 2020

tbenst commented Nov 28, 2020

musm commented Dec 3, 2020

jmert commented Dec 3, 2020 •

edited

Loading

giordano commented Dec 3, 2020 •

edited

Loading

musm commented Dec 3, 2020 •

edited

Loading

giordano commented Dec 3, 2020 •

edited

Loading

musm commented Dec 3, 2020

giordano commented Dec 3, 2020

musm commented Dec 3, 2020

giordano commented Dec 3, 2020

giordano commented Dec 3, 2020

jmert commented Dec 3, 2020 •

edited

Loading

musm commented Dec 3, 2020

giordano commented Dec 3, 2020

jmert commented Dec 3, 2020

giordano commented Dec 3, 2020 •

edited

Loading

jmert commented Dec 3, 2020

Cannot compile [email protected]+1 #751

Cannot compile [email protected]+1 #751

Comments

tbenst commented Nov 27, 2020 • edited Loading

musm commented Nov 27, 2020 • edited Loading

tbenst commented Nov 27, 2020

giordano commented Nov 28, 2020

tbenst commented Nov 28, 2020

musm commented Dec 3, 2020

jmert commented Dec 3, 2020 • edited Loading

giordano commented Dec 3, 2020 • edited Loading

musm commented Dec 3, 2020 • edited Loading

giordano commented Dec 3, 2020 • edited Loading

musm commented Dec 3, 2020

giordano commented Dec 3, 2020

musm commented Dec 3, 2020

giordano commented Dec 3, 2020

giordano commented Dec 3, 2020

jmert commented Dec 3, 2020 • edited Loading

musm commented Dec 3, 2020

giordano commented Dec 3, 2020

jmert commented Dec 3, 2020

giordano commented Dec 3, 2020 • edited Loading

jmert commented Dec 3, 2020

tbenst commented Nov 27, 2020 •

edited

Loading

musm commented Nov 27, 2020 •

edited

Loading

jmert commented Dec 3, 2020 •

edited

Loading

giordano commented Dec 3, 2020 •

edited

Loading

musm commented Dec 3, 2020 •

edited

Loading

giordano commented Dec 3, 2020 •

edited

Loading

jmert commented Dec 3, 2020 •

edited

Loading

giordano commented Dec 3, 2020 •

edited

Loading