Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes warnings during REPL startup while using custom SYSIMG's #350

Closed

Conversation

bmharsha
Copy link

@bmharsha bmharsha commented Aug 4, 2021

We encounter following warning during Julia REPL startup if we include KernelFunctions in the default SYSIMG, this commit fixes that issue

┌ Warning: Error requiring `PDMats` from `KernelFunctions`
│   exception =
│    SystemError: opening file "/home/bmharsha/.julia/packages/KernelFunctions/AxuTC/src/matrix/kernelpdmat.jl": No such file or directory
│    Stacktrace:
│      [1] systemerror(p::String, errno::Int32; extrainfo::Nothing)
│        @ Base ./error.jl:168
│      [2] #systemerror#62
│        @ ./error.jl:167 [inlined]
│      [3] systemerror
│        @ ./error.jl:167 [inlined]
│      [4] open(fname::String; lock::Bool, read::Nothing, write::Nothing, create::Nothing, truncate::Nothing, append::Nothing)
│        @ Base ./iostream.jl:293
│      [5] open
│        @ ./iostream.jl:282 [inlined]
│      [6] open(f::Base.var"#326#327"{String}, args::String; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
│        @ Base ./io.jl:328
│      [7] open
│        @ ./io.jl:328 [inlined]
│      [8] read
│        @ ./io.jl:434 [inlined]
│      [9] _include(mapexpr::Function, mod::Module, _path::String)
│        @ Base ./loading.jl:1166
│     [10] include(mod::Module, _path::String)
│        @ Base ./Base.jl:386
│     [11] include(x::String)
│        @ KernelFunctions ~/.julia/packages/KernelFunctions/AxuTC/src/KernelFunctions.jl:1
│     [12] top-level scope
│        @ ~/.julia/packages/KernelFunctions/AxuTC/src/KernelFunctions.jl:124
│     [13] eval
│        @ ./boot.jl:360 [inlined]
│     [14] eval
│        @ ~/.julia/packages/KernelFunctions/AxuTC/src/KernelFunctions.jl:1 [inlined]
│     [15] (::KernelFunctions.var"#209#215")()
│        @ KernelFunctions ~/.julia/packages/Requires/7Ncym/src/require.jl:99
│     [16] err(f::Any, listener::Module, modname::String)
│        @ Requires ~/.julia/packages/Requires/7Ncym/src/require.jl:47
│     [17] (::KernelFunctions.var"#208#214")()
│        @ KernelFunctions ~/.julia/packages/Requires/7Ncym/src/require.jl:98
│     [18] withpath(f::Any, path::String)
│        @ Requires ~/.julia/packages/Requires/7Ncym/src/require.jl:37
│     [19] (::KernelFunctions.var"#207#213")()
│        @ KernelFunctions ~/.julia/packages/Requires/7Ncym/src/require.jl:97
│     [20] listenpkg(f::Any, pkg::Base.PkgId)
│        @ Requires ~/.julia/packages/Requires/7Ncym/src/require.jl:20
│     [21] macro expansion
│        @ ~/.julia/packages/Requires/7Ncym/src/require.jl:95 [inlined]
│     [22] __init__()
│        @ KernelFunctions ~/.julia/packages/KernelFunctions/AxuTC/src/KernelFunctions.jl:123
└ @ Requires ~/.julia/packages/Requires/7Ncym/src/require.jl:49

We are encountering following warning during Julia REPL startup if we include `KernelFunctions` in the default SYSIMG, this commit fixes that issue

```
┌ Warning: Error requiring `PDMats` from `KernelFunctions`
│   exception =
│    SystemError: opening file "/home/bmharsha/.julia/packages/KernelFunctions/AxuTC/src/matrix/kernelpdmat.jl": No such file or directory
│    Stacktrace:
│      [1] systemerror(p::String, errno::Int32; extrainfo::Nothing)
│        @ Base ./error.jl:168
│      [2] #systemerror#62
│        @ ./error.jl:167 [inlined]
│      [3] systemerror
│        @ ./error.jl:167 [inlined]
│      [4] open(fname::String; lock::Bool, read::Nothing, write::Nothing, create::Nothing, truncate::Nothing, append::Nothing)
│        @ Base ./iostream.jl:293
│      [5] open
│        @ ./iostream.jl:282 [inlined]
│      [6] open(f::Base.var"JuliaGaussianProcesses#326#327"{String}, args::String; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
│        @ Base ./io.jl:328
│      [7] open
│        @ ./io.jl:328 [inlined]
│      [8] read
│        @ ./io.jl:434 [inlined]
│      [9] _include(mapexpr::Function, mod::Module, _path::String)
│        @ Base ./loading.jl:1166
│     [10] include(mod::Module, _path::String)
│        @ Base ./Base.jl:386
│     [11] include(x::String)
│        @ KernelFunctions ~/.julia/packages/KernelFunctions/AxuTC/src/KernelFunctions.jl:1
│     [12] top-level scope
│        @ ~/.julia/packages/KernelFunctions/AxuTC/src/KernelFunctions.jl:124
│     [13] eval
│        @ ./boot.jl:360 [inlined]
│     [14] eval
│        @ ~/.julia/packages/KernelFunctions/AxuTC/src/KernelFunctions.jl:1 [inlined]
│     [15] (::KernelFunctions.var"JuliaGaussianProcesses#209#215")()
│        @ KernelFunctions ~/.julia/packages/Requires/7Ncym/src/require.jl:99
│     [16] err(f::Any, listener::Module, modname::String)
│        @ Requires ~/.julia/packages/Requires/7Ncym/src/require.jl:47
│     [17] (::KernelFunctions.var"JuliaGaussianProcesses#208#214")()
│        @ KernelFunctions ~/.julia/packages/Requires/7Ncym/src/require.jl:98
│     [18] withpath(f::Any, path::String)
│        @ Requires ~/.julia/packages/Requires/7Ncym/src/require.jl:37
│     [19] (::KernelFunctions.var"JuliaGaussianProcesses#207#213")()
│        @ KernelFunctions ~/.julia/packages/Requires/7Ncym/src/require.jl:97
│     [20] listenpkg(f::Any, pkg::Base.PkgId)
│        @ Requires ~/.julia/packages/Requires/7Ncym/src/require.jl:20
│     [21] macro expansion
│        @ ~/.julia/packages/Requires/7Ncym/src/require.jl:95 [inlined]
│     [22] __init__()
│        @ KernelFunctions ~/.julia/packages/KernelFunctions/AxuTC/src/KernelFunctions.jl:123
└ @ Requires ~/.julia/packages/Requires/7Ncym/src/require.jl:49
```
@st--
Copy link
Member

st-- commented Aug 5, 2021

I'm happy with the change to Requires.@include in principle; I'm just wondering if replacing the joinpath with explicit / separation will work under Windows (I suppose we could temporarily revive the windows-based CI runs)? Due to the way the macro works, it only supports string literals as it is.

@bmharsha
Copy link
Author

bmharsha commented Aug 5, 2021

I'm just wondering if replacing the joinpath with explicit / separation will work under Windows (I suppose we could temporarily revive the windows-based CI runs)?

Yes, it does work under Windows, here is an examples where / is being used along with Requires.@include

https://github.com/FluxML/Zygote.jl/blob/master/src/Zygote.jl#L68

@bmharsha
Copy link
Author

bmharsha commented Aug 5, 2021

I ran KernelFunctions tests on Windows, tests passed.

(@v1.6) pkg> st
      Status `C:\Users\non admin\.julia\environments\v1.6\Project.toml`
  [ec8451be] KernelFunctions v0.10.11 `https://github.com/bmharsha/KernelFunctions.jl#harsha-sysimg`

(@v1.6) pkg> test KernelFunctions
     Testing KernelFunctions
  Downloaded artifact: LLVMExtra
      Status `C:\Users\non admin\AppData\Local\Temp\jl_C1ecjk\Project.toml`
  [39de3d68] AxisArrays v0.4.4
  [b4f34e82] Distances v0.10.3
  [e30172f5] Documenter v0.27.5
  [26cc04aa] FiniteDifferences v0.12.18
  [587475ba] Flux v0.12.6
  [f6369f11] ForwardDiff v0.10.19
  [ec8451be] KernelFunctions v0.10.11 `https://github.com/bmharsha/KernelFunctions.jl#harsha-sysimg`
  [2c470bb0] Kronecker v0.4.4
  [2ab3a3ac] LogExpFunctions v0.2.5
  [90014a1f] PDMats v0.11.1
  [37e2e3b7] ReverseDiff v1.9.0
  [276daf66] SpecialFunctions v1.6.0
  [e88e6eb3] Zygote v0.6.17
  [37e2e46d] LinearAlgebra `@stdlib/LinearAlgebra`
  [9a3f8284] Random `@stdlib/Random`
  [8dfed614] Test `@stdlib/Test`
      Status `C:\Users\non admin\AppData\Local\Temp\jl_C1ecjk\Manifest.toml`
  [a4c015fc] ANSIColoredPrinters v0.0.1
  [621f4979] AbstractFFTs v1.0.1
  [1520ce14] AbstractTrees v0.3.4
  [79e6a3ab] Adapt v3.3.1
  [4fba245c] ArrayInterface v3.1.22
  [39de3d68] AxisArrays v0.4.4
  [ab4f0b2a] BFloat16s v0.1.0
  [fa961155] CEnum v0.4.1
  [052768ef] CUDA v3.3.4
  [082447d4] ChainRules v0.8.24
  [d360d2e6] ChainRulesCore v0.10.13
  [944b1d66] CodecZlib v0.7.0
  [3da002f7] ColorTypes v0.11.0
  [5ae59095] Colors v0.12.8
  [bbf7d656] CommonSubexpressions v0.3.0
  [34da2185] Compat v3.32.0
  [a33af91c] CompositionsBase v0.1.1
  [9a962f9c] DataAPI v1.7.0
  [864edb3b] DataStructures v0.18.9
  [163ba53b] DiffResults v1.0.3
  [b552c78f] DiffRules v1.2.1
  [b4f34e82] Distances v0.10.3
  [ffbed154] DocStringExtensions v0.8.5
  [e30172f5] Documenter v0.27.5
  [da5c29d0] EllipsisNotation v1.1.0
  [e2ba6199] ExprTools v0.1.6
  [1a297f60] FillArrays v0.12.1
  [26cc04aa] FiniteDifferences v0.12.18
  [53c48c17] FixedPointNumbers v0.8.4
  [587475ba] Flux v0.12.6
  [f6369f11] ForwardDiff v0.10.19
  [069b7b12] FunctionWrappers v1.1.2
  [d9f16b24] Functors v0.2.3
  [0c68f7d7] GPUArrays v7.0.1
  [61eb1bfa] GPUCompiler v0.12.8
  [b5f81e59] IOCapture v0.2.2
  [7869d1d1] IRTools v0.4.3
  [615f187c] IfElse v0.1.0
  [8197267c] IntervalSets v0.5.3
  [c8e1da08] IterTools v1.3.0
  [692b3bcd] JLLWrappers v1.3.0
  [682c06a0] JSON v0.21.1
  [e5e0dc1b] Juno v0.8.4
  [ec8451be] KernelFunctions v0.10.11 `https://github.com/bmharsha/KernelFunctions.jl#harsha-sysimg`
  [2c470bb0] Kronecker v0.4.4
  [929cbde3] LLVM v4.2.0
  [2ab3a3ac] LogExpFunctions v0.2.5
  [1914dd2f] MacroTools v0.5.7
  [e89f7d12] Media v0.5.0
  [e1d29d7a] Missings v1.0.0
  [872c559c] NNlib v0.7.27
  [a00861dc] NNlibCUDA v0.1.7
  [77ba4419] NaNMath v0.3.5
  [356022a1] NamedDims v0.2.35
  [bac558e1] OrderedCollections v1.4.1
  [90014a1f] PDMats v0.11.1
  [69de0a69] Parsers v1.1.2
  [21216c6a] Preferences v1.2.2
  [74087812] Random123 v1.4.2
  [e6cf234a] RandomNumbers v1.5.3
  [b3c3ace0] RangeArrays v0.3.2
  [189a3867] Reexport v1.1.0
  [ae029012] Requires v1.1.3
  [37e2e3b7] ReverseDiff v1.9.0
  [708f8203] Richardson v1.4.0
  [79098fc4] Rmath v0.7.0
  [a2af1166] SortingAlgorithms v1.0.1
  [276daf66] SpecialFunctions v1.6.0
  [aedffcd0] Static v0.3.0
  [90137ffa] StaticArrays v1.2.10
  [82ae8749] StatsAPI v1.0.0
  [2913bbd2] StatsBase v0.33.9
  [4c63d2b9] StatsFuns v0.9.8
  [62fd8b95] TensorCore v0.1.1
  [a759f4b9] TimerOutputs v0.5.12
  [3bb67fe8] TranscodingStreams v0.9.5
  [a5390f91] ZipFile v0.9.3
  [e88e6eb3] Zygote v0.6.17
  [700de1a5] ZygoteRules v0.2.1
  [dad2f222] LLVMExtra_jll v0.0.7+0
  [efe28fd5] OpenSpecFun_jll v0.5.5+0
  [f50d1b31] Rmath_jll v0.3.0+0
  [0dad84c5] ArgTools `@stdlib/ArgTools`
  [56f22d72] Artifacts `@stdlib/Artifacts`
  [2a0f44e3] Base64 `@stdlib/Base64`
  [ade2ca70] Dates `@stdlib/Dates`
  [8bb1440f] DelimitedFiles `@stdlib/DelimitedFiles`
  [8ba89e20] Distributed `@stdlib/Distributed`
  [f43a241f] Downloads `@stdlib/Downloads`
  [b77e0a4c] InteractiveUtils `@stdlib/InteractiveUtils`
  [4af54fe1] LazyArtifacts `@stdlib/LazyArtifacts`
  [b27032c2] LibCURL `@stdlib/LibCURL`
  [76f85450] LibGit2 `@stdlib/LibGit2`
  [8f399da3] Libdl `@stdlib/Libdl`
  [37e2e46d] LinearAlgebra `@stdlib/LinearAlgebra`
  [56ddb016] Logging `@stdlib/Logging`
  [d6f4376e] Markdown `@stdlib/Markdown`
  [a63ad114] Mmap `@stdlib/Mmap`
  [ca575930] NetworkOptions `@stdlib/NetworkOptions`
  [44cfe95a] Pkg `@stdlib/Pkg`
  [de0858da] Printf `@stdlib/Printf`
  [9abbd945] Profile `@stdlib/Profile`
  [3fa0cd96] REPL `@stdlib/REPL`
  [9a3f8284] Random `@stdlib/Random`
  [ea8e919c] SHA `@stdlib/SHA`
  [9e88b42a] Serialization `@stdlib/Serialization`
  [1a1011a3] SharedArrays `@stdlib/SharedArrays`
  [6462fe0b] Sockets `@stdlib/Sockets`
  [2f01184e] SparseArrays `@stdlib/SparseArrays`
  [10745b16] Statistics `@stdlib/Statistics`
  [4607b0f0] SuiteSparse `@stdlib/SuiteSparse`
  [fa267f1f] TOML `@stdlib/TOML`
  [a4e569a6] Tar `@stdlib/Tar`
  [8dfed614] Test `@stdlib/Test`
  [cf7118a7] UUIDs `@stdlib/UUIDs`
  [4ec0a83e] Unicode `@stdlib/Unicode`
  [e66e0078] CompilerSupportLibraries_jll `@stdlib/CompilerSupportLibraries_jll`
  [deac9b47] LibCURL_jll `@stdlib/LibCURL_jll`
  [29816b5a] LibSSH2_jll `@stdlib/LibSSH2_jll`
  [c8ffd9c3] MbedTLS_jll `@stdlib/MbedTLS_jll`
  [14a3606d] MozillaCACerts_jll `@stdlib/MozillaCACerts_jll`
  [83775a58] Zlib_jll `@stdlib/Zlib_jll`
  [8e850ede] nghttp2_jll `@stdlib/nghttp2_jll`
  [3f19e933] p7zip_jll `@stdlib/p7zip_jll`
Precompiling project...
  55 dependencies successfully precompiled in 107 seconds (31 already precompiled)
     Testing Running tests...
[ Info: Packages Loaded
         [ Info: Ran tests on Transform
         ┌ Warning: `convert(::Type{<:Tuple}, t::Tangent{<:Any, <:Tuple})` is deprecated, use `backing(t)` instead.
│   caller = wrap_chainrules_output at chainrules.jl:66 [inlined]
└ @ Core C:\Users\non admin\.julia\packages\Zygote\TaBlo\src\compiler\chainrules.jl:66
     [ Info: Ran tests on BaseKernel
[ Info: Ran tests on Kernel
[ Info: Ran tests on Multi-Output Kernels
[ Info: Ran tests on Distances
[ Info: Ran tests on matrix
┌ Warning: `InplaceableThunk(t::Thunk, add!)` is deprecated, use `InplaceableThunk(add!, t)` instead.
│   caller = ip:0x0
└ @ Core :-1
┌ Warning: `convert(::Type{<:NamedTuple}, t::Tangent{<:Any, <:NamedTuple})` is deprecated, use `backing(t)` instead.
│   caller = wrap_chainrules_output at chainrules.jl:66 [inlined]
└ @ Core C:\Users\non admin\.julia\packages\Zygote\TaBlo\src\compiler\chainrules.jl:66
[ Info: SetupBuildDirectory: setting up build directory.
[ Info: Doctest: running doctests.
[ Info: Skipped ExpandTemplates step (doctest only).
[ Info: Skipped CrossReferences step (doctest only).
[ Info: Skipped CheckDocument step (doctest only).
[ Info: Skipped Populate step (doctest only).
[ Info: Skipped RenderDocument step (doctest only).
Test Summary:   | Pass  Broken  Total
KernelFunctions | 4798      10   4808
     Testing KernelFunctions tests passed

julia> versioninfo()
Julia Version 1.6.2
Commit 1b93d53fc4 (2021-07-14 15:36 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, broadwell)
Environment:
  

@devmotion
Copy link
Member

I am surprised that/if this is needed. Requires should replace all occurrences of include with @include automatically: JuliaPackaging/Requires.jl#85 (comment) If it does not work, maybe it should be fixed upstream.

@st--
Copy link
Member

st-- commented Aug 9, 2021

I am surprised that/if this is needed. Requires should replace all occurrences of include with @include automatically: JuliaPackaging/Requires.jl#85 (comment) If it does not work, maybe it should be fixed upstream.

Could it be that the issue isn't @include vs include() but the availability of a compile-time path (string literal) vs only constructing the path at runtime (joinpath)?

@bmharsha can you share a minimal setup for how to reproduce the warning message you encounter?

@bmharsha
Copy link
Author

bmharsha commented Aug 9, 2021

@bmharsha can you share a minimal setup for how to reproduce the warning message you encounter?

Sure, here are the minimal steps required to reproduce the issue

  • Execute following code in Julia REPL (Preferably in a clean environment)
using Pkg
Pkg.add("KernelFunctions")
Pkg.add("PackageCompiler")
Pkg.add("PDMats")
using PackageCompiler
const cpu_target = "generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)"
PackageCompiler.create_sysimage(
           [:KernelFunctions, :PDMats];         
           sysimage_path="Sysimg.so",
           incremental=true,
           cpu_target=cpu_target)
  • Once the SYSIMG is created, exit Julia and relocate/delete the packages folder from DEPOT_PATH. Example:- mv ~/.julia/packages .
  • Start Julia with the newly created SYSIMG using following command: ./julia-1.6.2/bin/julia -JSysimg.so

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants