Earth Mover Distance with MathOptInterface #19

davibarreira · 2020-11-28T18:22:05Z

After the discussion with @devmotion , I've changed the implementation of the "ead" and "ead2" functions. Now, these functions were implemented only with MathOptInterface. Hence, the number of dependencies were reduced. Leaving for the users to choose the optimizer to solve the LP problem (e.g. Tulip, GLPK, Gurobi...). You can discard the other pull requests, this one already has all the changes.

…lver for Earth-Mover Distance in OT

…d. 3 new tests implemented passed

…ation

…xample notebook and the tests

zsteve · 2020-12-03T18:36:56Z

Just an update that I'm planning to try merge soon. There are just a few things I need to tweak. Also I'm not able to install your branch of OptimalTransport.jl due to a library symbol issue coming from MathOptInterface. Any ideas?

(@v1.5) pkg> add MathOptInterface
   Updating registry at `~/.julia/registries/General`
   Updating git-repo `https://github.com/JuliaRegistries/General.git`
  Resolving package versions...
Updating `~/.julia/environments/v1.5/Project.toml`
  [b8f27783] + MathOptInterface v0.9.19
No Changes to `~/.julia/environments/v1.5/Manifest.toml`

julia> using MathOptInterface
[ Info: Precompiling MathOptInterface [b8f27783-ece8-5eb3-8dc8-9495eed66fee]
ERROR: LoadError: InitError: could not load library "/home/zsteve/.julia/artifacts/519367e9365948074c1fcc9f4365597f147a5ab7/lib/libmbedtls.so"
/usr/lib64/libmbedx509.so.0: undefined symbol: mbedtls_mutex_unlock
Stacktrace:
 [1] dlopen(::String, ::UInt32) at /build/julia/src/julia-1.5.3/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109
 [2] macro expansion at /home/zsteve/.julia/packages/JLLWrappers/KuIwt/src/products/library_generators.jl:61 [inlined]
 [3] __init__() at /home/zsteve/.julia/packages/MbedTLS_jll/qMb7d/src/wrappers/x86_64-linux-gnu.jl:16
 [4] top-level scope at none:2
 [5] eval at ./boot.jl:331 [inlined]
during initialization of module MbedTLS_jll
in expression starting at /home/zsteve/.julia/packages/JSONSchema/V68iq/src/JSONSchema.jl:4
ERROR: LoadError: LoadError: LoadError: Failed to precompile JSONSchema [7d188eb4-7ad8-530c-ae41-71a32a6d4692] to /home/zsteve/.julia/compiled/v1.5/JSONSchema/7Y9QQ_cj6EQ.ji.
Stacktrace:
 [1] include at ./Base.jl:368 [inlined]
 [2] include(::String) at /home/zsteve/.julia/packages/MathOptInterface/ZJFKw/src/FileFormats/FileFormats.jl:1
 [3] top-level scope at /home/zsteve/.julia/packages/MathOptInterface/ZJFKw/src/FileFormats/FileFormats.jl:13
 [4] include at ./Base.jl:368 [inlined]
 [5] include(::String) at /home/zsteve/.julia/packages/MathOptInterface/ZJFKw/src/MathOptInterface.jl:1
 [6] top-level scope at /home/zsteve/.julia/packages/MathOptInterface/ZJFKw/src/MathOptInterface.jl:147
 [7] top-level scope at none:2
 [8] eval at ./boot.jl:331 [inlined]
in expression starting at /home/zsteve/.julia/packages/MathOptInterface/ZJFKw/src/FileFormats/MOF/MOF.jl:6
in expression starting at /home/zsteve/.julia/packages/MathOptInterface/ZJFKw/src/FileFormats/FileFormats.jl:13
in expression starting at /home/zsteve/.julia/packages/MathOptInterface/ZJFKw/src/MathOptInterface.jl:147
ERROR: Failed to precompile MathOptInterface [b8f27783-ece8-5eb3-8dc8-9495eed66fee] to /home/zsteve/.julia/compiled/v1.5/MathOptInterface/tyub8_cj6EQ.ji.
Stacktrace:
 [1] run_repl(::REPL.AbstractREPL, ::Any) at /build/julia/src/julia-1.5.3/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:288

This seem to be a recent active issue that seems to affect also some other packages.
JuliaIO/HDF5.jl#751 (comment)

musm · 2020-12-03T18:38:54Z

@zsteve can you post the output of JuliaIO/HDF5.jl#751 (comment)

devmotion · 2020-12-03T19:00:04Z

Just an update that I'm planning to try merge soon.

It might be good to fix CI first, to ensure that everything is tested thoroughly (https://github.com/zsteve/OptimalTransport.jl/pull/20).

zsteve · 2020-12-03T19:19:27Z

@zsteve can you post the output of JuliaIO/HDF5.jl#751 (comment)

@musm

julia> using Libdl

julia> dllist()
38-element Array{String,1}:
 
"linux-vdso.so.1"
 "/usr/bin/../lib/libjulia.so.1"
 "/usr/bin/../lib/libdl.so.2"
 "/usr/bin/../lib/librt.so.1"
 "/usr/bin/../lib/libpthread.so.0"
 "/usr/bin/../lib/libc.so.6"
 "/usr/bin/../lib/libutf8proc.so.2"
 "/usr/bin/../lib/libunwind.so.8"
 "/usr/bin/../lib/libLLVM-10.so"
 "/usr/bin/../lib/libstdc++.so.6"
 "/usr/bin/../lib/libm.so.6"
 "/usr/bin/../lib/libgcc_s.so.1"
 "/lib64/ld-linux-x86-64.so.2"
 "/usr/bin/../lib/liblzma.so.5"
 "/usr/bin/../lib/../lib/libffi.so.7"
 "/usr/bin/../lib/../lib/libedit.so.0"
 "/usr/bin/../lib/../lib/libz.so.1"
 "/usr/bin/../lib/../lib/libncursesw.so.6"
 "/usr/bin/../lib/../lib/libxml2.so.2"
 "/usr/bin/../lib/../lib/libicuuc.so.67"
 "/usr/bin/../lib/../lib/libicudata.so.67"
 "/usr/lib/julia/sys.so"
 "/usr/bin/../lib/libpcre2-8.so"
 "/usr/bin/../lib/libgmp.so"
 "/usr/bin/../lib/libmpfr.so"
 "/usr/bin/../lib/libblas.so"
 "/usr/bin/../lib/libgomp.so.1"
 "/usr/bin/../lib/liblapack.so"
 "/usr/bin/../lib/libgfortran.so.5"
 "/usr/lib/../lib/libquadmath.so.0"
 "/usr/bin/../lib/libcholmod.so"
 "/usr/bin/../lib/libamd.so.2"
 "/usr/bin/../lib/libcolamd.so.2"
 "/usr/bin/../lib/libsuitesparseconfig.so.5"
 "/usr/bin/../lib/libccolamd.so.2"
 "/usr/bin/../lib/libcamd.so.2"
 "/usr/bin/../lib/libmetis.so"
 "/usr/bin/../lib/julia/libsuitesparse_wrapper.so"

musm · 2020-12-03T19:27:55Z

@zsteve
could you post

println.(dllist())

here or perhaps in a gist? After loading the library in question that errors.

zsteve · 2020-12-03T19:57:50Z

@musm
No problem, after using MathOptInterface which results in /usr/lib64/libmbedx509.so.0: undefined symbol: mbedtls_mutex_unlock, running
println.(dllist()) gives me

linux-vdso.so.1
/usr/bin/../lib/libjulia.so.1
/usr/bin/../lib/libdl.so.2
/usr/bin/../lib/librt.so.1
/usr/bin/../lib/libpthread.so.0
/usr/bin/../lib/libc.so.6
/usr/bin/../lib/libutf8proc.so.2
/usr/bin/../lib/libunwind.so.8
/usr/bin/../lib/libLLVM-10.so
/usr/bin/../lib/libstdc++.so.6
/usr/bin/../lib/libm.so.6
/usr/bin/../lib/libgcc_s.so.1
/lib64/ld-linux-x86-64.so.2
/usr/bin/../lib/liblzma.so.5
/usr/bin/../lib/../lib/libffi.so.7
/usr/bin/../lib/../lib/libedit.so.0
/usr/bin/../lib/../lib/libz.so.1
/usr/bin/../lib/../lib/libncursesw.so.6
/usr/bin/../lib/../lib/libxml2.so.2
/usr/bin/../lib/../lib/libicuuc.so.67
/usr/bin/../lib/../lib/libicudata.so.67
/usr/lib/julia/sys.so
/usr/bin/../lib/libpcre2-8.so
/usr/bin/../lib/libgmp.so
/usr/bin/../lib/libmpfr.so
/usr/bin/../lib/libblas.so
/usr/bin/../lib/libgomp.so.1
/usr/bin/../lib/liblapack.so
/usr/bin/../lib/libgfortran.so.5
/usr/lib/../lib/libquadmath.so.0
/usr/bin/../lib/libcholmod.so
/usr/bin/../lib/libamd.so.2
/usr/bin/../lib/libcolamd.so.2
/usr/bin/../lib/libsuitesparseconfig.so.5
/usr/bin/../lib/libccolamd.so.2
/usr/bin/../lib/libcamd.so.2
/usr/bin/../lib/libmetis.so
/usr/bin/../lib/julia/libsuitesparse_wrapper.so

If it helps, this is on Julia 1.5.3 in Arch environment. Here is the output of versioninfo()

Julia Version 1.5.3
Commit 788b2c77c1* (2020-11-09 13:37 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-10.0.1 (ORCJIT, skylake)

davibarreira · 2020-12-09T10:59:22Z

Is it working now?

zsteve · 2020-12-10T04:39:10Z

Unfortunately not, I just tried precompiling MathOptInterface again on my local machine and it's failing. It could be something to do with my local machine but I'll try to work something out in the coming days.

Apologies again for the slow response on my end, things are quite tense towards the end of this year...

davibarreira added 7 commits November 23, 2020 17:08

Implementation of functions to generate Simplex format and Simplex so…

8ce7305

…lver for Earth-Mover Distance in OT

Tests implemented and 3 methods for solve Earth-Movers Distance solve…

f6cdaa2

…d. 3 new tests implemented passed

Added more tests for function emd(). Passing in every test

27ba0dc

Created Example Notebook visualizing the effect of entropic regulariz…

262ef76

…ation

Adjusted the Dependencies

50be97d

Adjusted dependencies for the tests

02b9548

Changed the ead function to work with MathOptInterface. Updated the e…

726bbfa

…xample notebook and the tests

zsteve merged commit 10e3d89 into JuliaOptimalTransport:master Dec 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Earth Mover Distance with MathOptInterface #19

Earth Mover Distance with MathOptInterface #19

davibarreira commented Nov 28, 2020

zsteve commented Dec 3, 2020

musm commented Dec 3, 2020

devmotion commented Dec 3, 2020

zsteve commented Dec 3, 2020

musm commented Dec 3, 2020 •

edited

Loading

zsteve commented Dec 3, 2020 •

edited

Loading

davibarreira commented Dec 9, 2020

zsteve commented Dec 10, 2020

Earth Mover Distance with MathOptInterface #19

Earth Mover Distance with MathOptInterface #19

Conversation

davibarreira commented Nov 28, 2020

zsteve commented Dec 3, 2020

musm commented Dec 3, 2020

devmotion commented Dec 3, 2020

zsteve commented Dec 3, 2020

musm commented Dec 3, 2020 • edited Loading

zsteve commented Dec 3, 2020 • edited Loading

davibarreira commented Dec 9, 2020

zsteve commented Dec 10, 2020

musm commented Dec 3, 2020 •

edited

Loading

zsteve commented Dec 3, 2020 •

edited

Loading