Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: remove Gurobi requirement #106

Merged
merged 8 commits into from
Feb 5, 2021
Merged

Conversation

danielolsen
Copy link
Contributor

Purpose

Removes the requirement that Gurobi.jl be installed as a requirement of REISE.jl, since it can now be run with any optimizer (Closes #101). Allows the user to pass solver settings from python (Closes #102).

What is the code doing

  • In src/solver_specific/gurobi.jl: we move all code that depends on importing Gurobi (from REISE.jl and loop.jl)
    • run_scenario_gurobi wraps around run_scenario as defined in src/REISE.jl.
    • The new function definition with the call signature new_model(env::Gurobi.Env) creates another method for the new_model function defined in loop.jl; the one in loop.jl will take any type, but if we pass a Gurobi.Env object then Julia will use the most specific method available, the one defined in gurobi.jl (see https://docs.julialang.org/en/v1/manual/methods/ for more details).
  • In src/REISE.jl:
    • We remove the Gurobi import, instead importing Requires, and using this package to only include src/solver_specific/gurobi.jl if Gurobi is also imported.
    • We allow a dictionary solver_kwargs to be passed to run_scenario.
    • We return the JuMP.Model instead that we built in interval_loop so that when we are running with the shared gurobi environment in we can free the Gurobi environment in the most proper way according to their documentation.
  • In src/loop.jl, in interval_loop:
    • We refactor how we check which solver is being used to a more general form.
    • We return the JuMP.Model instance, to get returned via run_scenario and cleaned up as necessary in run_scenario_gurobi
  • In pyreisejl/utility/call.py:
    • We refactor the launch_scenario function into a class with methods, and we define a gurobi-specific subclass that does gurobi-specific things.
    • We refactor main to use this new object-based launch method.

Testing

Julia

From Julia, we can verify that this works by creating fresh environments, installing REISE.jl along with with different solvers (note that Gurobi.jl is not installed by default), and running as desired:

With Gurobi.jl
julia> import Pkg

julia> Pkg.activate("Gurobitest")
 Activating new environment at `C:\Users\DanielOlsen\repos\bes\Gurobitest\Project.toml`

julia> Pkg.develop(path=raw"C:\Users\DanielOlsen\repos\bes\REISE.jl")
Path `C:\Users\DanielOlsen\repos\bes\REISE.jl` exists and looks like the correct package. Using existing path.
  Resolving package versions...
Updating `C:\Users\DanielOlsen\repos\bes\Gurobitest\Project.toml`
  [b654829c] + REISE v0.1.0 `C:\Users\DanielOlsen\repos\bes\REISE.jl`
Updating `C:\Users\DanielOlsen\repos\bes\Gurobitest\Manifest.toml`
  [56f22d72] + Artifacts v1.3.0
  [6e4b80f9] + BenchmarkTools v0.5.0
  [a74b3585] + Blosc v0.7.0
  [0b7ba130] + Blosc_jll v1.14.3+1
  [e1450e63] + BufferedStreams v1.0.0
  [6e34b625] + Bzip2_jll v1.0.6+5
  [336ed68f] + CSV v0.8.3
  [49dc2e85] + Calculus v0.5.1
  [324d7699] + CategoricalArrays v0.8.3
  [d360d2e6] + ChainRulesCore v0.9.27
  [523fee87] + CodecBzip2 v0.7.2
  [944b1d66] + CodecZlib v0.7.0
  [bbf7d656] + CommonSubexpressions v0.3.0
  [34da2185] + Compat v3.25.0
  [e66e0078] + CompilerSupportLibraries_jll v0.3.4+0
  [8f4d0f93] + Conda v1.5.0
  [9a962f9c] + DataAPI v1.5.0
  [a93c6f00] + DataFrames v0.21.8
  [864edb3b] + DataStructures v0.18.9
  [e2d170a0] + DataValueInterfaces v1.0.0
  [163ba53b] + DiffResults v1.0.3
  [b552c78f] + DiffRules v1.0.2
  [f6369f11] + ForwardDiff v0.10.15
  [f67ccb44] + HDF5 v0.15.2
  [0234f1f7] + HDF5_jll v1.12.0+1
  [cd3eb016] + HTTP v0.9.2
  [83e8ac13] + IniFile v0.5.0
  [41ab1584] + InvertedIndices v1.0.0
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [692b3bcd] + JLLWrappers v1.2.0
  [682c06a0] + JSON v0.21.1
  [7d188eb4] + JSONSchema v0.3.3
  [4076af6c] + JuMP v0.21.6
  [deac9b47] + LibCURL_jll v7.70.0+2
  [29816b5a] + LibSSH2_jll v1.9.0+3
  [5ced341a] + Lz4_jll v1.9.2+2
  [23992714] + MAT v0.10.1
  [1914dd2f] + MacroTools v0.5.6
  [b8f27783] + MathOptInterface v0.9.19
  [739be429] + MbedTLS v1.0.3
  [c8ffd9c3] + MbedTLS_jll v2.16.8+1
  [e1d29d7a] + Missings v0.4.5
  [d8a4904e] + MutableArithmetics v0.2.14
  [77ba4419] + NaNMath v0.3.5
  [458c3c95] + OpenSSL_jll v1.1.1+6
  [efe28fd5] + OpenSpecFun_jll v0.5.3+4
  [bac558e1] + OrderedCollections v1.3.3
  [69de0a69] + Parsers v1.0.15
  [2dfb63ee] + PooledArrays v0.5.3
  [438e738f] + PyCall v1.92.2
  [b654829c] + REISE v0.1.0 `C:\Users\DanielOlsen\repos\bes\REISE.jl`
  [189a3867] + Reexport v0.2.0
  [ae029012] + Requires v1.1.2
  [91c51154] + SentinelArrays v1.2.16
  [a2af1166] + SortingAlgorithms v0.3.1
  [276daf66] + SpecialFunctions v1.2.1
  [90137ffa] + StaticArrays v1.0.1
  [856f2bd8] + StructTypes v1.2.3
  [3783bdb8] + TableTraits v1.0.0
  [bd369af6] + Tables v1.3.1
  [3bb67fe8] + TranscodingStreams v0.9.5
  [5c2747f8] + URIs v1.2.0
  [81def892] + VersionParsing v1.2.0
  [a5390f91] + ZipFile v0.9.3
  [83775a58] + Zlib_jll v1.2.11+18
  [3161d3a3] + Zstd_jll v1.4.8+0
  [8e850ede] + nghttp2_jll v1.40.0+2
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8bb1440f] + DelimitedFiles
  [8ba89e20] + Distributed
  [9fa8497b] + Future
  [b77e0a4c] + InteractiveUtils
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [1a1011a3] + SharedArrays
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode

julia> Pkg.add("Gurobi")
   Updating registry at `C:\Users\DanielOlsen\.julia\registries\General`
   Updating git-repo `https://github.com/JuliaRegistries/General.git`
  Resolving package versions...
Updating `C:\Users\DanielOlsen\repos\bes\Gurobitest\Project.toml`
  [2e9cd046] + Gurobi v0.9.7
Updating `C:\Users\DanielOlsen\repos\bes\Gurobitest\Manifest.toml`
  [fa961155] + CEnum v0.4.1
  [2e9cd046] + Gurobi v0.9.7

julia> import Gurobi

julia> import REISE

julia> REISE.run_scenario_gurobi(; interval=24, n_interval=1, start_index=1, inputfolder=raw"C:\Users\DanielOlsen\repos\bes\scenario_2066", outputfolder=raw"C:\Users\DanielOlsen\repos\bes\scenario_2066\output4")
Compute Server job ID: 2c0b1485-0665-427d-9028-c4958d18392b
Capacity available on 'https://ip-10-0-67-145:61000' - connecting...
Established HTTPS encrypted connection
Reading from folder: C:\Users\DanielOlsen\repos\bes\scenario_2066
...loading case.mat
...loading demand.csv
...loading hydro.csv
...loading wind.csv
...loading solar.csv
File case_storage.mat not found in C:\Users\DanielOlsen\repos\bes\scenario_2066
All scenario files loaded!
linearizing
All preparation complete!
Redirecting outputs, see stdout.log & stderr.err in outputfolder

Compute Server communication statistics:
  Sent: 24.958 MBytes in 32 msgs and 11.01s (2.27 MB/s)
  Received: 7.274 MBytes in 54 msgs and 10.95s (0.66 MB/s)

Connection closed successfully!

julia>
With GLPK
julia> import Pkg

julia> Pkg.activate("GLPKtest")
 Activating new environment at `C:\Users\DanielOlsen\repos\bes\GLPKtest\Project.toml`

julia> Pkg.develop(path=raw"C:\Users\DanielOlsen\repos\bes\REISE.jl")
Path `C:\Users\DanielOlsen\repos\bes\REISE.jl` exists and looks like the correct package. Using existing path.
  Resolving package versions...
Updating `C:\Users\DanielOlsen\repos\bes\GLPKtest\Project.toml`
  [b654829c] + REISE v0.1.0 `C:\Users\DanielOlsen\repos\bes\REISE.jl`
Updating `C:\Users\DanielOlsen\repos\bes\GLPKtest\Manifest.toml`
  [56f22d72] + Artifacts v1.3.0
  [6e4b80f9] + BenchmarkTools v0.5.0
  [a74b3585] + Blosc v0.7.0
  [0b7ba130] + Blosc_jll v1.14.3+1
  [e1450e63] + BufferedStreams v1.0.0
  [6e34b625] + Bzip2_jll v1.0.6+5
  [336ed68f] + CSV v0.8.3
  [49dc2e85] + Calculus v0.5.1
  [324d7699] + CategoricalArrays v0.8.3
  [d360d2e6] + ChainRulesCore v0.9.27
  [523fee87] + CodecBzip2 v0.7.2
  [944b1d66] + CodecZlib v0.7.0
  [bbf7d656] + CommonSubexpressions v0.3.0
  [34da2185] + Compat v3.25.0
  [e66e0078] + CompilerSupportLibraries_jll v0.3.4+0
  [8f4d0f93] + Conda v1.5.0
  [9a962f9c] + DataAPI v1.5.0
  [a93c6f00] + DataFrames v0.21.8
  [864edb3b] + DataStructures v0.18.9
  [e2d170a0] + DataValueInterfaces v1.0.0
  [163ba53b] + DiffResults v1.0.3
  [b552c78f] + DiffRules v1.0.2
  [f6369f11] + ForwardDiff v0.10.15
  [f67ccb44] + HDF5 v0.15.2
  [0234f1f7] + HDF5_jll v1.12.0+1
  [cd3eb016] + HTTP v0.9.2
  [83e8ac13] + IniFile v0.5.0
  [41ab1584] + InvertedIndices v1.0.0
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [692b3bcd] + JLLWrappers v1.2.0
  [682c06a0] + JSON v0.21.1
  [7d188eb4] + JSONSchema v0.3.3
  [4076af6c] + JuMP v0.21.6
  [deac9b47] + LibCURL_jll v7.70.0+2
  [29816b5a] + LibSSH2_jll v1.9.0+3
  [5ced341a] + Lz4_jll v1.9.2+2
  [23992714] + MAT v0.10.1
  [1914dd2f] + MacroTools v0.5.6
  [b8f27783] + MathOptInterface v0.9.19
  [739be429] + MbedTLS v1.0.3
  [c8ffd9c3] + MbedTLS_jll v2.16.8+1
  [e1d29d7a] + Missings v0.4.5
  [d8a4904e] + MutableArithmetics v0.2.14
  [77ba4419] + NaNMath v0.3.5
  [458c3c95] + OpenSSL_jll v1.1.1+6
  [efe28fd5] + OpenSpecFun_jll v0.5.3+4
  [bac558e1] + OrderedCollections v1.3.3
  [69de0a69] + Parsers v1.0.15
  [2dfb63ee] + PooledArrays v0.5.3
  [438e738f] + PyCall v1.92.2
  [b654829c] + REISE v0.1.0 `C:\Users\DanielOlsen\repos\bes\REISE.jl`
  [189a3867] + Reexport v0.2.0
  [ae029012] + Requires v1.1.2
  [91c51154] + SentinelArrays v1.2.16
  [a2af1166] + SortingAlgorithms v0.3.1
  [276daf66] + SpecialFunctions v1.2.1
  [90137ffa] + StaticArrays v1.0.1
  [856f2bd8] + StructTypes v1.2.3
  [3783bdb8] + TableTraits v1.0.0
  [bd369af6] + Tables v1.3.1
  [3bb67fe8] + TranscodingStreams v0.9.5
  [5c2747f8] + URIs v1.2.0
  [81def892] + VersionParsing v1.2.0
  [a5390f91] + ZipFile v0.9.3
  [83775a58] + Zlib_jll v1.2.11+18
  [3161d3a3] + Zstd_jll v1.4.8+0
  [8e850ede] + nghttp2_jll v1.40.0+2
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8bb1440f] + DelimitedFiles
  [8ba89e20] + Distributed
  [9fa8497b] + Future
  [b77e0a4c] + InteractiveUtils
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [1a1011a3] + SharedArrays
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode

julia> Pkg.add("GLPK")
   Updating registry at `C:\Users\DanielOlsen\.julia\registries\General`
   Updating git-repo `https://github.com/JuliaRegistries/General.git`
  Resolving package versions...
Updating `C:\Users\DanielOlsen\repos\bes\GLPKtest\Project.toml`
  [60bf3e95] + GLPK v0.14.6
Updating `C:\Users\DanielOlsen\repos\bes\GLPKtest\Manifest.toml`
  [b99e7846] + BinaryProvider v0.5.10
  [fa961155] + CEnum v0.4.1
  [60bf3e95] + GLPK v0.14.6
  [e8aa6df9] + GLPK_jll v4.64.0+0
  [781609d7] + GMP_jll v6.1.2+6

julia> import REISE

julia> import GLPK

julia> REISE.run_scenario(; interval=1, n_interval=3, start_index=1, inputfolder=raw"C:\Users\DanielOlsen\repos\bes\scenario_2066", outputfolder=raw"C:\Users\DanielOlsen\repos\bes\scenario_2066\output2", optimizer_factory=GLPK.Optimizer)
Reading from folder: C:\Users\DanielOlsen\repos\bes\scenario_2066
...loading case.mat
...loading demand.csv
...loading hydro.csv
...loading wind.csv
...loading solar.csv
File case_storage.mat not found in C:\Users\DanielOlsen\repos\bes\scenario_2066
All scenario files loaded!
linearizing
All preparation complete!
Redirecting outputs, see stdout.log & stderr.err in outputfolder

julia>

Python

> python call.py -s 2016-01-01 -e 2016-01-03 -int 24 -i C:\Users\DanielOlsen\repos\bes\scenario_2066 -o C:\Users\DanielOlsen\repos\bes\scenario_2066\output3
Validation complete!
Launching scenario with parameters:
{'interval': 24, 'n_interval': 3, 'start_index': 1, 'input_dir': 'C:\\Users\\DanielOlsen\\repos\\bes\\scenario_2066', 'execute_dir': None, 'threads': None}
Waiting for cloud server to start (pool default)...
Starting...
Starting...
Starting...
Starting...
Compute Server job ID: e745ace6-8416-474e-ad79-94da0cc1488b
Capacity available on 'https://ip-10-0-68-159:61000' - connecting...
Established HTTPS encrypted connection
Reading from folder: C:\Users\DanielOlsen\repos\bes\scenario_2066
...loading case.mat
...loading demand.csv
...loading hydro.csv
...loading wind.csv
...loading solar.csv
File case_storage.mat not found in C:\Users\DanielOlsen\repos\bes\scenario_2066
All scenario files loaded!
linearizing
All preparation complete!
Redirecting outputs, see stdout.log & stderr.err in outputfolder
Connection closed successfully!
Run time: 0:02:57

Compute Server communication statistics:
  Sent: 50.894 MBytes in 65 msgs and 17.45s (2.92 MB/s)
  Received: 21.755 MBytes in 150 msgs and 28.71s (0.76 MB/s)

Time to review

30-60 minutes.

@danielolsen danielolsen added the new feature Feature that is currently in progress. label Feb 2, 2021
@danielolsen danielolsen self-assigned this Feb 2, 2021
:param None/int threads: number of threads to use.
:param None/dict solver_kwargs: keyword arguments to pass to solver (if any).
:return: (*int*) runtime of scenario in seconds
"""
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I try to understand how you can avoid to instantiate the parent class using super(). The parameters used in REISE.run_scenario_gurobi) are all attributes of the Launcher class (to the exception of execute_dir and threads) are set in the constructor of Launcher. I guess I am missing something.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GurobiLauncher inherits the __init__ from Launcher since we don't redefine our own.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. I guess I have never been in a situation where the child class does not have a constructor

@jenhagg
Copy link
Collaborator

jenhagg commented Feb 2, 2021

Think we'll need to add a Pkg.add("Gurobi") or equivalent to the Dockerfile. This is nice though, maybe we could install both gurobi and glpk in docker and use the latter for CI tests.

@danielolsen
Copy link
Contributor Author

Think we'll need to add a Pkg.add("Gurobi") or equivalent to the Dockerfile. This is nice though, maybe we could install both gurobi and glpk in docker and use the latter for CI tests.

Good catch, if Gurobi is not a requirement of REISE.jl then yes we need to install it manually. It looks like we would modify l.24 of the current Dockerfile, but I don't entirely understand what's going on there currently:

  • Why is there a comma between Pkg.instantiate() and using REISE when all of the other statements are separated by semi-colons?
  • Why do we call using REISE at all?

@jenhagg
Copy link
Collaborator

jenhagg commented Feb 2, 2021

Think we'll need to add a Pkg.add("Gurobi") or equivalent to the Dockerfile. This is nice though, maybe we could install both gurobi and glpk in docker and use the latter for CI tests.

Good catch, if Gurobi is not a requirement of REISE.jl then yes we need to install it manually. It looks like we would modify l.24 of the current Dockerfile, but I don't entirely understand what's going on there currently:

* Why is there a comma between `Pkg.instantiate()` and `using REISE` when all of the other statements are separated by semi-colons?

* Why do we call `using REISE` at all?

Good question, I think @ahurli would know. I'm also doing a quick test to check that it works. edit: works as expected

@danielolsen
Copy link
Contributor Author

What's the final word on dockerfile changes? Are they needed, or can we avoid the add and let users of the dockerfile add whichever solver they want?

@jenhagg
Copy link
Collaborator

jenhagg commented Feb 5, 2021

I'd say we should add it by default, since there isn't currently a great mechanism to have users specify that. They would have to attach a shell to the container and run the command in a julia prompt anytime the image is updated, which could be as often as code is merged.

@jenhagg
Copy link
Collaborator

jenhagg commented Feb 5, 2021

In any case, I think the flexibility will come from something like a parameter to call.py where we choose the Launcher subclass at runtime. So we can add multiple solver packages in the image, but it won't restrict users.

@danielolsen
Copy link
Contributor Author

I guess I'm still a bit fuzzy on how this dockerfile is intended to be used. Is it supposed to be the minimum requirements to get this package up and running, or something that will cover all use cases for all users? Does it make sense to have multiple images to cover multiple use cases?

@jenhagg
Copy link
Collaborator

jenhagg commented Feb 5, 2021

That's a good question.. usually the answer depends on how much variation there is between use cases. Like in Todd's case for CERF, there is a large base image with several different images built on top of that for different cases which are totally different (this is my interpretation). Here, I think the trade off of having potentially several extra packages is worth it since the other option adds complexity of setting up multiple builds and having users make a choice. Also, it feels like still one use case here - the different solvers don't really change the "mode of operation". If anything, the rest api and command line use cases are more different.

To answer the question - right now the goal is more to cover all use cases, but in the future it would be reasonable to consider splitting into multiple images depending on the situation.

@danielolsen
Copy link
Contributor Author

Alright, I added what is needed (I think) to be able to launch Gurobi scenarios from within this Dockerfile without any extra steps. We add Gurobi to the environment, and then import it to enable the run_scenario_gurobi function.

@jenhagg
Copy link
Collaborator

jenhagg commented Feb 5, 2021

Huh, I thought I had it work without the import statement, but gonna trust you on that one.

@danielolsen
Copy link
Contributor Author

Huh, I thought I had it work without the import statement, but gonna trust you on that one.

Working as in compiling without an error, or working as in able to launch a scenario successfully?

@jenhagg
Copy link
Collaborator

jenhagg commented Feb 5, 2021

Huh, I thought I had it work without the import statement, but gonna trust you on that one.

Working as in compiling without an error, or working as in able to launch a scenario successfully?

I thought I had launched a scenario, but can't remember since I was switching branches a lot. Let's go ahead with this since I trust you on getting the julia environment right.

@danielolsen danielolsen merged commit 61eae94 into develop Feb 5, 2021
@danielolsen danielolsen deleted the daniel/no_gurobi_req branch February 5, 2021 22:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new feature Feature that is currently in progress.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Allow solver settings to be passed via pyreisejl Refactor the Gurobi.jl requirement
3 participants