feat: Add Dockerfile #85

ahurli · 2020-11-24T20:14:56Z

Purpose

Create a Dockerfile with documentation to run the REISE.jl

What is the code doing

There is now a Dockerfile that can be used to build an image and then run a container with a bash entrypoint to run a simulation within the container.

Where to look

The Dockerfile contains the information needed to run the image, and commands to build and run the container with the necessary mounts are documented in the README.

Time estimate

10min

README.md

Dockerfile

rouille · 2020-11-25T00:42:09Z

README.md

+Once the container is running, you can run a simulation using the `python`
+commands described above. For example:
+
+```


```bash will enable synthax highlighting

rouille · 2020-11-25T00:42:49Z

README.md

+`Gurobi` license file and another containing the necessary input files for the
+engine. 
+
+```


```bash will enable synthax highlighting

rouille · 2020-11-25T00:43:10Z

README.md

+Docker daemon installed and running, navigate to the `REISE.jl` folder
+containing the `Dockerfile` and build the image:
+
+```


```bash will enable synthax highlighting

Dockerfile

README.md

danielolsen · 2020-11-25T00:45:24Z

Dockerfile

+WORKDIR /build/gurobi_installer
+
+RUN wget -q https://packages.gurobi.com/9.0/gurobi9.0.2_linux64.tar.gz
+RUN mkdir /usr/share/gurobi902


Similarly, updating this to /usr/share/gurobi910

jenhagg · 2020-11-25T03:25:44Z

Dockerfile

-ENV GUROBI_HOME='/usr/share/gurobi902/linux64'
-ENV GRB_LICENSE_FILE='/usr/share/gurobi_license/gurobi.lic'
-ENV JULIA_PROJECT='/app'
+ENV PATH="$PATH:/usr/share/julia-1.4.2/bin" \


Should there be a semicolon delimiter?

I can't necessarily find any Docker documentation to validate this, but I can validate that these variables have been correctly assigned within the container:

# echo $JULIA_PROJECT /app # echo $PYTHONPATH /app/pyreisejl: # echo $GUROBI_HOME /usr/share/gurobi902/linux64

GUROBI_HOME should be /usr/share/gurobi910/linux64?

It's consistent with the version that this Dockerfile is using, but yes, I updated this as well when testing the new Gurobi and Julia versions!

danielolsen · 2020-11-25T19:53:44Z

Dockerfile

+RUN wget -q https://julialang-s3.julialang.org/bin/linux/x64/1.4/julia-1.4.2-linux-x86_64.tar.gz &&\ 
+    tar -xf julia-1.4.2-linux-x86_64.tar.gz -C /usr/share


We should also update to the latest Julia version (1.5.3).

ahurli · 2020-12-02T02:19:02Z

Everything builds correctly now using the new REISE.jl in #86, but I am unable to test anything out because I'm having an issue with Gurobi.

Both locally and in the containers, Gurobi (specifically gurobi_cl) errors out with the following error:

Using license file /usr/share/gurobi_license/gurobi.lic
Set parameter CloudAccessID
Set parameter CloudSecretKey
Set parameter LogFile to value gurobi.log
Failed to set up a license
Error 10032: Access to this operation is not allowed

I was able to successfully run simulations in the Nov 24th Docker version (with Gurobi 9.0.2), but even that image results in the same error now.

ahurli · 2020-12-02T21:22:27Z

After pulling a new cloud license and using that, I am now able to run simulations again.

Unfortunately, I am now seeing this error when trying to run Julia:

All scenario files loaded!
linearizing
Traceback (most recent call last):
  File "pyreisejl/utility/call.py", line 216, in <module>
    runtime = launch_scenario(
  File "pyreisejl/utility/call.py", line 94, in launch_scenario
    REISE.run_scenario(
RuntimeError: Julia exception: MethodError: no method matching d_create(::HDF5.HDF5Group, ::String, ::Array{Int32,1}, ::String, ::Array{Int64,1}, ::String, ::Int64)
Closest candidates are:
  d_create(::Union{HDF5.HDF5File, HDF5.HDF5Group}, ::String, !Matched::Type, ::Any, !Matched::String, ::Any, ::Any...) at /root/.julia/packages/HDF5/YX0jU/src/HDF5.jl:954
  d_create(::Union{HDF5.HDF5File, HDF5.HDF5Group}, ::String, !Matched::HDF5.HDF5Datatype, !Matched::HDF5.HDF5Dataspace, !Matched::String, ::Any, ::Any...) at /root/.julia/packages/HDF5/YX0jU/src/HDF5.jl:944
  d_create(::Union{HDF5.HDF5File, HDF5.HDF5Group}, ::String, !Matched::HDF5.HDF5Datatype, !Matched::Tuple{Vararg{Int64,N}} where N, !Matched::String, ::Any, ::Any...) at /root/.julia/packages/HDF5/YX0jU/src/HDF5.jl:952
  ...
Stacktrace:
 [1] m_writearray(::HDF5.HDF5Group, ::String, ::Array{Int32,1}, ::Bool) at /root/.julia/packages/MAT/2LFMT/src/MAT_HDF5.jl:331
 [2] m_write(::MAT.MAT_HDF5.MatlabHDF5File, ::HDF5.HDF5Group, ::String, ::Int32) at /root/.julia/packages/MAT/2LFMT/src/MAT_HDF5.jl:377
 [3] m_write(::MAT.MAT_HDF5.MatlabHDF5File, ::HDF5.HDF5Group, ::String, ::Array{Any,2}) at /root/.julia/packages/MAT/2LFMT/src/MAT_HDF5.jl:478
 [4] m_write(::MAT.MAT_HDF5.MatlabHDF5File, ::HDF5.HDF5Group, ::String, ::Array{String,1}, ::Array{Any,1}) at /root/.julia/packages/MAT/2LFMT/src/MAT_HDF5.jl:517
 [5] m_write(::MAT.MAT_HDF5.MatlabHDF5File, ::HDF5.HDF5Group, ::String, ::Dict{String,Any}) at /root/.julia/packages/MAT/2LFMT/src/MAT_HDF5.jl:523
 [6] m_write(::MAT.MAT_HDF5.MatlabHDF5File, ::HDF5.HDF5File, ::String, ::Array{String,1}, ::Array{Dict{String,Any},1}) at /root/.julia/packages/MAT/2LFMT/src/MAT_HDF5.jl:517
 [7] m_write(::MAT.MAT_HDF5.MatlabHDF5File, ::HDF5.HDF5File, ::String, ::Dict{String,Dict{String,Any}}) at /root/.julia/packages/MAT/2LFMT/src/MAT_HDF5.jl:523
 [8] write(::MAT.MAT_HDF5.MatlabHDF5File, ::String, ::Dict{String,Dict{String,Any}}) at /root/.julia/packages/MAT/2LFMT/src/MAT_HDF5.jl:545
 [9] matwrite(::String, ::Dict{String,Dict{String,Dict{String,Any}}}; compress::Bool) at /root/.julia/packages/MAT/2LFMT/src/MAT.jl:157
 [10] save_input_mat(::REISE.Case, ::REISE.Storage, ::String, ::String) at /app/src/save.jl:61
 [11] run_scenario(; num_segments::Int64, interval::Int64, n_interval::Int64, start_index::Int64, inputfolder::String, outputfolder::Nothing, threads::Nothing) at /app/src/REISE.jl:53
 [12] (::PyCall.var"#f_kw_closure#59"{typeof(REISE.run_scenario),Tuple{},Array{Tuple{Symbol,Any},1}})() at /root/.julia/packages/PyCall/zqDXB/src/callback.jl:36
 [13] _pyjlwrap_call(::Function, ::Ptr{PyCall.PyObject_struct}, ::Ptr{PyCall.PyObject_struct}) at /root/.julia/packages/PyCall/zqDXB/src/callback.jl:37
 [14] pyjlwrap_call(::Ptr{PyCall.PyObject_struct}, ::Ptr{PyCall.PyObject_struct}, ::Ptr{PyCall.PyObject_struct}) at /root/.julia/packages/PyCall/zqDXB/src/callback.jl:49

I've tried with both Julia 1.5.3 (the latest version released) and Julia 1.5.1 (matching the current installation on the compute server) and gotten the same error each time. I've even downloaded the files for a recent scenario (scenario 1682, currently running on the compute server) to see if the files I were using were too old, but still ran into the same issue. I am able to create the input.mat file for scenario_1682, but it is only 4.0K when the version saved on the compute server is 568M.

danielolsen · 2020-12-02T21:26:04Z

Which version of MAT.jl is being installed?

danielolsen · 2020-12-02T21:38:47Z

It looks like MAT.jl recently upgraded from v0.8.1 to v0.9, and as part of this there is a change in the version of HDF5.jl required (from {0.11, 0.12, 0.13} to 0.14), which changes the function signature of d_create. If you add MAT = "0.8" to the end of Project.toml, I believe this will solve the problem for now.

ahurli · 2020-12-03T17:30:15Z

The version of MAT was already 0.8.0, and it turns out this issue is solved in 0.8.1. I updated the Manifest.toml file to reflect this new version, after updating MAT only a minor version to 0.8.1.

As a side note, I did update all the packages in Julia while trying to figure out how to get this error to go away, and on initial testing it looks like everything works out, but I imagine we'll want to do more thorough testing.

danielolsen · 2020-12-03T17:57:10Z

Good catch, I think I see the relevant change in the MAT.jl v0.8.0 -> v0.8.1 diff: JuliaIO/MAT.jl@v0.8.0...v0.8.1#diff-c5711165d185d17fd8e62efe7908aed494576d39bb7f01dae9ed2ff28a30d4beL331. As a side note, the version of MAT.jl that's currently installed and running successfully on the server is v0.9.2 (since I'm deving the package, rather than instantiateing), so at lease one minor version of v0.9 seems to actually work just fine.

If you can run and extract a scenario successfully, I think that should be enough to say that this dockerfile and dependency specification is ready to ship.

danielolsen · 2020-12-03T22:15:10Z

Please give this PR a more descriptive name and squash down some of these smaller commits, and confirm that this works to run and extract a scenario, then I will approve.

refactor: combine Docker commands

ahurli requested review from ToddG, dmuldrew, rouille and jenhagg November 24, 2020 20:14

jenhagg reviewed Nov 25, 2020

View reviewed changes

README.md Outdated Show resolved Hide resolved

jenhagg reviewed Nov 25, 2020

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

jenhagg reviewed Nov 25, 2020

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

rouille reviewed Nov 25, 2020

View reviewed changes

danielolsen reviewed Nov 25, 2020

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

rouille reviewed Nov 25, 2020

View reviewed changes

README.md Show resolved Hide resolved

danielolsen reviewed Nov 25, 2020

View reviewed changes

jenhagg reviewed Nov 25, 2020

View reviewed changes

danielolsen mentioned this pull request Nov 25, 2020

Update Gurobi.jl to v0.9.3 #86

Merged

danielolsen reviewed Nov 25, 2020

View reviewed changes

ahurli force-pushed the anna/docker branch from 68b83c3 to 1d7b54e Compare December 3, 2020 17:25

ahurli changed the title ~~Anna/docker~~ Feat: Add Dockerfile Dec 4, 2020

ahurli added 4 commits December 4, 2020 12:40

feat: add dockerfile

a5785a0

docs: add Docker instructions to README

5569d7e

refactor: combine Docker commands

chore: update Julia and Gurobi versions

e831cdc

chore: update MAT.jl to 0.8.1

eead924

ahurli force-pushed the anna/docker branch from 1d7b54e to eead924 Compare December 4, 2020 20:44

jenhagg approved these changes Dec 7, 2020

View reviewed changes

ahurli merged commit 5628f5f into develop Dec 7, 2020

danielolsen added a commit that referenced this pull request Dec 7, 2020

Merge pull request #85 from Breakthrough-Energy/anna/docker

868dd5c

danielolsen mentioned this pull request Feb 12, 2021

chore: merge v0.2.0 into master #111

Merged

danielolsen changed the title ~~Feat: Add Dockerfile~~ feat: Add Dockerfile Feb 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add Dockerfile #85

feat: Add Dockerfile #85

ahurli commented Nov 24, 2020

rouille Nov 25, 2020 •

edited

Loading

rouille Nov 25, 2020

rouille Nov 25, 2020

danielolsen Nov 25, 2020

jenhagg Nov 25, 2020

ahurli Nov 30, 2020

BainanXia Nov 30, 2020

ahurli Dec 2, 2020

danielolsen Nov 25, 2020

ahurli Dec 2, 2020

ahurli commented Dec 2, 2020

ahurli commented Dec 2, 2020

danielolsen commented Dec 2, 2020

danielolsen commented Dec 2, 2020

ahurli commented Dec 3, 2020

danielolsen commented Dec 3, 2020

danielolsen commented Dec 3, 2020

		RUN wget -q https://julialang-s3.julialang.org/bin/linux/x64/1.4/julia-1.4.2-linux-x86_64.tar.gz &&\
		tar -xf julia-1.4.2-linux-x86_64.tar.gz -C /usr/share

feat: Add Dockerfile #85

feat: Add Dockerfile #85

Conversation

ahurli commented Nov 24, 2020

Purpose

What is the code doing

Where to look

Time estimate

rouille Nov 25, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahurli commented Dec 2, 2020

ahurli commented Dec 2, 2020

danielolsen commented Dec 2, 2020

danielolsen commented Dec 2, 2020

ahurli commented Dec 3, 2020

danielolsen commented Dec 3, 2020

danielolsen commented Dec 3, 2020

rouille Nov 25, 2020 •

edited

Loading