Challenges with DAPHNE container images from DockerHub

[pdamme]

We have made quite some progress with the DAPHNE containers recently to make DAPHNE run on various platforms (thanks to @corepointer). I know that the support is not yet meant to be 100% user-friendly, and that there are certain difficulties with different flavors of DAPHNE (e.g., with/without CUDA etc.).

I have tried using the DAPHNE container images available on DockerHub to obtain an environment for building and running DAPHNE. I followed the documentation in doc/GettingStarted.md and containers/Readme.md as far as possible. Unfortunately, the container images on DockerHub are currently not usable out of the box. We should fix this, since these container images can simplify the life of users and developers significantly (and we are currently getting several new users/developers, TU Berlin students).

tl;dr Required action points

• Rebuild both container images and push the updates to DockerHub:
  • daphne-dev: needs at least pandas and Eigen3.
  • daphne-dev-interactive: needs at least git, pandas, and Eigen3.
• Make the scripts for starting the containers work out of the box (for a typical case of working of the main branch).
• Fix the problems with the user identity inside the container.
• Update the documentation accordingly (which is a separate topic).

Here are a few more details:

Problem 1: containers/run-docker-example.sh (the script for running the containers) doesn't work out of the box. I know it's meant to be customized by users, but this is too difficult for a new user/developer. We should provide a working variant for the common case.
Details:

• I pulled the images by

  docker pull daphne-dev
  docker pull daphne-dev-interactive
  

• Then I tried running daphne-dev by ./containers/run-docker-example.sh, but that yielded the following error:

  docker: invalid reference format.
  See 'docker run --help'.
  

  As a quick fix, I removed the "$DEVICE_FLAGS" in the docker run command.
• After that, running ./containers/run-docker-example.sh succeeded, but the command immediately finished. As a user, I would have expected to be in the container. I fixed this by adding -it to the docker run command (but that may not be desired in all cases).
• After that, by running ./containers/run-docker-example.sh I ended up in the container. However, the output

  Use this as an example to start DAPHNE docker containers. Copy and customize for the various flavors.
  groups: cannot find name for group ID 1000
  I have no name!@60b4d4a9fe94:/home/<...>/prototype$
  

  suggests that something went wrong with the user/group. In any case, it looks suspicious. Nevertheless, I could execute commands inside the container.

Problem 2: The container image daphne-dev doesn't work out of the box; at least we need to rebuild it with Eigen3 and pandas.
Details:

• I tried building DAPHNE by ./build.sh --no-deps --installPrefix /usr/local as specified in the docs. That failed due to missing Eigen3:

  CMake Error at CMakeLists.txt:132 (find_package):
   By not providing "FindEigen3.cmake" in CMAKE_MODULE_PATH this project has
   asked CMake to find a package configuration file provided by "Eigen3", but
   CMake did not find one.
  
    Could not find a package configuration file provided by "Eigen3" with any
    of the following names:
  
      Eigen3Config.cmake
      eigen3-config.cmake
  
    Add the installation prefix of "Eigen3" to CMAKE_PREFIX_PATH or set
    "Eigen3_DIR" to a directory containing one of the above files.  If "Eigen3"
    provides a separate development package or SDK, be sure it has been
    installed.
  

  I assume that is because Eigen has recently been added as a dependency, but the container image may not have been updated?
• To have a fresh start, I executed ./build.sh --cleanAll.
• Then I tried building everything from source by ./build.sh, which finished successfully (well, actually it failed initially since the URL we downloaded Arrow 11.0 from has become invalid, but I have already fixed this in cb3071c)
• Executing some DAPHNE script worked: bin/daphne scripts/examples/hello-world.daph.
• Running the tests by ./test.sh failed, because pandas is missing in the container (probably we need a rebuild/repush of the container image for that as well).
• I tried installing pandas in the container, but that failed because the default container user doesn't have the rights and sudo is missing. But okay, for such purposes we have the daphne-dev-interactive container...

Problem 3: The container image daphne-dev-interactive doesn't work out of the box; at least we need to rebuild it with Eigen3, pandas, and git.
Details:

• I tried the same steps as above in the other container image.
• A first attempt of ./build.sh --no-deps --installPrefix /usr/local worked, but I assume this was only because the build artifacts were still around.
• Then I tried ./build.sh, which failed with ./build.sh: line 820: git: command not found. Same for ./test.sh.
• However, running some DaphneDSL scripts worked: bin/daphne scripts/examples/hello-world.daph.
• ./build.sh --cleanAll started doing something, then it also failed due to missing git.
• After that, ./build.sh --no-deps --installPrefix /usr/local failed due to missing Eigen3.
• I also tried installing git by apt install git, but the user doesn't have the permissions. I tried sudo apt install git (this container has sudo), but it failed with sudo: you do not exist in the passwd database, probably due to problems with the user identity.

I have also tried building the containers locally, but also found it not to work out of the box...

[pdamme]

@corepointer: Could you update the container images on DockerHub as follows, please?

• add Eigen3 and pandas (both containers, I guess it just needs a rebuild of the containers)
• add git (for daphne-dev-interactive, I guess it requires a little update to a Dockerfile as well)

I hope that doesn't cause any inconvenience. After that, I could also try to fix the remaining issues, but it's surely a good idea to coordinate on this, because the containers are used in multiple places, and you are the expert there :) .

[corepointer]

Thx for testing and reporting @pdamme! Just a moment too early - I've not gotten around to pushing the latest developments in containers yet. Otherwise many of the found issues would have not happened. Git, pandas, Eigen3 will all be there (the latter two are only in the github-actions image atm), as will be the documentation to use the images.

[pdamme]

Thanks for the quick response! Sounds very good, I'm looking forward to these updates and will try it again once they are available. Do you think you could make it within the next few days? (Just so that our students can get started more easily.)

[corepointer]

Yes, it'll be ready soon© I'll notify you.

[pdamme]

Thanks a lot! In the meantime, instructions for a temporary solution with DAPHNE in docker containers can be found in a branch: https://github.com/daphne-eu/daphne/tree/containers-tmp/containers-tmp . This is a reduced variant of the scripts in your PR #534. So take your time to finish your work on the containers :) .