Usage of environment-modules in containers

[xfong]

Version of Singularity:

What version of Singularity are you using? 3.7.4 and 3.8.*

Container is built with the 90-environment.sh to set up the environment-modules environment ( ". /etc/profile.d/modules.sh") and additional environment set up using "module load".

Expected behavior

All environment variables in the modules (e.g., PATH, CPATH, INCLUDE_PATH, LD_LIBRARY_PATH, PYTHONPATH, LOADEDMODULES, etc.) should be correctly defined and imported.

Actual behavior

The correct behavior is observed in 3.7.4 but is broken after 3.8.0.
PYTHONPATH is not correctly updated. Running "module list" in the updated environment shows an error that it is not able to locate certain variables in environment-modules. The container is the same for running in both versions of singularity.

Steps to reproduce this behavior

How can others reproduce this issue/problem?

• Build a container based on Ubuntu with environment-modules and has a module that tries to set up PYTHONPATH
• For the build, include lines in the %environment section to load the module upon entering the container (e.g., . /etc/profile.d/modules.sh && module load testModule)
• Launch an interactive shell in the container and run ". /etc/profile.d/modules.sh && module list"

What OS/distro are you running

Ubuntu 18.04

How did you install Singularity

Installed from source. The host machine is running Ubuntu 20.04

[DrDaveD]

Please provide a full container definition file to reproduce the problem. For instance I don't know where the "module" command is coming from.

[xfong]

The container definition file is "sample_test.txt". The "test" file that is called in the %files section is "test.txt".
The expected behavior after the container is created is that "singularity shell -e" into the container should result in the PYTHONPATH environment variable being set by the %environment section in the definition file.

sample_test.txt
test.txt

[DrDaveD]

Intermediate status update: @cclerget told me by slack that he found a workaround, by putting export PYTHONPATH= before the invocation of . /etc/profile.d/modules.sh in the container definition. I confirmed that in a test. He also found that the cause of the problem is a regression in a dependent package, mvdan.cc/sh, because downgrading to version 3.2.4 of that package makes the problem go away.

[cclerget]

Fix mvdan/sh#759 on its way

[DrDaveD]

Is this a significant enough problem for a 3.8.5 release or can it wait to 3.9?

[xfong]

Hi Dave, I think it is significant enough for 3.8.5 release because this preserves the expected behavior of the shell. The method is used for managing multiple environments within the container. When is 3.9 expected to be released? Best, Xuanyao (Kelvin) Fong

…

Sent from my iPhone

On Nov 18, 2021, at 01:59, DrDaveD ***@***.***> wrote:  Is this a significant enough problem for a 3.8.5 release or can it wait to 3.9? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[DrDaveD]

There's no official plan yet for 3.9 but my personal guess is that it will be in a couple of months.