METplus Installation through Spack-Stack

[EdwardSnyder-NOAA]

Good Afternoon,

I'm using spack-stack with the ufs-srw-dev environment to install MET and METplus on AWS through NOAA Parallel Works. Spack-stack appears to have installed MET and METplus correctly but I run into two problems when running the SRW App MET_verification test that makes me believe something wasn't sourced or setup properly.

1. When running the MET_verification test, I was getting the "cannot open shared object file: No such file or directory” for libpython. I simply added the path to the python used to install MET and METplus to LD_LIBRARY_PATH and that resolved this issue. I really don't think this is an 'issue', but added it for completion sake.
2. Then I got this error message while running the MET_verification test, which can be found in the met.plus.log files:
   09/13 15:58:30.998 metplus (command_runner.py:122) DEBUG: log_name is: pb2nc, output sent to: /contrib/Edward.Snyder/spack-stack-met-build/expt_dirs/MET_verification/log/metplus.log.pointstat_2019061500_sfc
   OUTPUT:
   WARNING:
   WARNING: get_filenames() -> can't stat "/contrib/EPIC/spack-stack/envs/ufs-srw-dev.intel/install/intel/2021.3.0/met-10.1.0-mqwqjxt/table_files"
   WARNING:
   WARNING:
   WARNING: get_filenames() -> can't stat "/contrib/EPIC/spack-stack/envs/ufs-srw-dev.intel/install/intel/2021.3.0/met-10.1.0-mqwqjxt/table_files"
   WARNING:
   ERROR :
   ERROR : recursive_envs() -> unable to open input file "/contrib/EPIC/spack-stack/envs/ufs-srw-dev.intel/install/intel/2021.3.0/met-10.1.0-mqwqjxt/config/ConfigConstants"
   ERROR :

Using this error message, I dug through the command_builder.py and noticed only MET_TMP_DIR and OMP_NUM_THREADS variables were passed so I added the MET_BASE variable (see below) to the command_builder.py and was able to get the MET_verification test to complete successfully without any errors.

• line 72 of command_builder.py

•               self.env_var_keys = [
                    'MET_TMP_DIR',
                    'OMP_NUM_THREADS',
                    'MET_BASE',
                ]
                
     self.env_var_dict['MET_BASE'] = self.config.getdir('MET_BASE')
  

Based on the second issue, I think something wasn't setup or sourced properly to pick up variables in the common.conf, or then again maybe this is a bug. Did I miss setting up or sourcing a config file before running the MET_verification test? Any guidance on this issue would be helpful. Please let me know if you need any additional information from me.

Thank you!

Eddie

[georgemccabe]

Hi Eddie,

How is MET installed in this process? Being able to see what was run and set in the environment would be helpful for diagnosing these issues. Both of these issues sound related to how MET was installed.

For (1), setting LDFLAGS to include the location of the python library files using rpath should prevent the need to set LD_LIBRARY_PATH to find those libraries.

For (2), a colleague has provided the following information:

When you configure MET using "--prefix=/a/b/c", then MET_BASE is defined in the compiled code as /a/b/c/share/met. In general, users do not need to set MET_BASE as environment variable.
If, after compiling, the data in /a/b/c/share/met is moved somewhere else, you can define $MET_BASE as an environment variable to override the compile-time setting... and point to its new location.
Since I don't see "share/met" included in the runtime error message, I'm guessing that they actually DO have $MET_BASE set as an env var and pointing to /contrib/EPIC/spack-stack/envs/ufs-srw-dev.intel/install/intel/2021.3.0/met-10.1.0-mqwqjxt. If so, tell them to unset it and try again.

If the installation location has indeed changed and you do need to set the MET_BASE environment variable when running METplus, then you can set the variable in the [user_env_vars] section of the common.conf METplus configuration file. Setting variables in the [config] section does not set environment variables, but variables found in the [user_env_vars] section will set the environment variables. However, in this case you likely do not need to set MET_BASE and should instead ensure that it is unset in the environment before running METplus.

[EdwardSnyder-NOAA]

Thank you for your response!

MET was installed using Spack-stack, which is a multi-platform package manager that builds and installs software. Spack uses package.py as a recipe script for installing packages. I attached the MET package.py file as a text file, and the MET build files from the installation.

I had a similar thought about unsetting the MET_BASE environment variable however it was already empty. So I deduced that this variable was being set during the MET verification task and the only way to fix the problem was to pass in the MET_BASE variable to the command_builder.py script.

spack-build-env.txt
spack-build-out.txt
package.txt

[georgemccabe]

Thanks for the info. I was not familiar with Spack-Stack but it sounds useful.

Do you have any outstanding issues/questions? If not, I'd like to lock this discussion to prevent future posts.

[EdwardSnyder-NOAA]

No more questions or issues. I appreciate the help. Thanks again!