Skip to content

Commit

Permalink
Add scripts for setting up an environment on GFDL's PP/AN (#365)
Browse files Browse the repository at this point in the history
This PR adds scripts for setting up an environment to run the pace example notebooks on GFDL's Post Processing and Analysis Cluster (PP/AN).
  • Loading branch information
spencerkclark authored Oct 25, 2022
1 parent e7ec2e5 commit 749ce94
Show file tree
Hide file tree
Showing 3 changed files with 85 additions and 0 deletions.
26 changes: 26 additions & 0 deletions examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,3 +37,29 @@ The `notebooks` directory contains several notebooks which explain different fun
The Jupyter notebooks rely on an MPI-environment and use ipyparallel and mpi4py for setting up the parallel environment. If you are not familiar with these packages, please read up on them before getting started.

If you prefer to run outside of a Docker container, you will need to install several pre-requisites. The Dockerfile can serve as a recipe of what they are and how to install them.

## Building an environment to run these examples on the GFDL Post Processing and Analysis Cluster (PP/AN)

Within the `build_scripts` directory are a couple scripts relevant for setting up an environment to run these notebook examples on the [GFDL Post Processing and Analysis Cluster (PP/AN)](https://www.noaa.gov/organization/information-technology/ppan). To create an environment, log in to either the `jhan` or `jhanbigmem` node on the analysis cluster clone this repository, and then navigate to the `build_scripts` directory:

```
$ ssh analysis
$ git clone --recursive https://github.com/ai2cm/pace.git
$ cd pace/examples/build_scripts
```

From there, run the installation script. This script takes two arguments. The first is to a directory you would like to install all the software into, and the second is a name you would like to give the conda environment the script creates. Since conda environments can take up a reasonable amount of space, it is recommended that you install things in your `/work` directory rather than your `/home` directory:

```
$ bash build_ppan.sh /work/$USER/pace-software pace
```

This will take a few minutes to complete, but assuming it finishes succesfully you should then be able to activate the environment and run the example notebooks. To do so, first source the `activate_ppan.sh` script. This loads some modules, activates the appropriate conda environment, and sets some environment variables necessary for running the examples. You will then be able to start JupyterLab following [GFDL's recommended approach](https://wiki.gfdl.noaa.gov/index.php/Python_at_GFDL#Using_Jupyter_Hubs_or_Notebooks_on_GFDL_Workstations_and_PPAN):

```
$ source activate_ppan.sh /work/$USER/pace-software pace
$ cd ..
$ jhp launch lab
```

It will take a minute or so for the server to start up. If you are on GFDL's network or have [configured your proxy settings appropriately](https://wiki.gfdl.noaa.gov/index.php/Creating_a_GFDL_SSH_Tunnel), you will then be able to navigate to the URL produced by the command in your local browser. This will take you to the JupyterLab interface, where you can open and run the example notebooks. Note that it can take some time to initially connect to the notebook kernel, so be patient.
13 changes: 13 additions & 0 deletions examples/build_scripts/activate_ppan.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
#!/bin/bash

INSTALL_PREFIX=$1
ENVIRONMENT_NAME=$2

PACE_DIR=$(pwd)/../..
FV3NET_DIR=$INSTALL_PREFIX/fv3net

module load conda
module load intel_compilers/2021.3.0

conda activate $ENVIRONMENT_NAME
export PYTHONPATH=$FV3NET_DIR/external/fv3viz:$PACE_DIR/external/gt4py/src
46 changes: 46 additions & 0 deletions examples/build_scripts/build_ppan.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
#!/bin/bash

set -e

INSTALL_PREFIX=$1
ENVIRONMENT_NAME=$2

PACE_DIR=$(pwd)/../..
FV3NET_DIR=$INSTALL_PREFIX/fv3net

# module load necessary system software
module load conda
module load intel_compilers/2021.3.0

export MPICC=$(which mpicc)

CONDA_PREFIX=$INSTALL_PREFIX/conda
conda config --add pkgs_dirs $CONDA_PREFIX/pkgs
conda config --add envs_dirs $CONDA_PREFIX/envs

# enter the pace directory
cd $PACE_DIR

# create a conda environment with cartopy and its dependencies installed
conda create -c conda-forge -y --name $ENVIRONMENT_NAME python=3.8 matplotlib==3.5.2 cartopy==0.18.0

# enter the environment and update it
conda activate $ENVIRONMENT_NAME
pip3 install --upgrade --no-cache-dir pip setuptools wheel

# install the Pace dependencies, GT4Py, and Pace
pip3 install --no-cache-dir -r requirements_dev.txt -c constraints.txt

# clone fv3net
git clone https://github.com/ai2cm/fv3net.git $FV3NET_DIR

# install jupyter and ipyparallel
pip3 install --no-cache-dir \
ipyparallel==8.4.1 \
jupyterlab==3.4.4 \
jupyterlab_code_formatter==1.5.2 \
isort==5.10.1 \
black==22.3.0

# install vcm
python3 -m pip install $FV3NET_DIR/external/vcm

0 comments on commit 749ce94

Please sign in to comment.