EAMxx EAMxx is almost completely different in all ways from the atmosphere model used for E3SM versions 1-3.
EAMxx was built from the ground up using C++ in order to embrace modern software practices and to allow \"performance portability\" across various supercomputers. The latter goal is achieved by using the Kokkos library. EAMxx is a \"clean-start\" model with almost no similarity to the E3SM atmosphere model used in versions 1-3. Currently only the km-scale explicit-convection version called SCREAM (the Simple Cloud-Resolving E3SM Atmosphere Model) is available, but a low-resolution version is in the works.
Like the documentation for other component models, EAMxx documentation is divided into:
Put another way, all information about how to customize runs without changing code is included in the User Guide, general information about software design which is needed for intelligently modifying code goes in the Developer Guide, and details about the specific process implementations in the current model version are included in the Technical Guide.
"},{"location":"common/eamxx_params/","title":"EAMxx runtime configurable parameters","text":""},{"location":"common/eamxx_params/#atmosphere-processes-parameters","title":"Atmosphere Processes Parameters","text":""},{"location":"common/eamxx_params/#sc_import","title":"sc_import","text":"sc_export::prescribed_constants::values:
sc_export::prescribed_from_file::fields:
sc_export::prescribed_from_file::fields_alt_name:
sc_export::number_of_subcycles:
Follow these simple instructions to build and test EAMxx's standalone configuration for yourself. This document makes use of the following paths:
${RUN_ROOT_DIR}: the root directory where EAMxx is built and run${EAMXX_SRC_DIR}: the directory into which you've cloned the scream repoEAMxx's configuration and build system is based on CMake. CMake has been around a while and has gained a lot of traction in recent years, especially in the HPC community. It has good reference documentation, but it can be tricky to use if you've never encountered it. Ask a EAMxx team member for help if you're stuck on something CMake-related.
If you see a CMakeLists.txt files or a file with a .cmake suffix, that's just part of the build system. You might also see files with CTest as part of their name. These files are related to CTest, CMake's testing tool.
First, make sure you're on one of the machines supported by EAMxx, or that you have the following software installed:
First, make sure you've cloned the EAMxx repo (including all submodules) to EAMXX_SRC_DIR using the following command:
git clone --recurse-submodules https://github.com/E3SM-Project/scream\nIf you have already cloned the project and forgot to type --recurse-submodules, you can change to $EAMXX_SRC_DIR and using the following command to initialize, fetch and checkout all submodules:
git submodule update --init --recursive\nIf you're running a branch that's not master, check out this branch with
git checkout <branch>\nChange to your $RUN_ROOT_DIR directory and use CMake to configure your build.
If you're building SCREAM on one of our supported platforms, you can tell CMake to use the appropriate machine file using the -C flag. Machine files are located in $EAMXX_SRC_DIR/components/eamxx/cmake/machine-files. Take a look and see whether your favorite machine has one.
For example, to configure SCREAM on the Quartz machine at LLNL:
cd $RUN_ROOT_DIR\ncmake \\\n -DCMAKE_CXX_COMPILER=$(which mpicxx) \\\n -DCMAKE_BUILD_TYPE=Debug \\\n -C ${EAMXX_SRC_DIR}/components/eamxx/cmake/machine-files/quartz.cmake \\\n ${EAMXX_SRC_DIR}/components/eamxx\nIf you're building on a machine that doesn't have a ready-made machine file, you can try configuring your build by manually passing options to CMake. This usually looks something like the following, which configures EAMxx to compile CPU code using Kokkos's OpenMP backend:
cd $RUN_ROOT_DIR\ncmake \\\n -D CMAKE_BUILD_TYPE=Debug \\\n -D CMAKE_C_COMPILER=mpicc \\\n -D CMAKE_CXX_COMPILER=mpicxx \\\n -D CMAKE_Fortran_COMPILER=mpif90 \\\n -D MPIEXEC_EXECUTABLE=`which mpiexec` \\\n -D EKAT_MPI_NP_FLAG:STRING=-n \\\n -D SCREAM_DYNAMICS_DYCORE=HOMME \\\n -D SCREAM_DOUBLE_PRECISION:BOOL=ON \\\n -D SCREAM_INPUT_ROOT:PATH=/path/to/scream-input \\\n -D Kokkos_ENABLE_DEBUG=TRUE \\\n -D Kokkos_ENABLE_AGGRESSIVE_VECTORIZATION=OFF \\\n -D Kokkos_ENABLE_SERIAL=ON \\\n -D Kokkos_ENABLE_OPENMP=ON \\\n -D Kokkos_ENABLE_LIBDL=OFF \\\n -D Kokkos_ENABLE_PROFILING=OFF \\\n -D Kokkos_ENABLE_DEPRECATED_CODE=OFF \\\n -D KOKKOS_ENABLE_ETI:BOOL=OFF \\\n -D NetCDF_C_PATHS=/path/to/netcdf-c-dir \\\n -D NetCDF_Fortran_PATHS=/path/to/netcdf-f90-dir \\\n -D PnetCDF_C_PATHS=/path/to/pnetcdf-dir \\\n -D PnetCDF_Fortran_PATHS=/path/to/pnetcdf-f90-dir \\\n ${EAMXX_SRC_DIR}/components/eamxx\nIn either case, EAMxx requires MPI-aware compilers. Let's examine these options (only some of which are required on any given machine) to make sure we know what they do:
CMAKE_BUILD_TYPE: specifies whether you are building EAMxx in a developer-friendly configuration (Debug), for a production run (Release) or for performance profiling or some other specialized purpose. Typically, you'll set this option to Debug or Release.CMAKE_{C,CXX,Fortran}_COMPILER: the name of the command used to invoke an MPI-enabled C, C++, or Fortran compiler to build EAMxxMPIEXEC_EXECUTABLE: the name of the command used to run EAMxx using MPI, typically mpiexec or mpirun, but possibly different depending on your desired machineEKAT_MPI_NP_FLAG: the flag passed to MPIEXEC_EXECUTABLE that you use to specify the number of desired MPI processes. This is typically -n for mpiexec and -np for mpirun.SCREAM_DYNAMICS_DYCORE: specifies the dycore used for configuring EAMxx, which is NONE if you are not configuring EAMxx to run its dycore-related tests, or HOMME if you want to use HOMMExxSCREAM_DOUBLE_PRECISION: indicates whether EAMxx's Real type is a double-precision (ON) or single-precision (OFF) floating point typeSCREAM_INPUT_ROOT: specifies the location of the top-level folder that stores input data files for EAMxx. This folder is populated with input files which are downloaded automatically during EAMxx's build process.Kokkos_) are described in the Kokkos WikiNetCDF_C_PATHS: specifies one or more folders in which the NetCDF C library and headers are installed. In the simplest configuration, the headers should be located in ${NetCDF_C_PATHS}/include and the library should live in ${NetCDF_C_PATHS}/lib.NetCDF_Fortran_PATHS: specifies one or more folders in which the NetCDF Fortran library and modules are installed. Analogous to ${NetCDF_C_PATHS}, .mod files should be in ${NetCDF_Fortran_PATHS}/include, and the library should be installed in ${NetCDF_Fortran_PATHS}/lib.PnetCDF_C_PATHS: specifies one or more folders in which the pNetCDF C library and headers are installed, analogous to NetCDF_C_PATHS.PnetCDF_Fortran_PATHS: specifies one or more folders in which the pNetCDF Fortran library and modules are installed, analogous to NetCDF_Fortran_PATHS.Above, we've configured Debug builds to make it easier to find and fix errors. For performance testing, you should configure a Release build and make use of other options, depending on your architecture.
Now you can build SCREAM from that same directory:
make -j\nThe -j flag tells Make to use threads to compile in parallel. If you like, you can set the number of threads by passing it as an argument to -j (e.g. make -j8).
You can run EAMxx's tests to make sure your build works by following the instructions here.
"},{"location":"developer/","title":"SCREAM Developer Guide","text":""},{"location":"developer/ci_nightly/","title":"Continuous Integration and Nightly Testing","text":""},{"location":"developer/ci_nightly/#autotester","title":"Autotester","text":"EAMxx using github actions and a Sandia product called Autotester 2 to run CI testing on a CPU and GPU machine for every github pull request. By default, we run the e3sm_scream_v1_at suite and the standalone eamxx tests (test-all-scream).
"},{"location":"developer/ci_nightly/#nightly-overview-cdash","title":"Nightly overview, CDash","text":"Our nightly testing is much more extensive than the CI testing. You can see our dashboard here under the section \"E3SM_SCREAM\": https://my.cdash.org/index.php?project=E3SM
We run a variety of CIME test suites and standalone testing on a number of platforms. We even do some performance testing on frontier.
"},{"location":"developer/cime_testing/","title":"Full Model Testing","text":"Full model system testing of eamxx is done through CIME test cases (much like the rest of E3SM).
We offer a number of test suites, including:
Example for running a suite:
cd $repo/cime/scripts\n./create_test e3sm_scream_v1_at --wait\nExample for running a single test case:
cd $repo/cime/scripts\n./create_test SMS.ne4_ne4.F2010-SCREAMv1 --wait\nThere are many behavioral tweaks you can make to a test case, like changing the run length, test type, etc. Most of this is not specific to eamxx and works for any CIME case. This generic stuff is well-documentated here: http://esmci.github.io/cime/versions/master/html/users_guide/testing.html
When it comes to things specific to eamxx, you have grids, compsets, and testmods.
Common EAMxx grids are:
More grid info can be found here: https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/933986549/ATM+Grid+Resolution+Summary
Common EAMxx compsets are:
Full info on supported compsets can be found by looking at this file: $scream_repo/components/eamxx/cime_config/config_compsets.xml
Common EAMxx testmods are:
More info on running EAMxx can be found here: https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/3386015745/How+To+Run+EAMxx+SCREAMv1
"},{"location":"developer/field/","title":"Field","text":"In EAMxx, a Field is a data structure holding two things: pointers to the data and pointers to metadata. Both the data and metadata are stored in std::shared_ptr instances, to ensure consistency across all copies of the field. This allows for fast shallow copy semantic for this class.
The data is stored on both CPU and device memory (these may be the same, depending on the Kokkos backend). In EAMxx, we always assume and guarantee that the device data is up to date. That implies that the data must be explicitly synced to host before using it on host, and explicitly synced to device after host manipulation, in order to ensure correctness. In order to access the data, users must use the get_view/ get_strided_view methods, which takes two template arguments: the data type, and an enum specifying whether CPU or device data is needed. The data type is used to reinterpret the generic pointer stored inside to a view of the correct scalar type and layout. It is a possibly const-qualified type, and if the field was marked as \"read-only\", the method ensures that the provided data type is const. A read-only field can be created via the getConst method, which returns a shallow copy of the field, but marked as read-only. The enum specifying host or device data is optional, with device being the default.
The metadata is a collection of information on the field, such as name, layout, units, allocation size, and more. Part of the metadata is immutable after creation (e.g., name, units, or layout), while some metadata can be partially or completely modified. The metadata is contained in the FieldHeader data structure, which contains four parts:
FieldIdentifier: stores the field's name, layout, units, data type, and name of the grid where it's defined. These information are condensed in a single string, that can be used to uniquely identify a field, allowing to distinguish between different version of the same field. The layout is stored in the FieldLayout data structure, which includes:std::vector<FieldTag>, they give context to the field's extents.std::vector<int>, as well as a 1d Kokkos::View.FieldTracking: stores information on the usage of the field, as well as its possible connections to other fields. In particular, the tracked items are:FieldAllocProp: stores information about the allocation. While the field is not yet allocated, users can request special allocations for the field, for instance to accommodate packing (for SIMD), which may require padding. Upon allocation, this information is then used by the Field structure to extract the actual data, wrapped in a properly shaped Kokkos::View. The alloc props are also responsible of tracking additional information in case the field is a \"slice\" of a higher-dimensional one, a fact that can affect how the data is accessed.std::map<std::string,ekat::any>, allows to catch any metadata that does not fit in the above structures. This is a last resort structure, intended to accommodate the most peculiar corner cases, and should be used sparingly.In EAMxx, the AbstractGrid is an interface used to access information regarding the horizontal and vertical discretization. The most important information that the grid stores is:
PointGrid (a class derived from AbstractGrid) is a simple collection of points, so the \"native\" indexing system coincides with the LIDs. However, for a SEGrid (a derived class, for spectral element grids), the \"native\" indexing is a triplet (ielem,igp,jgp), specifying the element index, and the two indices of the Gauss point within the element.std::map<std::string,Field>, this represent any data that is intrinsically linked to the grid (either along the horizontal or vertical direction), such as lat/lon coordinates, vertical coordinates, area associated with the DOF.Grids can also be used to retrieve the layout of a 2d/3d scalar/vector field, which allows certain downstream classes to perform certain operations without assuming anything on the horizontal grid.
In general, grid objects are passed around the different parts of EAMxx as const objects (read-only). The internal data can only be modified during construction, which usually is handled by a GridsManager object.
In EAMxx, I/O is handled through the SCORPIO library, currently a submodule of E3SM. The scream_io library within eamxx allows to interface the EAMxx infrastructure classes with the SCORPIO library.
EAMxx uses Kokkos for performance portable abstractions for parallel execution of code and data management to various HPC platforms, including OpenMP, Cuda, HIP, and SYCL. Here we give a brief overview of the important concepts for understanding Kokkos in EAMxx. For a more in depth description, see the Kokkos wiki.
"},{"location":"developer/kokkos_ekat/#kokkosdevice","title":"Kokkos::Device","text":"Kokkos::Device is a struct which contain the type definitions for two main Kokkos concepts: execution space (Kokkos::Device::execution_space), the place on-node where parallel operations (like for-loops, reductions, etc.) are executed, and the memory space (Kokkos::Device::memory_space), the memory location on-node where data is stored. Given your machine architecture, Kokkos defines a default \"device\" space, given by
Kokkos::Device<Kokkos::DefaultExecutionSpace,\n Kokkos::DefaultExecutionSpace::memory_space>\nwhere all performance critical code should be executed (e.g., on an NVIDIA machine, this device would be the GPU accelerators) and a default \"host\" space, given by
Kokkos::Device<Kokkos::DefaultHostExecutionSpace,\n Kokkos::DefaultHostExecutionSpace::memory_space>\nwhere data can be accessed by the CPU cores and is necessary for I/O interfacing, for example. Currently, these default spaces are the ones used by EAMxx. On CPU-only machines, host and device represent the same space.
"},{"location":"developer/kokkos_ekat/#kokkos-views","title":"Kokkos Views","text":"The main data struct provided by Kokkos used in EAMxx in the Kokkos::View. This is a multi-dimensional data array that can live on either device or host memory space. These Views are necessary when running on GPU architectures as data structures like std::vector and std::array will be unavailable on device.
Views are constructed in EAMxx most commonly with the following template and input arguments
Kokkos::View<DataType, LayoutType, DeviceType>(const std::string& label,\n int dim0, int dim1, ...)\nwhere
DataType: scalar type of the view, given as ScalarType+*(x's number of run-time dimensions). E.g., a 2D view of doubles will have DataType = double**. There is also an ability to define compile-time dimensions by using [], see [Kokkos wiki section on views] (https://kokkos.org/kokkos-core-wiki/API/core/view/view.html).LayoutType: mapping of indices into the underlying 1D memory storage. Types are:LayoutRight (used in EAMxx): strides increase from the right most to the left most dimension, right-most dimension is contiguousLayoutLeft: strides increase from the left most to the right most dimension, left-most dimension is contiguousLayoutStride: strides can be arbitrary for each dimensionDeviceType: provides space where data live, defaults to the default deviceThe following example defines a view \"temperature\" which has dimensions columns and levels:
Kokkos::View<double**, Kokkos::LayoutRight, Kokkos::DefaultDevice> temperature(\n \"temperature\", ncols, nlevs);\nKokkos provides Kokkos::deep_copy(dst, src) which copies data between views of the same dimensions, or a scalar values into a view. Common uses
Kokkos::deep_copy(view0, view1); // Copy all data from view1 into view0\nKokkos::deep_copy(view0, 5); // Set all values of view0 to 5\nAs seen in the next section, we can use deep_copy() to copy data between host and device.
We will often need to have memory allocation the resides on device (for computation), and then need that identical data on host (say, for output). Kokkos has a concept of mirror views, where data can be copied from host to device and vice versa.
Here is an example using the device view temperature from above
// Allocate view on host that exactly mirrors the size of layout of the device\nview\nauto host_temperature = Kokkos::create_mirror_view(temperature);\n\n// Copy all data from device to host\nKokkos::deep_copy(host_temperature, temperature);\nKokkos also offers an all-in-one option
// Note: must hand the host device instance as first argument\nauto host_temperature = Kokkos::create_mirror_view_and_copy(\n Kokkos::DefaultHostDevice(), temperature);\nThe most basic parallel execution pattern used by EAMxx is the Kokkos::parallel_for which defines a for-loop with completely independent iterations. The parallel_for takes in an optional label for debugging, an execution policy, which defines a range and location (host or device) for the code to be run, and a lambda describing the body of code to be executed. The following are execution policies used in EAMxx
int count: 1D iteration range [0, count)RangePolicy<ExecSpace>(int beg, int end): 1D iteration range for indices [beg, end)MDRangePolicy<ExecSpace, Kokkos::Rank<N>>(int[N] beg, int[N] end): multi- dimensional iteration range [beg, end)TeamPolicy<ExecSpace>(int league_size, int team_size, int vector_size): 1D iteration over league_size, assigned to thread teams of size team_size, each with vector_size vector lanes. Both team_size and vector_size can be given Kokkos::AUTO as input for Kokkos to automatically compute.If no ExecSpace template is given, the default execution space is used.
For lambda capture, use KOKKOS_LAMBDA macro which sets capture automatically based on architecture.
Example using RangePolicy to initialize a view
Kokkos::View<double**, Kokkos::LayoutRight> temperature(\"temperature\", ncols,\n nlevs);\nKokkos::parallel_for(\"Init_temp\",\n Kokkos::RangePolicy(0, ncols*nlevs),\n KOKKOS_LAMBDA (const int idx) {\n int icol = idx/nlevs;\n int ilev = idx%nlevs;\n\n temperature(icol, ilev) = 0;\n});\nSame example with TeamPolicy
Kokkos::parallel_for(\"Init_temp\",\n Kokkos::TeamPolicy(ncols*nlevs, Kokkos::AUTO, Kokkos::AUTO),\n KOKKOS_LAMBDA (const TeamPolicy::member_type& team) {\n // league_rank() gives the index for this team\n int icol = team.league_rank()/nlevs;\n int ilev = team.league_rank()%nlevs;\n\n temperature(icol, ilev) = 0;\n});\nUsing TeamPolicy, we can have up to three nested levels of parallelism: team parallelism, thread parallelism, vector parallelism. These nested policies can be called within the lambda body using the following execution policies
TeamThreadRange(team, begin, end): execute over threads of a teamTeamVectorRange(team, begin, end): execute over threads and vector lanes of a teamThreadVectorRange(team, begin, end): execute over vector lanes of a threadAn example of using these policies
Kokkos::View<double***> Q(\"tracers\", ncols, ntracers, nlevs);\nKokkos::parallel_for(Kokkos::TeamPolicy(ncols, Kokkos::AUTO),\n KOKKOS_LAMBDA (TeamPolicy::member_type& team) {\n int icol = team.league_rank();\n Kokkos::parallel_for(Kokkos::TeamVectorRange(team, nlevs), [&](int ilev) {\n temperature(icol, ilev) = 0;\n });\n\n Kokkos::parallel_for(Kokkos::TeamThreadRange(team, nlevs), [&](int ilev) {\n Kokkos::parallel_for(Kokkos::ThreadVectorRange(team, ntracers), [&](int iq) {\n Q(icol, iq, ilev) = 0;\n });\n });\n});\nIMPORTANT! Nested policies cannot be used in arbitrary order. ThreadVectorRange must be used inside a TeamThreadRange, and TeamVectorRange must be the only level of nested parallelism.
Kokkos::parallel_for(TeamPolicy(...), ... {\n // OK\n Kokkos::parallel_for(TeamThreadRange, ... {\n\n });\n\n // OK\n Kokkos::parallel_for(TeamThreadRange, ... {\n Kokkos::parallel_for(ThreadVectorRange, ... {\n\n });\n });\n\n // OK\n Kokkos::parallel_for(TeamVectorRange, ...{\n\n });\n\n // WRONG,ThreadVectorRange must be nested in TeamThreadRange\n Kokkos::parallel_for(ThreadVectorRange, ...{\n\n });\n\n // WRONG, a TeamVectorRange must be the only nested level\n Kokkos::parallel_for(TeamVectorRange, ...{\n Kokkos::parallel_for(ThreadVectorRange, ... {\n\n });\n });\n});\nUsing these incorrectly can be very tricky to debug as the code almost certainly will not error out, but race conditions will exist among threads.
"},{"location":"developer/kokkos_ekat/#ekat","title":"EKAT","text":""},{"location":"developer/kokkos_ekat/#kokkostypes","title":"KokkosTypes","text":""},{"location":"developer/kokkos_ekat/#exespaceutils","title":"ExeSpaceUtils","text":""},{"location":"developer/kokkos_ekat/#vectorization-packs","title":"Vectorization: Packs","text":""},{"location":"developer/kokkos_ekat/#scratch-memory-worspacemanager","title":"Scratch Memory: WorspaceManager","text":""},{"location":"developer/kokkos_ekat/#algorithms","title":"Algorithms","text":""},{"location":"developer/managers/","title":"FieldManager and GridsManager","text":""},{"location":"developer/processes/","title":"Atmospheric Processes","text":"In EAMxx, AtmosphereProcess (AP) is an abstract class representing a portion of the atmosphere timestep algorithm. In simple terms, an AP is an object that given certain input fields performs some calculations to compute some output fields. The concrete AP classes allow to create a buffer layer between particular packages (e.g., dynamics dycore, physics parametrizations) and the atmosphere driver (AD), allowing separation of concerns, so that the AD does not need to know details about the package, and the package does not need to know about the EAMxx infrastructure.
To enhance this separation of concerns, EAMxx implements two more classes for handling APs:
AtmosphereProcessGroup (APG), which allows to group together a set of AP's, which can be seen from outside as a single process;AtmosphereProcessFactory class, which allows an APG to create its internal processes without any knowledge of what they are.This infrastructure allows the AD to view the whole atmosphere as a single APG, and to be completely agnostic to what processes are run, and in which order. This design allows to have a code base that is cleaner, self-container, and easy to test via a battery of targeted unit tests.
In EAMxx, we already have a few concrete AP's, interfacing the AD to the Hommexx non-hydrostatic dycore as well as some physics parametrizations (P3, SHOC, RRMTPG, etc). In the next section we describe the interfaces of an AP class, and we show an example of how to write a new concrete AP class.
"},{"location":"developer/processes/#atmosphere-process-interfaces","title":"Atmosphere process interfaces","text":"An AP has several interfaces, which can be grouped into three categories:
Among the above, the initialization sequence is the most complex, and consists of several steps:
GridsManager to the AP's, so that they can get information about the grids they need. At this point, all AP's have all the information they need to establish the layout of the input and output fields they need, and can store a list of these \"requests\"While the base AP class provides an (empty) implementation for some methods, in case derived classes do not need a feature, some methods are purely virtual, and concrete classes will have to override them. Looking at existing concrete AP implementations is a good way to have a first idea of what a new AP class needs to implement. Here, we show go over the possible implementation of these methods in a hypothetical AP class. The header file may look something like this
#include <share/atm_process/atmosphere_process.hpp>\n\nclass MyProcess : public AtmosphereProcess\n{\npublic:\n using gm_ptr = std::shared_ptr<const GridsManager>;\n\n MyProcess(const ekat::Comm& comm, const ekat::ParameterList& pl);\n\n std::string name () const override { return \"my_fancy_process\"; }\n void set_grids (const gm_ptr& grids_manager) override;\n size_t requested_buffer_size_in_bytes () const override;\n void init_buffers (const ATMBufferManager& buffer_manager) override;\nprotected:\n\n void initialize_impl (const RunType run_type) override;\n void run_impl (const double dt) override;\n void finalize_impl () override;\n\n using view_1d = typename KokkosTypes<DefaultDevice>::view_1d<Real>;\n using view_2d = typename KokkosTypes<DefaultDevice>::view_2d<Real>;\n\n view_1d m_temp1;\n view_2d m_temp2;\n\n int m_ncols;\n int m_nlevs;\n bool m_has_blah;\n};\nA few comments:
override when overriding a method; in case of small typos (e.g., missing a & or a const, the compiler will be erroring out, since the signature will not match any virtual method in the base class;finalize_impl is often empty; unless the AP is managing external resources, everything should be correctly released during destruction;Here is a possible implementation of the methods, with some inline comments to explain
MyProcess::MyProcess (const ekat::Comm& comm, const ekat::ParameterList& pl)\n : AtmosphereProcess(comm,pl)\n{\n // The base class copies pl into protected member m_params\n m_has_blah = m_params.get<bool>(\"enable_blah\");\n}\n\nvoid MyProcess::set_grids (const std::shared_ptr<GridsManager>& gm)\n{\n using namespace ekat::units;\n const auto nondim = Units::nondimensional();\n\n auto grid = gm->get_grid(\"Physics\");\n m_ncols = grid->get_num_local_dofs();\n m_nlevs = grid->get_num_vertical_levels();\n\n // In these names, 2d refers to \"horizontal only\", while 3d to \"horiz+vert\".\n // But the grid stores dofs linearly, so there is only one array dimension\n FieldLayout scalar2d = grid->get_2d_scalar_layout();\n FieldLayout vector3d = grid->get_3d_vector_layout(true,2);\n\n // Declare fields needed:\n // - Required: 'input' (read-only)\n // - Computed: 'output'\n // - Updated: 'input'+'output'\n // Tell the AD we need 'velocity' to accommodate a Pack scalar type\n add_field<Required>(\"coeff_2d\",scalar2d,nondim,grid->name);\n add_field<Updated>(\"velocity\",vector3d,m/s,grid->name,SCREAM_PACK_SIZE);\n}\n\nsize_t MyProcess::requested_buffer_size_in_bytes ()\n{\n // We use temp2 only if the blah feature is on\n return m_ncols + (m_has_blah ? m_ncols*m_nlev : 0);\n}\n\nvoid MyProcess::init_buffers (const ATMBufferManager& buffer_manager)\n{\n auto mem = reinterpret_cast<Real*>(buffer_manager.get_memory());\n\n m_temp1 = view_1d<Real>(mem,m_ncols);\n mem += m_ncols;\n\n if (m_has_blah) {\n m_temp2 = view_2d<Real>(mem,m_ncols,m_nlevs);\n mem += m_ncols*m_nlevs;\n }\n\n // Make sure we use exactly the mem we said we would\n size_t used_mem = (mem - buffer_manager.get_memory())*sizeof(Real);\n EKAT_REQUIRE_MSG(used_mem==requested_buffer_size_in_bytes(),\n \"Error! Used memory != requested memory for MyProcess.\"\n \" used memory: \" + std::to_string(used_mem) + \"\\n\"\n \" requested: \" + std::to_string(requested_buffer_size_in_bytes()) + \"\\n\");\n}\n\nvoid MyProcess::initialize_impl(const RunType run_type)\n{\n // Can complete any initialization of the background pkg\n my_process_pkg_init(m_has_blah);\n}\n\nvoid MyProcess:run_impl (const double dt)\n{\n using Policy = typename KokkosTypes<DefaultDevice>::TeamPolicy\n using Member = typename KokkosTypes<DefaultDevice>::MemberType\n using PackT = ekat::Pack<Real,SCREAM_PACK_SIZE>;\n\n // Create team policy\n Policy policy(m_ncols,m_nlevs,1);\n\n // Create local copies of class members (or else use KOKKOS_CLASS_LAMBDA)\n auto temp1 = m_temp1;\n auto temp2 = m_temp2;\n auto do_blah = m_has_blah;\n\n // Get views from fields. We \n auto coeff2d = get_field_in(\"coeff_2d\").get_view<const Real*>();\n auto velocity = get_field_out(\"velocity\").get_view<PackT**>();\n\n // Since we process velocity with a Pack scalar type, find out how many packs\n // we have in each column\n auto nlevs_packs = ekat::PackInfo<SCREAM_PACK_SIZE>::num_packs(m_nlevs);\n\n // Call some function in the background pkg\n do_some_work (coeff_2d,velocity,temp1,temp2,do_blah);\n\n // Do some more work here\n auto work = KOKKOS_LAMBDA (const Member& team) {\n int icol = team.league_rank();\n auto col_work = [&](const int ilev) {\n velocity(icol,ilev) *= coeff_2d;\n };\n Kokkos::parallel_for(Kokkos::TeamVectorRange(team,nlevs_packs),col_work);\n };\n Kokkos::parallel_for(policy,work);\n Kokkos::fence();\n}\n\nvoid MyProcess::finalize_impl ()\n{\n // If the background package needs to cleanup something, do it now\n my_process_pkg_cleanup();\n}\nAll EAMxx-specific code can be found in components/eamxx within the EAMxx repo. Here's how things are organized:
cime_config: Tools and XML files for integrating EAMxx with E3SM via the CIME framework.cmake: CMake functions and macros used by the configuration/build system.data: Data files used by our tests.docs: Documentation for the EAMxx project, including design documents, instructions for building and testing EAMxx, and this document.scripts: Miscellaneous scripts that implement workflows for running tests and analyzing performance.src: All C++ source code (and any bridges to Fortran) for EAMxx are stored here. We describe the contents of this directory in greater detail below.tests: Implements standalone, end-to-end tests for various EAMxx components (RRTMG, HOMME, P3, SHOC, etc).In addition, you'll notice the following files in components/eamxx:
CMakeLists.txt: The CMake file that defines EAMxx's configuration/build system.CTestConfig.cmake: This CTest file contains parameters that determine how our test results are reported to the E3SM CDash Site.README.md: EAMxx's top-level README file, which describes the project and its purpose.mkdocs.yml: The configuration file for mkdocs, the tool we currently use to build and publish our documentation.src Directory","text":"Herein l\u0456es the source code for EAMxx. Broadly, here's where things are:
control: Contains the atmosphere driver and basic tests for it.dynamics: Here's where HOMME lives within EAMxx, along with code for interfacing with it using EAMxx's data structures.mct_coupling: Glue code for embedding EAMxx within E3SM as an atmosphere component using the MCT coupler.physics: Source code for physics-related atmospheric processes, includingp3: The C++/Kokkos implementation of P3 microphysics within EAMxx.shoc: The C++/Kokkos implementation of SHOC macrophysics within EAMxx.rrtmgp: A stub for the radiation processes as represented in EAMxx.share: Utilities and data structures common to these processes.share: Utilities used by various components within EAMxx. Of note:io: EAMxx's interface to the SCORPIO library.diagnostics: A collection of simple classes used to compute diagnostic quantities.Each of these directories contains a CMakeLists.txt file for defining how things are build, and a tests/ subdirectory that houses relevant unit and verification tests.
You'll also see some other files in the src/ directory itself, such as
scream_config.h.in: A template for generating a C++ header file with EAMxx configuration information.In this section we describe our testing methodology for standalone EAMxx configurations. We use several types of tests
We also support a test-all-scream configuration that runs all of the standalone tests for an EAMxx configuration. Note, your current machine must be known to EAMxx before test-all-scream will work. A machine can be made known to EAMxx by editing the eamxx/scripts/machines_specs.py files. There are some instructions on what to do at the top of this file.
test-all-scream has a good help dump
cd $scream_repo/components/eamxx\n./scripts/test-all-scream -h\nIf you are unsure of the cmake configuration for you development cycle, one trick you can use is to run test-all-scream for the dbg test and just copy the cmake command it prints (then ctrl-C the process).
cd $scream_repo/components/eamxx\n./scripts/test-all-scream -t dbg -m $machine\n# wait for a few seconds*\n# Ctrl-C *\n# Copy the contents of DCMAKE_COMMAND that was passed to ctest *\n# Add \"cmake\" to beginning of contents and path to eamxx at the end. *\nConsiderations for using test-all-scream:
cd eamxx/scripts; eval $(./scripts/scream-env-cmd $machine)Before running the tests, generate a baseline file:
cd $RUN_ROOT_DIR\nmake baseline\nThe tests will run, automatically using the baseline file, which is located in the CMake-configurable path ${SCREAM_BASELINES_DIR}. By default, this path is set to an invalid string. If baselines tests are enabled, we check that a valid path has been provided.
To run all of SCREAM's tests, make sure you're in $RUN_ROOT_DIR and type
ctest -VV\nThis runs everything and reports results in an extra-verbose (-VV) manner.
You can also run subsets of the SCREAM tests. For example, to run only the P3 regression tests (again, from the $RUN_ROOT_DIR directory), use
ctest -R p3_regression\nWe can create groupings of tests by using labels. For example, we have a driver label that runs tests for SCREAM's standalone driver. You can see a list of available labels by typing
ctest --print-labels\nTo see which tests are associated with a given label (e.g. driver), use
ctest -L driver -N\np3_regression Suite","text":"p3_regression uses a baseline file to compare any new or altered implementations with our P3 Fortran reference implementation. If you're working on the C++/Kokkos implementation, you can invoke any new tests to the function Baseline::run_and_cmp in ${SCREAM_SRC_DIR}/components/eamxx/p3/tests/p3_run_and_cmp.cpp.
If the reference Fortran implementation changes enough that a new baseline file is required, make sure to let other SCREAM team members know, in order to minimize disruptions.
"},{"location":"developer/style_guide/","title":"SCREAM C++ Style Guide","text":"Here's our style guide. Let the holy wars begin!
"},{"location":"developer/style_guide/#types","title":"Types","text":""},{"location":"developer/style_guide/#functions-and-methods","title":"Functions and Methods","text":""},{"location":"developer/style_guide/#variables","title":"Variables","text":""},{"location":"technical/","title":"EAMxx Technical Guide","text":"The goal of this document is to describe the specific equations, parameterizations, and numerical methods used in the current version of EAMxx. Because our master-branch implementation changes every time we make a new commit, this documentation will also evolve continuously. As such, documentation for master should always be considered to be preliminary and under construction. If you want trustworthy documentation, pull it from an official model release.
"},{"location":"technical/#overview","title":"Overview","text":"Currently, EAMxx is only configured for km-scale convection-permitting runs. In order to provide scientifically-credible simulations at lower resolution, parameterizations for the following processes would be needed:
The only configuration of EAMxx that is currently implemented is the convection-permitting version, commonly known as the Simple Cloud-Resolving E3SM Atmosphere Model (SCREAM). Processes in EAMxx-SCREAM are:
By default processes are called in this order, but which processes to include and in what order is modifiable at run time. After all atmospheric processes are called, output is written. Surface components are then called before the next atmosphere step starts. These processes are described in more detail in Caldwell et al. (2021)6. As in EAM, dynamics operates on a spectral element grid and all other processes use a finite-volume grid that divides each spectral element into 4 quadrilaterals. This physics grid is described in Hannah et al. (2021)8.
Mark A. Taylor, Oksana Guba, Andrew Steyer, Paul A. Ullrich, David M. Hall, and Christopher Eldred. An energy consistent discretization of the nonhydrostatic equations in primitive variables. Journal of Advances in Modeling Earth Systems, 12(1):e2019MS001783, 2020. e2019MS001783 10.1029/2019MS001783. URL: https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2019MS001783, arXiv:https://agupubs.onlinelibrary.wiley.com/doi/pdf/10.1029/2019MS001783, doi:https://doi.org/10.1029/2019MS001783.\u00a0\u21a9
A. M. Bradley, P. A. Bosler, and O. Guba. Islet: interpolation semi-lagrangian element-based transport. Geoscientific Model Development, 15(16):6285\u20136310, 2022. URL: https://gmd.copernicus.org/articles/15/6285/2022/, doi:10.5194/gmd-15-6285-2022.\u00a0\u21a9
F. Fiedler and H. A. Panofsky. The geostrophic drag coefficient and the \u2018effective\u2019 roughness length. Quarterly Journal of the Royal Meteorological Society, 98(415):213\u2013220, 1972. URL: https://rmets.onlinelibrary.wiley.com/doi/abs/10.1002/qj.49709841519, arXiv:https://rmets.onlinelibrary.wiley.com/doi/pdf/10.1002/qj.49709841519, doi:https://doi.org/10.1002/qj.49709841519.\u00a0\u21a9
B. Stevens, S. Fiedler, S. Kinne, K. Peters, S. Rast, J. M\u00fcsse, S. J. Smith, and T. Mauritsen. Macv2-sp: a parameterization of anthropogenic aerosol optical properties and an associated twomey effect for use in cmip6. Geoscientific Model Development, 10(1):433\u2013452, 2017. URL: https://gmd.copernicus.org/articles/10/433/2017/, doi:10.5194/gmd-10-433-2017.\u00a0\u21a9
Hugh Morrison and Jason A. Milbrandt. Parameterization of cloud microphysics based on the prediction of bulk ice particle properties. part i: scheme description and idealized tests. Journal of the Atmospheric Sciences, 72(1):287 \u2013 311, 2015. URL: https://journals.ametsoc.org/view/journals/atsc/72/1/jas-d-14-0065.1.xml, doi:10.1175/JAS-D-14-0065.1.\u00a0\u21a9
P. M. Caldwell, C. R. Terai, B. Hillman, N. D. Keen, P. Bogenschutz, W. Lin, H. Beydoun, M. Taylor, L. Bertagna, A. M. Bradley, T. C. Clevenger, A. S. Donahue, C. Eldred, J. Foucar, J.-C. Golaz, O. Guba, R. Jacob, J. Johnson, J. Krishna, W. Liu, K. Pressel, A. G. Salinger, B. Singh, A. Steyer, P. Ullrich, D. Wu, X. Yuan, J. Shpund, H.-Y. Ma, and C. S. Zender. Convection-permitting simulations with the e3sm global atmosphere model. Journal of Advances in Modeling Earth Systems, 13(11):e2021MS002544, 2021. e2021MS002544 2021MS002544. URL: https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2021MS002544, arXiv:https://agupubs.onlinelibrary.wiley.com/doi/pdf/10.1029/2021MS002544, doi:https://doi.org/10.1029/2021MS002544.\u00a0\u21a9\u21a9
Robert Pincus, Eli J. Mlawer, and Jennifer S. Delamere. Balancing accuracy, efficiency, and flexibility in radiation calculations for dynamical models. Journal of Advances in Modeling Earth Systems, 11(10):3074\u20133089, 2019. URL: https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2019MS001621, arXiv:https://agupubs.onlinelibrary.wiley.com/doi/pdf/10.1029/2019MS001621, doi:https://doi.org/10.1029/2019MS001621.\u00a0\u21a9
Walter M. Hannah, Andrew M. Bradley, Oksana Guba, Qi Tang, Jean-Christophe Golaz, and Jon Wolfe. Separating physics and dynamics grids for improved computational efficiency in spectral element earth system models. Journal of Advances in Modeling Earth Systems, 13(7):e2020MS002419, 2021. e2020MS002419 2020MS002419. URL: https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2020MS002419, arXiv:https://agupubs.onlinelibrary.wiley.com/doi/pdf/10.1029/2020MS002419, doi:https://doi.org/10.1029/2020MS002419.\u00a0\u21a9
The goal of the AeroCom algorithm is to calculate properties at cloud top based on the AeroCom recommendation. There are two main parts of the algorithm: probabilistically determining \"cloud top\" and then \"calculating properties\" at said cloud top.
We treat model columns independently, so we loop over all columns in parallel. We then loop over all layers in serial (due to needing an accumulative product), starting at 2 (second highest) layer because the highest is assumed to have no clouds. Let's take a photonic approach from above the model top. Let's say that \\(p_{k}\\) is the probability of a photon passing through the layer \\(k\\). We follow the maximum-random overlap assumption. In all cases, we assume the cloudiness (or cloudy fraction) is completely opaque.
We assume the highest layer has no clouds, thus the \\(p_{k} = 1\\) for the highest layer. Note that \\(p_{k}\\) is initialized as 1 for all layers. We also clip the cloudy fraction \\(C_{i,k}\\) to ensure that \\(C_{i,k} \\in [0+\\epsilon, 1-\\epsilon]\\), where \\(\\epsilon = 0.001\\). Starting at the second highest layer, \\(k+1\\), we check if some \"cloudy\" conditions are met. These conditions are now arbitrarily defined by a cloudiness threshold of \\(\\epsilon\\) (i.e., \\(C_{i,k}>\\epsilon\\)) and a non-zero threshold on the total (both liquid and ice) droplet number concentration (i.e., \\(cQ_{i,k} + iQ_{i,k} > 0\\)). If the conditions are met, we estimate the cloud-top cloud fraction using an accumulative product following the maximum-random overlap assumption.
\\[c_{i} = 1 - \\prod_{k=2}^{K} p_{k} = 1 - \\prod_{k=2}^{K} \\frac{1 - \\max(C_{i,k}, C_{i,k-1})}{1-C_{i,k-1}}\\]In order to estimate cloud-top properties, we weight by the probability of \"remaining cloudiness\" or \\(p_{k-1} - p_{k}\\).
Type Equation cloud property \\(x_{i} = \\sum_{k=2}^{K} X_{i,k} \\Phi_{i,k} (p_{k-1} - p_{k})\\) cloud content \\(x_{i} = \\sum_{k=2}^{K} \\Phi_{i,k} (p_{k-1} - p_{k})\\) other property \\(x_{i} = \\sum_{k=2}^{K} X_{i,k} (p_{k-1} - p_{k})\\)In the above, \\(\\Phi_{i,k}\\) is the thermodynamic phase defined by the cloud droplet number concentration ratios.
\\[i\\Phi_{i,k} = \\frac{iQ_{i,k}}{iQ_{i,k} + cQ_{i,k}}\\] \\[c\\Phi_{i,k} = \\frac{cQ_{i,k}}{iQ_{i,k} + cQ_{i,k}}\\]The thermodynamic phase is used only for cloud properties (e.g., cloud-top cloud droplet number concentration) or cloud content (e.g., cloud liquid content). Further, \\(X_{i,k}\\) is the three-dimensional cloud property of interest which is needed if we are converting a property from three-dimensional (\\(X\\)) to its two-dimensional counterpart (\\(x\\)). \"Other\" properties here include temperature and pressure which are not dependent on the thermodynamic phase.
Most studies in this topic refer a technical report by Tiedtke et al. (1979)1. Another more recent general reference that may be of interest is that of R\u00e4is\u00e4nen et al. (2004)2.
M. Tiedtke, J.-F. Geleyn, A. Hollingsworth, and J.-F. Louis. ECMWF model parameterisation of sub-grid scale processes. Technical Report, ECMWF, Shinfield Park, Reading, January 1979. 10.\u00a0\u21a9
Petri R\u00e4is\u00e4nen, Howard W Barker, Marat F Khairoutdinov, Jiangnan Li, and David A Randall. Stochastic generation of subgrid-scale cloudy columns for large-scale models. Quarterly Journal of the Royal Meteorological Society: A journal of the atmospheric sciences, applied meteorology and physical oceanography, 130(601):2047\u20132067, 2004.\u00a0\u21a9
This section contains documentation on how to create, setup, and run CIME cases with EAMxx as the atmosphere component. It is assumed that the reader has a familiarity with CIME case control system. In particular, we assume that the user knows how to create a case, and what the case.setup, case.build, and case.submit commands do.
This user guide is still under construction. In the meantime, in case you can't find the information you need, you may visit our public confluence EAMxx user guide.
"},{"location":"user/clean_clear_sky/","title":"Clean- and clean-clear-sky diagnostics","text":"In order to decompose the aerosol effective radiative forcing, additional diagnostic radiation calls are needed. These extra diagnostics are optionally added to the main radiation call. The extra diagnostics are:
The extra calls are controlled by runtime flags extra_clnclrsky_diag and extra_clnsky_diag (they take either true or false as their values).
./atmchange extra_clnclrsky_diag=true\n ./atmchange extra_clnsky_diag=true\nBelow is an example output file to output the extra (clean and clean-clear-sky) radiation diagnostics atop the atmosphere.
%YAML 1.1\n---\nfilename_prefix: monthly.outputs\nAveraging Type: Average\nMax Snapshots Per File: 1\nFields:\n Physics PG2:\n Field Names:\n - SW_clnclrsky_flux_up_at_model_top\n - LW_clnclrsky_flux_up_at_model_top\n - SW_clnsky_flux_up_at_model_top\n - LW_clnsky_flux_up_at_model_top\noutput_control:\n Frequency: 1\n frequency_units: nmonths\n MPI Ranks in Filename: false\nCOSP is partially implemented and supported in EAMxx. Currently, minimal outputs from the ISCCP, MODIS, and MISR simulators have been enabled.
"},{"location":"user/cosp/#running-with-cosp","title":"Running with COSP","text":"Turning COSP on simply requires adding the cosp process to atm_procs_list via atmchange in a case directory:
./atmchange physics::atm_procs_list=\"mac_aero_mic,rrtmgp,cosp\"\natmchange: ./atmchange physics::cosp::cosp_frequency_units=\"steps\"\n./atmchange physics::cosp::cosp_frequency=1\nCOSP can be run with or without subcolumn sampling. This is configured by changing the cosp_subcolumns namelist variable via atmchange. A value of 1 implies no subcolumn sampling, while values greater than 1 specify the number of subcolumns to use for subcolumn sampling (assuming maximum-random overlap). E.g.,
./atmchange physics::cosp:cosp_subcolumns=1\n./atmchange physics::cosp::cosp_subcolumns=10\nSCOPS/PREC_SCOPS. The default for high resolution cases (e.g., ne1024) should be to not use subcolumns, while lower resolutions (e.g., ne30) should enable subcolumn sampling. Output streams need to be added manually. A minimal example:
./atmchange output_yaml_files=scream_daily_output.yaml\ncat << EOF > scream_cosp_daily_output.yaml\nAveraging Type: Average\nFields:\n Physics PG2:\n Field Names:\n - isccp_cldtot\n - isccp_ctptau\n - modis_ctptau\n - misr_cthtau\n - cosp_sunlit\nMax Snapshots Per File: 1\nfilename_prefix: eamxx\noutput_control:\n Frequency: 1\n frequency_units: ndays\nEOF\nThe following output fields are available:
ISCCP, MODIS, and MISR outputs are valid only for daytime/sunlit columns (to be consistent with available satellite retrievals). In order to aggregate only daytime columns in time averages, these outputs are multiplied by the sunlit flag (0 or 1) at each COSP calculation time. Time averages of these quantities are then aggregated, along with the cosp sunlit flag each time COSP is called. In order to back out the daytime-only time averages from the outputs, one needs to divide the output fields by cosp_sunlit. E.g.,
isccp_ctptau = mean(isccp_ctptau) / mean(cosp_sunlit)\nTo run the DP configuration of EAMxx (DP-EAMxx) please refer to the official DP resource page. At this location you will find full documentation of case descriptions and access to run scripts. Using these scripts, you should be able to get DP-EAMxx up and running in a matter of minutes on any machine that EAMxx currently runs on (CPU or GPU).
"},{"location":"user/eamxx_cases/","title":"Basics of EAMxx cases","text":"This section explains how to create a case which uses EAMxx as the atmosphere model, as well as what are the currently supported compsets and grids for EAMxx.
"},{"location":"user/model_input/","title":"Model inputs","text":"This section explains how input parameters are passed to EAMxx, and how the user can change their value. The full list of the currently configuraable runtime parameters for EAMxx can be found here.
The infrastructure for reading inputs into EAMxx involves a few scripts/files:
atmchange and atmquery: these scripts are located in SRCDIR/components/eamxx/scripts, and are soft-linked in the case folder. As their names suggest, they can be used to query and change the runtime configuration parameter of EAMxx. Since these two scripts are the only scripts that the average user needs to know and interact with, in the next sections we give a brief overview of how they can be used and how their output can be interpreted. Additionally, for both of the scripts a short help can also be obtained using the -h flag.buildnml: this script, located in SRCDIR/components/eamxx/cime_config, is called by CIME's case management scripts (case.setup, case.build, and case.submit), and is responsible for creating the input files that EAMxx will read to load its runtime parameters. Users should not have to modify this script, nor should they have to manually call it, but it is useful to know what it does. When buildnml runs, it creates a few files, containing EAMxx input parameters:
scream_input.yaml: this YAML file is located in the RUNDIR/data folder, and will be read by EAMxx at runtime to load all of its configuration parameters. More precisely, this file contains parameters that need to be used inside the EAMxx interfaces.namelist.nl: this namelist file is located in the RUNDIR/data folder, and will be parsed at runtime to get all the parameters for the HOMME dycore (ADD REF). This file only contains dycore-specific parameters that are only recognized inside HOMME, and does not contain any parameter pertaining EAMxx infrastructure.namelist_scream.xml: this XML file is located in the case directory, and contains all the runtime parameters that EAMxx will read in at runtime. buildnml uses this XML file as an intermediate file during the generation of scream_input.yaml and namelist.nl. More specifically, buildnml generates this file using case information to select the proper configurations from the file namelist_defaults_scream.xml, located in SRCDIR/components/eamxx/cime_config. Despite the fact that the only files that are needed at runtime are scream_input.yaml and namelist.nl, we generate and keep this XML file around to make the implementation of atmquery easier.Since these files are automatically generated when buildnml runs, users should not manually modify them. Any manual modification will be lost the next time buildnml runs (e.g., at case.submit time).
This script is the simplest way for the user to check the value and properties of EAMxx input parameters. A basic usage of the script is
$ ./atmquery my_param\nwhich will retrieve the value of the parameter called my_param, by locating the XML node \"my_param\" in the file namelist_scream.xml in the RUNDIR folder. Obviously, an XML file can have multiple nodes with the same tag, and the script is implemented to error out if multiple matches are found. In such a scenario, the user needs to provide also the parents nodes names, using enough parents to uniquely identify the node (in most cases, one parent is enough). To specify a parent, the user can prepend the parent name and :: to the node name:
$ ./atmquery foo::my_param\nThe output will contain the fully scoped parameter name, along with the value. E.g.,
$ ./atmquery foo::my_param\n namelist_defaults::node1::node2::foo::my_param: 10\nIt is sometimes desirable to query all the nodes that have a particular name, or that contain a particular string. We can do that by using the --grep flag:
$ ./atmquery --grep sub\n iop_options::iop_dosubsidence: false\n ctl_nl::hypervis_subcycle: 1\n ctl_nl::hypervis_subcycle_tom: 1\n ctl_nl::hypervis_subcycle_q: 6\n atmosphere_processes::number_of_subcycles: 1\n sc_import::number_of_subcycles: 1\n homme::number_of_subcycles: 1\n physics::number_of_subcycles: 1\nTODO: This difference between basic and --grep is not really intuitive: as pointed out in this issue, we should change this. If we do, don't forget to update this following part of the docs. Using the --grep option has another effect: if the match is not a leaf of the XML tree, all its subelements are printed:
$ ./atmquery --grep homme\n homme\n Moisture: moist\n BfbHash: 18\n number_of_subcycles: 1\n enable_precondition_checks: true\n enable_postcondition_checks: true\n repair_log_level: trace\n internal_diagnostics_level: 0\n compute_tendencies: None\nSimilarly to the CIME utility xmlchange, the options --value, --type, --valid-values, and --full can be used to respectively retrieve just the parameter value (useful for shell scripting), the parameter's type, a list of valid values for parameter (when applicable), or all of the above:
$ ./atmquery atm_log_level --value\n info\n$ ./atmquery atm_log_level --type\n namelist_defaults::driver_options::atm_log_level: string\n$ ./atmquery atm_log_level --valid-values\n namelist_defaults::driver_options::atm_log_level: ['trace', 'debug', 'info', 'warn', 'error']\n$ ./atmquery atm_log_level --full\n namelist_defaults::driver_options::atm_log_level\n value: info\n type: string\n valid values: ['trace', 'debug', 'info', 'warn', 'error']\nFinally, the option --listall can be used to list the whole content of the XML file, which will be displayed with each node indented in its parent scope:
$ ./atmquery --listall\n namelist_defaults\n grids_manager\n Type: Homme\n physics_grid_type: PG2\n physics_grid_rebalance: None\n dynamics_namelist_file_name: ./data/namelist.nl\n vertical_coordinate_filename: /some/path/to/coords/file.nc\n initial_conditions\n Filename: /some/path/to/ic/file.nc\n topography_filename: /some/path/to/topo/file.nc\n ...\nWhen buildnml runs, the model inputs are deduced from the case configuration settings (e.g., the grid, the compset, etc.) and the namelist_scream_defaults.xml file, located in the eamxx source tree. The user can change any of these parameters using the atmchange script. A basic usage of the script is
$ ./atmchange my_param=10\nAs for atmquery, if there are multiple matches for a given parameter name, the user must specify a unique scoped name, which allows atmchange to uniquely identify the XML node to modify:
$ ./atmquery homme::number_of_subcycles\n namelist_defaults::atmosphere_processes::homme::number_of_subcycles: 1\n$ ./atmchange number_of_subcycles=10\nERROR: internal_diagnostics_level is ambiguous. Use ANY in the node path to allow multiple matches. Matches:\n namelist_defaults::atmosphere_processes::number_of_subcycles\n namelist_defaults::atmosphere_processes::sc_import::number_of_subcycles\n namelist_defaults::atmosphere_processes::homme::number_of_subcycles\n namelist_defaults::atmosphere_processes::physics::number_of_subcycles\n namelist_defaults::atmosphere_processes::physics::mac_aero_mic::number_of_subcycles\n namelist_defaults::atmosphere_processes::physics::mac_aero_mic::tms::number_of_subcycles\n namelist_defaults::atmosphere_processes::physics::mac_aero_mic::shoc::number_of_subcycles\n namelist_defaults::atmosphere_processes::physics::mac_aero_mic::cldFraction::number_of_subcycles\n namelist_defaults::atmosphere_processes::physics::mac_aero_mic::spa::internal_diagnostics_level\n namelist_defaults::atmosphere_processes::physics::mac_aero_mic::p3::number_of_subcycles\n namelist_defaults::atmosphere_processes::physics::rrtmgp::number_of_subcycles\n namelist_defaults::atmosphere_processes::sc_export::number_of_subcycles\n$ ./atmchange homme::number_of_subcycles=10\nRegenerating /path/to/namelist_scream.xml. Manual edits will be lost.\n$ ./atmquery homme::number_of_subcycles\n namelist_defaults::atmosphere_processes::homme::number_of_subcycles: 10\nIn some cases, the user may be interested in changing all nodes with a given name. In that case, you can use 'ANY' as a node name:
$ ./atmquery --grep number_of_subcycles\n atmosphere_processes::number_of_subcycles: 1\n sc_import::number_of_subcycles: 1\n homme::number_of_subcycles: 1\n physics::number_of_subcycles: 1\n mac_aero_mic::number_of_subcycles: 24\n tms::number_of_subcycles: 1\n shoc::number_of_subcycles: 1\n cldFraction::number_of_subcycles: 1\n spa::number_of_subcycles: 1\n p3::number_of_subcycles: 1\n rrtmgp::number_of_subcycles: 1\n sc_export::number_of_subcycles: 1\n$ ./atmchange ANY::number_of_subcycles=3\nRegenerating /path/to/namelist_scream.xml. Manual edits will be lost.\n$ ./atmquery --grep number_of_subcycles\n atmosphere_processes::number_of_subcycles: 3\n sc_import::number_of_subcycles: 3\n homme::number_of_subcycles: 3\n physics::number_of_subcycles: 3\n mac_aero_mic::number_of_subcycles: 3\n tms::number_of_subcycles: 3\n shoc::number_of_subcycles: 3\n cldFraction::number_of_subcycles: 3\n spa::number_of_subcycles: 3\n p3::number_of_subcycles: 3\n rrtmgp::number_of_subcycles: 3\n sc_export::number_of_subcycles: 3\n$ ./atmchange mac_aero_mic::ANY::number_of_subcycles=1\nRegenerating /path/to/namelist_scream.xml. Manual edits will be lost.\n$ ./atmquery --grep number_of_subcycles\n atmosphere_processes::number_of_subcycles: 3\n sc_import::number_of_subcycles: 3\n homme::number_of_subcycles: 3\n physics::number_of_subcycles: 3\n mac_aero_mic::number_of_subcycles: 1\n tms::number_of_subcycles: 1\n shoc::number_of_subcycles: 1\n cldFraction::number_of_subcycles: 1\n spa::number_of_subcycles: 1\n p3::number_of_subcycles: 1\n rrtmgp::number_of_subcycles: 3\n sc_export::number_of_subcycles: 3\nSince the XML file stores constraints on the parameter value (like its type or valid values), attempting to use the wrong type will cause an error:
$ ./atmquery --type se_ne\n namelist_defaults::ctl_nl::se_ne: integer\n$ ./atmchange se_ne=hello\nERROR: Could not refine 'hello' as type 'integer':\nThere are three main types supported: integer, float, string, logical. When passing a string to atmchange, the script will try to interpret it acoording to the parameter type, and throw an error if that's not possible: for \"string\", anything works; for \"integer\", only digits are allowed, possibly with a negative sign in front; for \"float\", only digits are allowed, possibly with a negative sign in front and a decimal point; for \"logical\", only the strings \"true\" and \"false\" are allowed (case insensitive). There are two additional types supported: \"file\" and \"array(T)\", where \"T\" is any of the other supported types (but not another array): - \"file\" is used to inform CIME of the input files that have to be download from E3SM data servers, like initial conditions files, or certain lookup tables. - \"array(T)\" allows to specify a list of items (of the same type), which will be parsed inside EAMxx as a std::vector<T>.
For type \"string\" and \"array(T)\", it is also possible to append to the currently stored value
$ ./atmquery homme::compute_tendencies\n namelist_defaults::atmosphere_processes::homme::compute_tendencies:\n value: a, b\n type: array(string)\n valid values: []\n$ ./atmchange homme::compute_tendencies+=c\n$ ./atmquery homme::compute_tendencies --full\n namelist_defaults::atmosphere_processes::homme::compute_tendencies\n value: a, b, c\n type: array(string)\n valid values: []\nThe atmchange script can be used to change any of the runtime parameters of EAMxx. In particular, it can be used to add, remove, or reorder atmosphere processes. When adding an atmosphere process, we must first make sure that the defaults for that process are present in namelist_defaults_scream.xml. For instance, the default settings for the \"physics\" atmosphere process group include the following:
$ ./atmquery physics::atm_procs_list\n namelist_defaults::atmosphere_processes::physics::atm_procs_list: mac_aero_mic,rrtmgp\nwhere \"mac_aero_mic\" is itself an atmosphere process group, consisting of macrophysics, aerosols, and microphysics processes. If one wanted to add the \"cosp\" atmosphere process to this list, and change the number of its subcycles, it could do so via
$ ./atmchange physics::atm_procs_list+=cosp\n$ ./atmchange cosp::number_of_subcycles=3\nNotice that if we swapped the two commands, we would get an error, since the node \"cosp\" is not present in the XML generated from the defaults until we decide to add it.
It is also possible to declare a new (empty) atmosphere process group, which can then be filled with valid atmosphere processes via subsequent calls to atmchange. The syntax to trigger this behavior consists in specifying a process name that begins and ends with an underscore:
$ ./atmchange physics::atm_procs_list+=_my_group_\nThis adds a new process to the list of processes in \"physics\", called \"_my_group_\", and which is itself an atmosphere process group. Hence, we can then do
$ ./atmchange _my_group_::atm_procs_list+=A,B\nwhere A and B must be valid atmosphere process names (i.e., present in namelist_defaults_scream.xml) or be themselves new atmosphere process groups (i.e., beginning/ending with an underscore)
atmchange can also be used to completely change a list of atmosphere processes:
$ ./atmchange physics::atm_procs_list=A,B,C\nNotice that we used \"=\" instead of \"+=\", which means we will be overwriting the value, rather than appending. Any atmosphere process that was previously in the list but is no longer in it will be removed from the generated namelist_defaults.xml (and scream_input.yaml) files, along with all their nested parameters.
EAMxx allows the user to configure the desired model output via YAML files, with each YAML file associated to a different output file. In order to add an output stream, one needs to run atmchange output_yaml_files+=/path/to/my/output/yaml (more information on how to use atmchange can be found here). During the buildnml phase of the case management system, a copy of these YAML files will be copied into the RUNDIR/data folder. During this process, the files will be parsed, and any CIME-related variable will be resolved accordingly. Therefore, it is not advised to put the original YAML files in RUNDIR/data, since upon buildnml execution, all the CIME vars will no longer be available in the YAML file, making it harder to tweak it, and even harder to share with other users/cases. Another consequence of this is that the user should not modify the YAML files in RUNDIR/data, since any modification will be lost on the next run of buildnml.
The following is a basic example of an output request.
%YAML 1.1\n---\nfilename_prefix: my_output\nAveraging Type: Average\nMax Snapshots Per File: 10\nFields:\n Physics:\n Field Names:\n - T_mid\n - qv\n Dynamics:\n Field Names:\n - dp3d_dyn\n - omega_dyn\noutput_control:\n Frequency: 6\n frequency_units: nhours\nNotice that lists can be equivalently specified in YAML as Field Names: [f1, f2, f3]. The user can specify fields to be outputted from any of the grids used in the simulation. In the example above, we requested fields from both the Physics and Dynamics grid. The meaning of the other parameters is as follows:
Averaging Type: how the fields are integrated in time before being saved. Valid options are
output_control section. In the case above, each snapshot saved to file corresponds to an average of the output fields over 6h windows.filename_prefix: the prefix of the output file, which will be created in the run directory. The full filename will be $prefix.$avgtype.$frequnits_x$freq.$timestamp.nc, where $timestamp corresponds to the first snapshot saved in the file for Instant output, or the beginning of the first averaging window for the other averaging types
Max Snapshots Per File: specifies how many time snapshots can be put in a file. Once this number is reached, EAMxx will close the file and open a new one.Frequency: how many units of time are between two consecutive writes to file. For Instant output the fields are \"sampled\" at this frequency, while for other averaging types the fields are \"integrated\" in time over this windowfrequency_units: units of the output frequency. Valid options are nsteps (the number of atmosphere time steps), nsecs, nmins, nhours, ndays, nmonths, nyears.In addition to the fields computed by EAMxx as part of the timestep, the user can request to output derived quantities, which will be computed on the fly by the I/O interface of EAMxx. There are two types of diagnostic outputs:
quantities computed as a function of EAMxx fields. These are simply physical quantities that EAMxx does not keep in persistent storage. As of May 2024, the available derived quantities are (case sensitive):
PotentialTemperatureAtmosphereDensityExnerVirtualTemperaturez_intz_midgeopotential_intgeopotential_middzDryStaticEnergySeaLevelPressureLiqWaterPathIceWaterPathVapWaterPathRainWaterPathRimeWaterPathShortwaveCloudForcingLongwaveCloudForcingRelativeHumidityZonalVapFluxMeridionalVapFluxprecip_liq_surf_mass_fluxprecip_ice_surf_mass_fluxprecip_total_surf_mass_fluxsurface_upward_latent_heat_fluxwind_speedAerosolOpticalDepth550nmNumberPathAeroComCldTODO: add some information about what each diagnostic is, perhaps a formula
lower-dimensional slices of a field. These are hyperslices of an existing field or of another diagnostic output. As of August 2023, given a field X, the available options are:
X_at_lev_N: slice the field X at the N-th vertical level index. Recall that in EAMxx N=0 corresponds to the model top.X_at_model_bot, X_at_model_top: special case for top and bottom of the model.X_at_Ymb, X_at_YPa, X_at_YhPa: interpolates the field X at a vertical position specified by the give pressure Y. Available units are mb (millibar), Pa, and hPa.X_at_Ym_above_Z: interpolates the field X at a vertical height of Y meters above Z, with Z=surface or Z=sealevel.The following options can be used to to save fields on a different grid from the one they are computed on.
horiz_remap_file: a path to a map file (as produced by ncremap) between the grid where the fields are defined and a coarser grid. EAMxx will use this to remap fields on the fly, allowing to reduce the size of the output file. Note: with this feature, the user can only specify fields from a single grid.vertical_remap_file: similar to the previous option, this map file is used to refine/coarsen fields in the vertical direction.IOGrid: this parameter can be specified inside one of the grids sections, and will denote the grid (which must exist in the simulation) where the fields must be remapped before being saved to file. This feature is really only used to save fields on the dynamics grid without saving twice the DOFs at the interface of two spectral elements. E.g., for a scalar quantity defined only in the horizontal direction, native output from the Dynamics grid would produce arrays of length nelems*ngp*ngp, where ngp is the number of Gauss points along each axis in the 2d spectral element, and nelems is the number of horizontal elements. However, due to continuity, the values on the Gauss points on the element boundary must match the values on the neighboring element, resulting in duplicated data. By remapping to a \"unique\" version of the dynamics grid (which in EAMxx is referred to as \"Physics GLL\"), we can save roughly 45% of storage. Note: this feature cannot be used along with the horizontal/vertical remap.It is also possible to request tendencies of fields that are updated by atmosphere processes, on a per-process basis (here, \"updated\" means that the field is both an input as well as an output of the atmosphere process). Since the term \"tendency\" can be used with slightly different connotations, we clarify what we mean by that when it comes to model output: if process P updates field A, by the tendency of A from process P we mean (A_after_P - A_before_P) / dt, where dt is the atmosphere timestep.
As of May 2024, the user needs two things in order to get tendencies from a process. E.g., to get the tendencies of T_mid and horiz_winds from the process shoc, one needs:
atmchange shoc::compute_tendencies=T_mid,horiz_winds;shoc_T_mid_tend and shoc_horiz_winds_tend to the list of fields in the desired output YAML file.The YAML file shown at the top of this section, together with the remap options in the following section, covers most of the options used in a typical run. There are however particular use cases that require some less common options, which we list here (in parentheses, the location in the YAML file and the type of the parameter value).
flush_frequency (toplevel list, integer): this parameter can be used to specify how often the IO library should sync the in-memory data to file. If not specified, the IO library is free to decide when it should flush the data. This option can be helpful for debugging, in case a crash is occurring after a certain number of steps, but before the IO library would automatically flush to file.Floating Point Precision (toplevel list, string): this parameter specifies the precision to be used for floating point variables in the output file. By default, EAMxx uses single precision. Valid values are single, float, double, and real. The first two are synonyms, while the latter resolves to single or double depending on EAMxx cmake configuration parameter SCREAM_DOUBLE_PRECISION.file_max_storage_type (toplevel list, string): this parameter determines how the capacity of the file is specified. By default, it is set to num_snapshots, which makes EAMxx read Max Snapshots Per File (explained in the first section). However, the user can specify one_year or one_month, which will make EAMxx create one output file per year/month of simulation, fitting however many snapshots are needed in each file (depending on the output frequency). If one_year or one_month are used, the option Max Snapshots Per File is ignored.MPI Ranks in Filename (toplevel list, boolean): this option specifies whether the number of MPI ranks in the atm communicator should be put inside the output file name. By default, this is false, since it is usually not important. This option is mostly important for standalone unit testing, where several versions of the same test (corresponding to different numbers of MPI ranks) are running concurrently, so that different file names are needed to avoid resource contention.save_grid_data (output_control sublist, boolean): this option allows to specify whether grid data (such as lat/lon) should be added to the output stream. By default, it is true.iotype (toplevel list, string): this option allows the user to request a particular format for the output file. The possible values are default, netcdf, pnetcdf,adios,hdf5, wheredefault` means \"whatever is the PIO type from the case settings\".skip_t0_output (output_control sublist, boolean): this option is relevant only for Instant output, where fields are also outputed at the case start time (i.e., after initialization but before the beginning of the first timestep). By default it is set to false.output_yaml_files atm option (which can be queried via atmquery output_yaml_files). The user can specify a few options, in order to tweak the restart behavior:Perform Restart (Restart sublist, boolean): this parameter is true by default, but can be set to false to force the model to ignore any history restart files, and start the output stream from scratch, as if this was an initial run.filename_prefix (Restart sublist, string): by default, this parameter is set to match the value of filename_prefix from the toplevel list. It can be set to something else in case we want to restart a previous simulation that was using a different filename prefix.force_new_file (Restart sublist, boolean): this parameter allows to start a fresh new output file upon restarts. By default, is is set to false, so that EAMxx will attempt to resume filling the last produced output file (if any, and if it can accommodate more snapshots).Nudging is supported in EAMxx. Currently, it is possible to nudge EAMxx to the output from a different EAMxx run or to reanalysis. Nudging data can be on your model grid or an arbitrary coarser grid. Inline interpolating of finer-grid nudging data to a coarser model resolution isn't implemented yet but may be in the future.
"},{"location":"user/nudging/#data-to-nudge-towards","title":"Data to nudge towards","text":"The user is expected to prepapre (and then use atmchange to point to) nudging data files that are compliant with EAMxx specification. In practice, this means that the data files must contain variable names known to EAMxx (only U, V, T_mid, and qv are supported now). The files can be specified with an explicit list or via pattern matching. The files must contain an arbitary global attribute case_t0, and it is recommended to be the same as the time dimension unit (the files must be time-dimensioned). Finally, the dimension order must be such that lev is the last dimension, so most likely, the user must transpose the dimensions.
Pressure can be explicitly provided in the nudging data as time-varying p_mid corresponding to the option TIME_DEPENDENT_3D_PROFILE for source_pressure_type. Alternatively, the data can contain a time-invariant pressure variable p_lev corresponding to the option TIME_DEPENDENT_3D_PROFILE for source_pressure_type.
In regionally refined model applications, it is possible to use weighted nudging, for example, to avoid nudging the refined region. To achieve that, the user can use atmchange to set use_nudging_weights (boolean) and provide nudging_weights_file that has the weight to apply for nudging (for example, zeros in the refined region). Currently, weighted nudging is only supported if the user provides the nudging data at the target grid.
To enable nudging as a process, one must declare it in the atm_procs_list runtime parameter.
./atmchange physics::atm_procs_list=\"mac_aero_mic,rrtmgp,cosp,nudging\"\nThe following options are needed to specify the nudging.
./atmchange nudging::nudging_filenames_patterns=\"/pathto/nudging_files/*.nc\" # can provide file name explicitly here instead (or multiple patterns)\n./atmchange nudging::source_pressure_type=TIME_DEPENDENT_3D_PROFILE # see section on pressure\n./atmchange nudging::nudging_fields=U,V # can include qv, T_mid as well\n./atmchange nudging::nudging_timescale=21600 # in seconds\nTo gain a deeper understanding of these parameters and options, please refer to code implementation of the nudging process.
"},{"location":"user/rrm_eamxx/","title":"Running EAMxx with a Regionally Refined Mesh (RRM)","text":"Running EAMxx with a RRM allows you run a select region of the globe at high resolution (i.e. 3 km) with the remainder of the globe at a lower resolution (i.e. 25 or 100 km). This document will point you to the steps required and resources available to assist in developing and running a new RRM.
"},{"location":"user/rrm_eamxx/#choose-your-rrm","title":"Choose Your RRM","text":"What region of the globe do you want to refine? Your first step should be to check library of RRM grids/cases that have already been developed to potentially avoid duplicate work. If you found a RRM that suits your needs, you can skip the next step (\"Generate Your RRM\").
"},{"location":"user/rrm_eamxx/#generate-your-rrm","title":"Generate Your RRM","text":"Please refer to the offical E3SM guide for developing new atmosphere grids, which provides detailed guidance for developing your RRM.
After you have made all the necessary files for your RRM, you will need to configure your code branch so that it knows about your new grid. The steps required to do this are documented at the top of the library of RRM grids/cases page.
"},{"location":"user/rrm_eamxx/#make-your-initial-condition-file","title":"Make Your Initial Condition File","text":"The easiest way to generate an initial condition is to use the HICCUP tool, which is a set of flexible and robust python routines to streamline the task of generating a new atmospheric initial condition for E3SM. Otherwise, please see the step-by-step instuctions if you prefer to manually generate your initial condition.
"},{"location":"user/rrm_eamxx/#assemble-nudging-data-optional","title":"Assemble Nudging Data (Optional)","text":"If you wish to nudge your simulation, assemble your nudging data in the format required by EAMxx. Please refer to the nudging documentation.
In the event that you only want to nudge a portion of your domain, then you will need to generate a nudging weights file. A common use case for this is when you want the high-resolution region to remain free-running (unnudged) while nudging the coarse domain towards reanalysis or model data. Please use this script as an example of how to generate your nudging weights file.
"},{"location":"user/rrm_eamxx/#run-your-rrm","title":"Run your RRM","text":"Congratulations, you are now ready to run your EAMxx RRM. If you are running your RRM in free running mode (not using any nudging) then you simply need to modify an existing EAMxx script and change the resolution to match the one you created for your RRM.
If you are using nudging, then please see this example script of how to run a nudged EAMxx RRM run. This example script uses the California 3-km RRM, which is on the master branch.
"}]} \ No newline at end of file +{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"The E3SM Atmosphere Model in C++ (EAMxx)","text":"EAMxx EAMxx is almost completely different in all ways from the atmosphere model used for E3SM versions 1-3.
EAMxx was built from the ground up using C++ in order to embrace modern software practices and to allow \"performance portability\" across various supercomputers. The latter goal is achieved by using the Kokkos library. EAMxx is a \"clean-start\" model with almost no similarity to the E3SM atmosphere model used in versions 1-3. Currently only the km-scale explicit-convection version called SCREAM (the Simple Cloud-Resolving E3SM Atmosphere Model) is available, but a low-resolution version is in the works.
Like the documentation for other component models, EAMxx documentation is divided into:
Put another way, all information about how to customize runs without changing code is included in the User Guide, general information about software design which is needed for intelligently modifying code goes in the Developer Guide, and details about the specific process implementations in the current model version are included in the Technical Guide.
"},{"location":"common/eamxx_params/","title":"EAMxx runtime configurable parameters","text":""},{"location":"common/eamxx_params/#atmosphere-processes-parameters","title":"Atmosphere Processes Parameters","text":""},{"location":"common/eamxx_params/#sc_import","title":"sc_import","text":"sc_export::prescribed_constants::values:
sc_export::prescribed_from_file::fields:
sc_export::prescribed_from_file::fields_alt_name:
sc_export::number_of_subcycles:
Follow these simple instructions to build and test EAMxx's standalone configuration for yourself. This document makes use of the following paths:
${RUN_ROOT_DIR}: the root directory where EAMxx is built and run${EAMXX_SRC_DIR}: the directory into which you've cloned the scream repoEAMxx's configuration and build system is based on CMake. CMake has been around a while and has gained a lot of traction in recent years, especially in the HPC community. It has good reference documentation, but it can be tricky to use if you've never encountered it. Ask a EAMxx team member for help if you're stuck on something CMake-related.
If you see a CMakeLists.txt files or a file with a .cmake suffix, that's just part of the build system. You might also see files with CTest as part of their name. These files are related to CTest, CMake's testing tool.
First, make sure you're on one of the machines supported by EAMxx, or that you have the following software installed:
First, make sure you've cloned the EAMxx repo (including all submodules) to EAMXX_SRC_DIR using the following command:
git clone --recurse-submodules https://github.com/E3SM-Project/scream\nIf you have already cloned the project and forgot to type --recurse-submodules, you can change to $EAMXX_SRC_DIR and using the following command to initialize, fetch and checkout all submodules:
git submodule update --init --recursive\nIf you're running a branch that's not master, check out this branch with
git checkout <branch>\nChange to your $RUN_ROOT_DIR directory and use CMake to configure your build.
If you're building SCREAM on one of our supported platforms, you can tell CMake to use the appropriate machine file using the -C flag. Machine files are located in $EAMXX_SRC_DIR/components/eamxx/cmake/machine-files. Take a look and see whether your favorite machine has one.
For example, to configure SCREAM on the Quartz machine at LLNL:
cd $RUN_ROOT_DIR\ncmake \\\n -DCMAKE_CXX_COMPILER=$(which mpicxx) \\\n -DCMAKE_BUILD_TYPE=Debug \\\n -C ${EAMXX_SRC_DIR}/components/eamxx/cmake/machine-files/quartz.cmake \\\n ${EAMXX_SRC_DIR}/components/eamxx\nIf you're building on a machine that doesn't have a ready-made machine file, you can try configuring your build by manually passing options to CMake. This usually looks something like the following, which configures EAMxx to compile CPU code using Kokkos's OpenMP backend:
cd $RUN_ROOT_DIR\ncmake \\\n -D CMAKE_BUILD_TYPE=Debug \\\n -D CMAKE_C_COMPILER=mpicc \\\n -D CMAKE_CXX_COMPILER=mpicxx \\\n -D CMAKE_Fortran_COMPILER=mpif90 \\\n -D MPIEXEC_EXECUTABLE=`which mpiexec` \\\n -D EKAT_MPI_NP_FLAG:STRING=-n \\\n -D SCREAM_DYNAMICS_DYCORE=HOMME \\\n -D SCREAM_DOUBLE_PRECISION:BOOL=ON \\\n -D SCREAM_INPUT_ROOT:PATH=/path/to/scream-input \\\n -D Kokkos_ENABLE_DEBUG=TRUE \\\n -D Kokkos_ENABLE_AGGRESSIVE_VECTORIZATION=OFF \\\n -D Kokkos_ENABLE_SERIAL=ON \\\n -D Kokkos_ENABLE_OPENMP=ON \\\n -D Kokkos_ENABLE_LIBDL=OFF \\\n -D Kokkos_ENABLE_PROFILING=OFF \\\n -D Kokkos_ENABLE_DEPRECATED_CODE=OFF \\\n -D KOKKOS_ENABLE_ETI:BOOL=OFF \\\n -D NetCDF_C_PATHS=/path/to/netcdf-c-dir \\\n -D NetCDF_Fortran_PATHS=/path/to/netcdf-f90-dir \\\n -D PnetCDF_C_PATHS=/path/to/pnetcdf-dir \\\n -D PnetCDF_Fortran_PATHS=/path/to/pnetcdf-f90-dir \\\n ${EAMXX_SRC_DIR}/components/eamxx\nIn either case, EAMxx requires MPI-aware compilers. Let's examine these options (only some of which are required on any given machine) to make sure we know what they do:
CMAKE_BUILD_TYPE: specifies whether you are building EAMxx in a developer-friendly configuration (Debug), for a production run (Release) or for performance profiling or some other specialized purpose. Typically, you'll set this option to Debug or Release.CMAKE_{C,CXX,Fortran}_COMPILER: the name of the command used to invoke an MPI-enabled C, C++, or Fortran compiler to build EAMxxMPIEXEC_EXECUTABLE: the name of the command used to run EAMxx using MPI, typically mpiexec or mpirun, but possibly different depending on your desired machineEKAT_MPI_NP_FLAG: the flag passed to MPIEXEC_EXECUTABLE that you use to specify the number of desired MPI processes. This is typically -n for mpiexec and -np for mpirun.SCREAM_DYNAMICS_DYCORE: specifies the dycore used for configuring EAMxx, which is NONE if you are not configuring EAMxx to run its dycore-related tests, or HOMME if you want to use HOMMExxSCREAM_DOUBLE_PRECISION: indicates whether EAMxx's Real type is a double-precision (ON) or single-precision (OFF) floating point typeSCREAM_INPUT_ROOT: specifies the location of the top-level folder that stores input data files for EAMxx. This folder is populated with input files which are downloaded automatically during EAMxx's build process.Kokkos_) are described in the Kokkos WikiNetCDF_C_PATHS: specifies one or more folders in which the NetCDF C library and headers are installed. In the simplest configuration, the headers should be located in ${NetCDF_C_PATHS}/include and the library should live in ${NetCDF_C_PATHS}/lib.NetCDF_Fortran_PATHS: specifies one or more folders in which the NetCDF Fortran library and modules are installed. Analogous to ${NetCDF_C_PATHS}, .mod files should be in ${NetCDF_Fortran_PATHS}/include, and the library should be installed in ${NetCDF_Fortran_PATHS}/lib.PnetCDF_C_PATHS: specifies one or more folders in which the pNetCDF C library and headers are installed, analogous to NetCDF_C_PATHS.PnetCDF_Fortran_PATHS: specifies one or more folders in which the pNetCDF Fortran library and modules are installed, analogous to NetCDF_Fortran_PATHS.Above, we've configured Debug builds to make it easier to find and fix errors. For performance testing, you should configure a Release build and make use of other options, depending on your architecture.
Now you can build SCREAM from that same directory:
make -j\nThe -j flag tells Make to use threads to compile in parallel. If you like, you can set the number of threads by passing it as an argument to -j (e.g. make -j8).
You can run EAMxx's tests to make sure your build works by following the instructions here.
"},{"location":"developer/","title":"SCREAM Developer Guide","text":""},{"location":"developer/ci_nightly/","title":"Continuous Integration and Nightly Testing","text":""},{"location":"developer/ci_nightly/#autotester","title":"Autotester","text":"EAMxx using github actions and a Sandia product called Autotester 2 to run CI testing on a CPU and GPU machine for every github pull request. By default, we run the e3sm_scream_v1_at suite and the standalone eamxx tests (test-all-scream).
"},{"location":"developer/ci_nightly/#nightly-overview-cdash","title":"Nightly overview, CDash","text":"Our nightly testing is much more extensive than the CI testing. You can see our dashboard here under the section \"E3SM_SCREAM\": https://my.cdash.org/index.php?project=E3SM
We run a variety of CIME test suites and standalone testing on a number of platforms. We even do some performance testing on frontier.
"},{"location":"developer/cime_testing/","title":"Full Model Testing","text":"Full model system testing of eamxx is done through CIME test cases (much like the rest of E3SM).
We offer a number of test suites, including:
Example for running a suite:
cd $repo/cime/scripts\n./create_test e3sm_scream_v1_at --wait\nExample for running a single test case:
cd $repo/cime/scripts\n./create_test SMS.ne4_ne4.F2010-SCREAMv1 --wait\nThere are many behavioral tweaks you can make to a test case, like changing the run length, test type, etc. Most of this is not specific to eamxx and works for any CIME case. This generic stuff is well-documentated here: http://esmci.github.io/cime/versions/master/html/users_guide/testing.html
When it comes to things specific to eamxx, you have grids, compsets, and testmods.
Common EAMxx grids are:
More grid info can be found here: https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/933986549/ATM+Grid+Resolution+Summary
Common EAMxx compsets are:
Full info on supported compsets can be found by looking at this file: $scream_repo/components/eamxx/cime_config/config_compsets.xml
Common EAMxx testmods are:
More info on running EAMxx can be found here: https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/3386015745/How+To+Run+EAMxx+SCREAMv1
"},{"location":"developer/field/","title":"Field","text":"In EAMxx, a Field is a data structure holding two things: pointers to the data and pointers to metadata. Both the data and metadata are stored in std::shared_ptr instances, to ensure consistency across all copies of the field. This allows for fast shallow copy semantic for this class.
The data is stored on both CPU and device memory (these may be the same, depending on the Kokkos backend). In EAMxx, we always assume and guarantee that the device data is up to date. That implies that the data must be explicitly synced to host before using it on host, and explicitly synced to device after host manipulation, in order to ensure correctness. In order to access the data, users must use the get_view/ get_strided_view methods, which takes two template arguments: the data type, and an enum specifying whether CPU or device data is needed. The data type is used to reinterpret the generic pointer stored inside to a view of the correct scalar type and layout. It is a possibly const-qualified type, and if the field was marked as \"read-only\", the method ensures that the provided data type is const. A read-only field can be created via the getConst method, which returns a shallow copy of the field, but marked as read-only. The enum specifying host or device data is optional, with device being the default.
The metadata is a collection of information on the field, such as name, layout, units, allocation size, and more. Part of the metadata is immutable after creation (e.g., name, units, or layout), while some metadata can be partially or completely modified. The metadata is contained in the FieldHeader data structure, which contains four parts:
FieldIdentifier: stores the field's name, layout, units, data type, and name of the grid where it's defined. These information are condensed in a single string, that can be used to uniquely identify a field, allowing to distinguish between different version of the same field. The layout is stored in the FieldLayout data structure, which includes:std::vector<FieldTag>, they give context to the field's extents.std::vector<int>, as well as a 1d Kokkos::View.FieldTracking: stores information on the usage of the field, as well as its possible connections to other fields. In particular, the tracked items are:FieldAllocProp: stores information about the allocation. While the field is not yet allocated, users can request special allocations for the field, for instance to accommodate packing (for SIMD), which may require padding. Upon allocation, this information is then used by the Field structure to extract the actual data, wrapped in a properly shaped Kokkos::View. The alloc props are also responsible of tracking additional information in case the field is a \"slice\" of a higher-dimensional one, a fact that can affect how the data is accessed.std::map<std::string,ekat::any>, allows to catch any metadata that does not fit in the above structures. This is a last resort structure, intended to accommodate the most peculiar corner cases, and should be used sparingly.In EAMxx, the AbstractGrid is an interface used to access information regarding the horizontal and vertical discretization. The most important information that the grid stores is:
PointGrid (a class derived from AbstractGrid) is a simple collection of points, so the \"native\" indexing system coincides with the LIDs. However, for a SEGrid (a derived class, for spectral element grids), the \"native\" indexing is a triplet (ielem,igp,jgp), specifying the element index, and the two indices of the Gauss point within the element.std::map<std::string,Field>, this represent any data that is intrinsically linked to the grid (either along the horizontal or vertical direction), such as lat/lon coordinates, vertical coordinates, area associated with the DOF.Grids can also be used to retrieve the layout of a 2d/3d scalar/vector field, which allows certain downstream classes to perform certain operations without assuming anything on the horizontal grid.
In general, grid objects are passed around the different parts of EAMxx as const objects (read-only). The internal data can only be modified during construction, which usually is handled by a GridsManager object.
In EAMxx, I/O is handled through the SCORPIO library, currently a submodule of E3SM. The scream_io library within eamxx allows to interface the EAMxx infrastructure classes with the SCORPIO library.
EAMxx uses Kokkos for performance portable abstractions for parallel execution of code and data management to various HPC platforms, including OpenMP, Cuda, HIP, and SYCL. Here we give a brief overview of the important concepts for understanding Kokkos in EAMxx. For a more in depth description, see the Kokkos wiki.
"},{"location":"developer/kokkos_ekat/#kokkosdevice","title":"Kokkos::Device","text":"Kokkos::Device is a struct which contain the type definitions for two main Kokkos concepts: execution space (Kokkos::Device::execution_space), the place on-node where parallel operations (like for-loops, reductions, etc.) are executed, and the memory space (Kokkos::Device::memory_space), the memory location on-node where data is stored. Given your machine architecture, Kokkos defines a default \"device\" space, given by
Kokkos::Device<Kokkos::DefaultExecutionSpace,\n Kokkos::DefaultExecutionSpace::memory_space>\nwhere all performance critical code should be executed (e.g., on an NVIDIA machine, this device would be the GPU accelerators) and a default \"host\" space, given by
Kokkos::Device<Kokkos::DefaultHostExecutionSpace,\n Kokkos::DefaultHostExecutionSpace::memory_space>\nwhere data can be accessed by the CPU cores and is necessary for I/O interfacing, for example. Currently, these default spaces are the ones used by EAMxx. On CPU-only machines, host and device represent the same space.
"},{"location":"developer/kokkos_ekat/#kokkos-views","title":"Kokkos Views","text":"The main data struct provided by Kokkos used in EAMxx in the Kokkos::View. This is a multi-dimensional data array that can live on either device or host memory space. These Views are necessary when running on GPU architectures as data structures like std::vector and std::array will be unavailable on device.
Views are constructed in EAMxx most commonly with the following template and input arguments
Kokkos::View<DataType, LayoutType, DeviceType>(const std::string& label,\n int dim0, int dim1, ...)\nwhere
DataType: scalar type of the view, given as ScalarType+*(x's number of run-time dimensions). E.g., a 2D view of doubles will have DataType = double**. There is also an ability to define compile-time dimensions by using [], see [Kokkos wiki section on views] (https://kokkos.org/kokkos-core-wiki/API/core/view/view.html).LayoutType: mapping of indices into the underlying 1D memory storage. Types are:LayoutRight (used in EAMxx): strides increase from the right most to the left most dimension, right-most dimension is contiguousLayoutLeft: strides increase from the left most to the right most dimension, left-most dimension is contiguousLayoutStride: strides can be arbitrary for each dimensionDeviceType: provides space where data live, defaults to the default deviceThe following example defines a view \"temperature\" which has dimensions columns and levels:
Kokkos::View<double**, Kokkos::LayoutRight, Kokkos::DefaultDevice> temperature(\n \"temperature\", ncols, nlevs);\nKokkos provides Kokkos::deep_copy(dst, src) which copies data between views of the same dimensions, or a scalar values into a view. Common uses
Kokkos::deep_copy(view0, view1); // Copy all data from view1 into view0\nKokkos::deep_copy(view0, 5); // Set all values of view0 to 5\nAs seen in the next section, we can use deep_copy() to copy data between host and device.
We will often need to have memory allocation the resides on device (for computation), and then need that identical data on host (say, for output). Kokkos has a concept of mirror views, where data can be copied from host to device and vice versa.
Here is an example using the device view temperature from above
// Allocate view on host that exactly mirrors the size of layout of the device\nview\nauto host_temperature = Kokkos::create_mirror_view(temperature);\n\n// Copy all data from device to host\nKokkos::deep_copy(host_temperature, temperature);\nKokkos also offers an all-in-one option
// Note: must hand the host device instance as first argument\nauto host_temperature = Kokkos::create_mirror_view_and_copy(\n Kokkos::DefaultHostDevice(), temperature);\nThe most basic parallel execution pattern used by EAMxx is the Kokkos::parallel_for which defines a for-loop with completely independent iterations. The parallel_for takes in an optional label for debugging, an execution policy, which defines a range and location (host or device) for the code to be run, and a lambda describing the body of code to be executed. The following are execution policies used in EAMxx
int count: 1D iteration range [0, count)RangePolicy<ExecSpace>(int beg, int end): 1D iteration range for indices [beg, end)MDRangePolicy<ExecSpace, Kokkos::Rank<N>>(int[N] beg, int[N] end): multi- dimensional iteration range [beg, end)TeamPolicy<ExecSpace>(int league_size, int team_size, int vector_size): 1D iteration over league_size, assigned to thread teams of size team_size, each with vector_size vector lanes. Both team_size and vector_size can be given Kokkos::AUTO as input for Kokkos to automatically compute.If no ExecSpace template is given, the default execution space is used.
For lambda capture, use KOKKOS_LAMBDA macro which sets capture automatically based on architecture.
Example using RangePolicy to initialize a view
Kokkos::View<double**, Kokkos::LayoutRight> temperature(\"temperature\", ncols,\n nlevs);\nKokkos::parallel_for(\"Init_temp\",\n Kokkos::RangePolicy(0, ncols*nlevs),\n KOKKOS_LAMBDA (const int idx) {\n int icol = idx/nlevs;\n int ilev = idx%nlevs;\n\n temperature(icol, ilev) = 0;\n});\nSame example with TeamPolicy
Kokkos::parallel_for(\"Init_temp\",\n Kokkos::TeamPolicy(ncols*nlevs, Kokkos::AUTO, Kokkos::AUTO),\n KOKKOS_LAMBDA (const TeamPolicy::member_type& team) {\n // league_rank() gives the index for this team\n int icol = team.league_rank()/nlevs;\n int ilev = team.league_rank()%nlevs;\n\n temperature(icol, ilev) = 0;\n});\nUsing TeamPolicy, we can have up to three nested levels of parallelism: team parallelism, thread parallelism, vector parallelism. These nested policies can be called within the lambda body using the following execution policies
TeamThreadRange(team, begin, end): execute over threads of a teamTeamVectorRange(team, begin, end): execute over threads and vector lanes of a teamThreadVectorRange(team, begin, end): execute over vector lanes of a threadAn example of using these policies
Kokkos::View<double***> Q(\"tracers\", ncols, ntracers, nlevs);\nKokkos::parallel_for(Kokkos::TeamPolicy(ncols, Kokkos::AUTO),\n KOKKOS_LAMBDA (TeamPolicy::member_type& team) {\n int icol = team.league_rank();\n Kokkos::parallel_for(Kokkos::TeamVectorRange(team, nlevs), [&](int ilev) {\n temperature(icol, ilev) = 0;\n });\n\n Kokkos::parallel_for(Kokkos::TeamThreadRange(team, nlevs), [&](int ilev) {\n Kokkos::parallel_for(Kokkos::ThreadVectorRange(team, ntracers), [&](int iq) {\n Q(icol, iq, ilev) = 0;\n });\n });\n});\nIMPORTANT! Nested policies cannot be used in arbitrary order. ThreadVectorRange must be used inside a TeamThreadRange, and TeamVectorRange must be the only level of nested parallelism.
Kokkos::parallel_for(TeamPolicy(...), ... {\n // OK\n Kokkos::parallel_for(TeamThreadRange, ... {\n\n });\n\n // OK\n Kokkos::parallel_for(TeamThreadRange, ... {\n Kokkos::parallel_for(ThreadVectorRange, ... {\n\n });\n });\n\n // OK\n Kokkos::parallel_for(TeamVectorRange, ...{\n\n });\n\n // WRONG,ThreadVectorRange must be nested in TeamThreadRange\n Kokkos::parallel_for(ThreadVectorRange, ...{\n\n });\n\n // WRONG, a TeamVectorRange must be the only nested level\n Kokkos::parallel_for(TeamVectorRange, ...{\n Kokkos::parallel_for(ThreadVectorRange, ... {\n\n });\n });\n});\nUsing these incorrectly can be very tricky to debug as the code almost certainly will not error out, but race conditions will exist among threads.
"},{"location":"developer/kokkos_ekat/#ekat","title":"EKAT","text":""},{"location":"developer/kokkos_ekat/#kokkostypes","title":"KokkosTypes","text":""},{"location":"developer/kokkos_ekat/#exespaceutils","title":"ExeSpaceUtils","text":""},{"location":"developer/kokkos_ekat/#vectorization-packs","title":"Vectorization: Packs","text":""},{"location":"developer/kokkos_ekat/#scratch-memory-worspacemanager","title":"Scratch Memory: WorspaceManager","text":""},{"location":"developer/kokkos_ekat/#algorithms","title":"Algorithms","text":""},{"location":"developer/managers/","title":"FieldManager and GridsManager","text":""},{"location":"developer/processes/","title":"Atmospheric Processes","text":"In EAMxx, AtmosphereProcess (AP) is an abstract class representing a portion of the atmosphere timestep algorithm. In simple terms, an AP is an object that given certain input fields performs some calculations to compute some output fields. The concrete AP classes allow to create a buffer layer between particular packages (e.g., dynamics dycore, physics parametrizations) and the atmosphere driver (AD), allowing separation of concerns, so that the AD does not need to know details about the package, and the package does not need to know about the EAMxx infrastructure.
To enhance this separation of concerns, EAMxx implements two more classes for handling APs:
AtmosphereProcessGroup (APG), which allows to group together a set of AP's, which can be seen from outside as a single process;AtmosphereProcessFactory class, which allows an APG to create its internal processes without any knowledge of what they are.This infrastructure allows the AD to view the whole atmosphere as a single APG, and to be completely agnostic to what processes are run, and in which order. This design allows to have a code base that is cleaner, self-container, and easy to test via a battery of targeted unit tests.
In EAMxx, we already have a few concrete AP's, interfacing the AD to the Hommexx non-hydrostatic dycore as well as some physics parametrizations (P3, SHOC, RRMTPG, etc). In the next section we describe the interfaces of an AP class, and we show an example of how to write a new concrete AP class.
"},{"location":"developer/processes/#atmosphere-process-interfaces","title":"Atmosphere process interfaces","text":"An AP has several interfaces, which can be grouped into three categories:
Among the above, the initialization sequence is the most complex, and consists of several steps:
GridsManager to the AP's, so that they can get information about the grids they need. At this point, all AP's have all the information they need to establish the layout of the input and output fields they need, and can store a list of these \"requests\"While the base AP class provides an (empty) implementation for some methods, in case derived classes do not need a feature, some methods are purely virtual, and concrete classes will have to override them. Looking at existing concrete AP implementations is a good way to have a first idea of what a new AP class needs to implement. Here, we show go over the possible implementation of these methods in a hypothetical AP class. The header file may look something like this
#include <share/atm_process/atmosphere_process.hpp>\n\nclass MyProcess : public AtmosphereProcess\n{\npublic:\n using gm_ptr = std::shared_ptr<const GridsManager>;\n\n MyProcess(const ekat::Comm& comm, const ekat::ParameterList& pl);\n\n std::string name () const override { return \"my_fancy_process\"; }\n void set_grids (const gm_ptr& grids_manager) override;\n size_t requested_buffer_size_in_bytes () const override;\n void init_buffers (const ATMBufferManager& buffer_manager) override;\nprotected:\n\n void initialize_impl (const RunType run_type) override;\n void run_impl (const double dt) override;\n void finalize_impl () override;\n\n using view_1d = typename KokkosTypes<DefaultDevice>::view_1d<Real>;\n using view_2d = typename KokkosTypes<DefaultDevice>::view_2d<Real>;\n\n view_1d m_temp1;\n view_2d m_temp2;\n\n int m_ncols;\n int m_nlevs;\n bool m_has_blah;\n};\nA few comments:
override when overriding a method; in case of small typos (e.g., missing a & or a const, the compiler will be erroring out, since the signature will not match any virtual method in the base class;finalize_impl is often empty; unless the AP is managing external resources, everything should be correctly released during destruction;Here is a possible implementation of the methods, with some inline comments to explain
MyProcess::MyProcess (const ekat::Comm& comm, const ekat::ParameterList& pl)\n : AtmosphereProcess(comm,pl)\n{\n // The base class copies pl into protected member m_params\n m_has_blah = m_params.get<bool>(\"enable_blah\");\n}\n\nvoid MyProcess::set_grids (const std::shared_ptr<GridsManager>& gm)\n{\n using namespace ekat::units;\n const auto nondim = Units::nondimensional();\n\n auto grid = gm->get_grid(\"Physics\");\n m_ncols = grid->get_num_local_dofs();\n m_nlevs = grid->get_num_vertical_levels();\n\n // In these names, 2d refers to \"horizontal only\", while 3d to \"horiz+vert\".\n // But the grid stores dofs linearly, so there is only one array dimension\n FieldLayout scalar2d = grid->get_2d_scalar_layout();\n FieldLayout vector3d = grid->get_3d_vector_layout(true,2);\n\n // Declare fields needed:\n // - Required: 'input' (read-only)\n // - Computed: 'output'\n // - Updated: 'input'+'output'\n // Tell the AD we need 'velocity' to accommodate a Pack scalar type\n add_field<Required>(\"coeff_2d\",scalar2d,nondim,grid->name);\n add_field<Updated>(\"velocity\",vector3d,m/s,grid->name,SCREAM_PACK_SIZE);\n}\n\nsize_t MyProcess::requested_buffer_size_in_bytes ()\n{\n // We use temp2 only if the blah feature is on\n return m_ncols + (m_has_blah ? m_ncols*m_nlev : 0);\n}\n\nvoid MyProcess::init_buffers (const ATMBufferManager& buffer_manager)\n{\n auto mem = reinterpret_cast<Real*>(buffer_manager.get_memory());\n\n m_temp1 = view_1d<Real>(mem,m_ncols);\n mem += m_ncols;\n\n if (m_has_blah) {\n m_temp2 = view_2d<Real>(mem,m_ncols,m_nlevs);\n mem += m_ncols*m_nlevs;\n }\n\n // Make sure we use exactly the mem we said we would\n size_t used_mem = (mem - buffer_manager.get_memory())*sizeof(Real);\n EKAT_REQUIRE_MSG(used_mem==requested_buffer_size_in_bytes(),\n \"Error! Used memory != requested memory for MyProcess.\"\n \" used memory: \" + std::to_string(used_mem) + \"\\n\"\n \" requested: \" + std::to_string(requested_buffer_size_in_bytes()) + \"\\n\");\n}\n\nvoid MyProcess::initialize_impl(const RunType run_type)\n{\n // Can complete any initialization of the background pkg\n my_process_pkg_init(m_has_blah);\n}\n\nvoid MyProcess:run_impl (const double dt)\n{\n using Policy = typename KokkosTypes<DefaultDevice>::TeamPolicy\n using Member = typename KokkosTypes<DefaultDevice>::MemberType\n using PackT = ekat::Pack<Real,SCREAM_PACK_SIZE>;\n\n // Create team policy\n Policy policy(m_ncols,m_nlevs,1);\n\n // Create local copies of class members (or else use KOKKOS_CLASS_LAMBDA)\n auto temp1 = m_temp1;\n auto temp2 = m_temp2;\n auto do_blah = m_has_blah;\n\n // Get views from fields. We \n auto coeff2d = get_field_in(\"coeff_2d\").get_view<const Real*>();\n auto velocity = get_field_out(\"velocity\").get_view<PackT**>();\n\n // Since we process velocity with a Pack scalar type, find out how many packs\n // we have in each column\n auto nlevs_packs = ekat::PackInfo<SCREAM_PACK_SIZE>::num_packs(m_nlevs);\n\n // Call some function in the background pkg\n do_some_work (coeff_2d,velocity,temp1,temp2,do_blah);\n\n // Do some more work here\n auto work = KOKKOS_LAMBDA (const Member& team) {\n int icol = team.league_rank();\n auto col_work = [&](const int ilev) {\n velocity(icol,ilev) *= coeff_2d;\n };\n Kokkos::parallel_for(Kokkos::TeamVectorRange(team,nlevs_packs),col_work);\n };\n Kokkos::parallel_for(policy,work);\n Kokkos::fence();\n}\n\nvoid MyProcess::finalize_impl ()\n{\n // If the background package needs to cleanup something, do it now\n my_process_pkg_cleanup();\n}\nAll EAMxx-specific code can be found in components/eamxx within the EAMxx repo. Here's how things are organized:
cime_config: Tools and XML files for integrating EAMxx with E3SM via the CIME framework.cmake: CMake functions and macros used by the configuration/build system.data: Data files used by our tests.docs: Documentation for the EAMxx project, including design documents, instructions for building and testing EAMxx, and this document.scripts: Miscellaneous scripts that implement workflows for running tests and analyzing performance.src: All C++ source code (and any bridges to Fortran) for EAMxx are stored here. We describe the contents of this directory in greater detail below.tests: Implements standalone, end-to-end tests for various EAMxx components (RRTMG, HOMME, P3, SHOC, etc).In addition, you'll notice the following files in components/eamxx:
CMakeLists.txt: The CMake file that defines EAMxx's configuration/build system.CTestConfig.cmake: This CTest file contains parameters that determine how our test results are reported to the E3SM CDash Site.README.md: EAMxx's top-level README file, which describes the project and its purpose.mkdocs.yml: The configuration file for mkdocs, the tool we currently use to build and publish our documentation.src Directory","text":"Herein l\u0456es the source code for EAMxx. Broadly, here's where things are:
control: Contains the atmosphere driver and basic tests for it.dynamics: Here's where HOMME lives within EAMxx, along with code for interfacing with it using EAMxx's data structures.mct_coupling: Glue code for embedding EAMxx within E3SM as an atmosphere component using the MCT coupler.physics: Source code for physics-related atmospheric processes, includingp3: The C++/Kokkos implementation of P3 microphysics within EAMxx.shoc: The C++/Kokkos implementation of SHOC macrophysics within EAMxx.rrtmgp: A stub for the radiation processes as represented in EAMxx.share: Utilities and data structures common to these processes.share: Utilities used by various components within EAMxx. Of note:io: EAMxx's interface to the SCORPIO library.diagnostics: A collection of simple classes used to compute diagnostic quantities.Each of these directories contains a CMakeLists.txt file for defining how things are build, and a tests/ subdirectory that houses relevant unit and verification tests.
You'll also see some other files in the src/ directory itself, such as
scream_config.h.in: A template for generating a C++ header file with EAMxx configuration information.In this section we describe our testing methodology for standalone EAMxx configurations. We use several types of tests
We also support a test-all-scream configuration that runs all of the standalone tests for an EAMxx configuration. Note, your current machine must be known to EAMxx before test-all-scream will work. A machine can be made known to EAMxx by editing the eamxx/scripts/machines_specs.py files. There are some instructions on what to do at the top of this file.
test-all-scream has a good help dump
cd $scream_repo/components/eamxx\n./scripts/test-all-scream -h\nIf you are unsure of the cmake configuration for you development cycle, one trick you can use is to run test-all-scream for the dbg test and just copy the cmake command it prints (then ctrl-C the process).
cd $scream_repo/components/eamxx\n./scripts/test-all-scream -t dbg -m $machine\n# wait for a few seconds*\n# Ctrl-C *\n# Copy the contents of DCMAKE_COMMAND that was passed to ctest *\n# Add \"cmake\" to beginning of contents and path to eamxx at the end. *\nConsiderations for using test-all-scream:
cd eamxx/scripts; eval $(./scripts/scream-env-cmd $machine)Before running the tests, generate a baseline file:
cd $RUN_ROOT_DIR\nmake baseline\nThe tests will run, automatically using the baseline file, which is located in the CMake-configurable path ${SCREAM_BASELINES_DIR}. By default, this path is set to an invalid string. If baselines tests are enabled, we check that a valid path has been provided.
To run all of SCREAM's tests, make sure you're in $RUN_ROOT_DIR and type
ctest -VV\nThis runs everything and reports results in an extra-verbose (-VV) manner.
You can also run subsets of the SCREAM tests. For example, to run only the P3 regression tests (again, from the $RUN_ROOT_DIR directory), use
ctest -R p3_regression\nWe can create groupings of tests by using labels. For example, we have a driver label that runs tests for SCREAM's standalone driver. You can see a list of available labels by typing
ctest --print-labels\nTo see which tests are associated with a given label (e.g. driver), use
ctest -L driver -N\np3_regression Suite","text":"p3_regression uses a baseline file to compare any new or altered implementations with our P3 Fortran reference implementation. If you're working on the C++/Kokkos implementation, you can invoke any new tests to the function Baseline::run_and_cmp in ${SCREAM_SRC_DIR}/components/eamxx/p3/tests/p3_run_and_cmp.cpp.
If the reference Fortran implementation changes enough that a new baseline file is required, make sure to let other SCREAM team members know, in order to minimize disruptions.
"},{"location":"developer/style_guide/","title":"SCREAM C++ Style Guide","text":"Here's our style guide. Let the holy wars begin!
"},{"location":"developer/style_guide/#types","title":"Types","text":""},{"location":"developer/style_guide/#functions-and-methods","title":"Functions and Methods","text":""},{"location":"developer/style_guide/#variables","title":"Variables","text":""},{"location":"technical/","title":"EAMxx Technical Guide","text":"The goal of this document is to describe the specific equations, parameterizations, and numerical methods used in the current version of EAMxx. Because our master-branch implementation changes every time we make a new commit, this documentation will also evolve continuously. As such, documentation for master should always be considered to be preliminary and under construction. If you want trustworthy documentation, pull it from an official model release.
"},{"location":"technical/#overview","title":"Overview","text":"Currently, EAMxx is only configured for km-scale convection-permitting runs. In order to provide scientifically-credible simulations at lower resolution, parameterizations for the following processes would be needed:
The only configuration of EAMxx that is currently implemented is the convection-permitting version, commonly known as the Simple Cloud-Resolving E3SM Atmosphere Model (SCREAM). Processes in EAMxx-SCREAM are:
By default processes are called in this order, but which processes to include and in what order is modifiable at run time. After all atmospheric processes are called, output is written. Surface components are then called before the next atmosphere step starts. These processes are described in more detail in Caldwell et al. (2021)6. As in EAM, dynamics operates on a spectral element grid and all other processes use a finite-volume grid that divides each spectral element into 4 quadrilaterals. This physics grid is described in Hannah et al. (2021)8.
Mark A. Taylor, Oksana Guba, Andrew Steyer, Paul A. Ullrich, David M. Hall, and Christopher Eldred. An energy consistent discretization of the nonhydrostatic equations in primitive variables. Journal of Advances in Modeling Earth Systems, 12(1):e2019MS001783, 2020. e2019MS001783 10.1029/2019MS001783. URL: https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2019MS001783, arXiv:https://agupubs.onlinelibrary.wiley.com/doi/pdf/10.1029/2019MS001783, doi:https://doi.org/10.1029/2019MS001783.\u00a0\u21a9
A. M. Bradley, P. A. Bosler, and O. Guba. Islet: interpolation semi-lagrangian element-based transport. Geoscientific Model Development, 15(16):6285\u20136310, 2022. URL: https://gmd.copernicus.org/articles/15/6285/2022/, doi:10.5194/gmd-15-6285-2022.\u00a0\u21a9
F. Fiedler and H. A. Panofsky. The geostrophic drag coefficient and the \u2018effective\u2019 roughness length. Quarterly Journal of the Royal Meteorological Society, 98(415):213\u2013220, 1972. URL: https://rmets.onlinelibrary.wiley.com/doi/abs/10.1002/qj.49709841519, arXiv:https://rmets.onlinelibrary.wiley.com/doi/pdf/10.1002/qj.49709841519, doi:https://doi.org/10.1002/qj.49709841519.\u00a0\u21a9
B. Stevens, S. Fiedler, S. Kinne, K. Peters, S. Rast, J. M\u00fcsse, S. J. Smith, and T. Mauritsen. Macv2-sp: a parameterization of anthropogenic aerosol optical properties and an associated twomey effect for use in cmip6. Geoscientific Model Development, 10(1):433\u2013452, 2017. URL: https://gmd.copernicus.org/articles/10/433/2017/, doi:10.5194/gmd-10-433-2017.\u00a0\u21a9
Hugh Morrison and Jason A. Milbrandt. Parameterization of cloud microphysics based on the prediction of bulk ice particle properties. part i: scheme description and idealized tests. Journal of the Atmospheric Sciences, 72(1):287 \u2013 311, 2015. URL: https://journals.ametsoc.org/view/journals/atsc/72/1/jas-d-14-0065.1.xml, doi:10.1175/JAS-D-14-0065.1.\u00a0\u21a9
P. M. Caldwell, C. R. Terai, B. Hillman, N. D. Keen, P. Bogenschutz, W. Lin, H. Beydoun, M. Taylor, L. Bertagna, A. M. Bradley, T. C. Clevenger, A. S. Donahue, C. Eldred, J. Foucar, J.-C. Golaz, O. Guba, R. Jacob, J. Johnson, J. Krishna, W. Liu, K. Pressel, A. G. Salinger, B. Singh, A. Steyer, P. Ullrich, D. Wu, X. Yuan, J. Shpund, H.-Y. Ma, and C. S. Zender. Convection-permitting simulations with the e3sm global atmosphere model. Journal of Advances in Modeling Earth Systems, 13(11):e2021MS002544, 2021. e2021MS002544 2021MS002544. URL: https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2021MS002544, arXiv:https://agupubs.onlinelibrary.wiley.com/doi/pdf/10.1029/2021MS002544, doi:https://doi.org/10.1029/2021MS002544.\u00a0\u21a9\u21a9
Robert Pincus, Eli J. Mlawer, and Jennifer S. Delamere. Balancing accuracy, efficiency, and flexibility in radiation calculations for dynamical models. Journal of Advances in Modeling Earth Systems, 11(10):3074\u20133089, 2019. URL: https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2019MS001621, arXiv:https://agupubs.onlinelibrary.wiley.com/doi/pdf/10.1029/2019MS001621, doi:https://doi.org/10.1029/2019MS001621.\u00a0\u21a9
Walter M. Hannah, Andrew M. Bradley, Oksana Guba, Qi Tang, Jean-Christophe Golaz, and Jon Wolfe. Separating physics and dynamics grids for improved computational efficiency in spectral element earth system models. Journal of Advances in Modeling Earth Systems, 13(7):e2020MS002419, 2021. e2020MS002419 2020MS002419. URL: https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2020MS002419, arXiv:https://agupubs.onlinelibrary.wiley.com/doi/pdf/10.1029/2020MS002419, doi:https://doi.org/10.1029/2020MS002419.\u00a0\u21a9
The goal of the AeroCom algorithm is to calculate properties at cloud top based on the AeroCom recommendation. There are two main parts of the algorithm: probabilistically determining \"cloud top\" and then \"calculating properties\" at said cloud top.
We treat model columns independently, so we loop over all columns in parallel. We then loop over all layers in serial (due to needing an accumulative product), starting at 2 (second highest) layer because the highest is assumed to have no clouds. Let's take a photonic approach from above the model top. Let's say that \\(p_{k}\\) is the probability of a photon passing through the layer \\(k\\). We follow the maximum-random overlap assumption. In all cases, we assume the cloudiness (or cloudy fraction) is completely opaque.
We assume the highest layer has no clouds, thus the \\(p_{k} = 1\\) for the highest layer. Note that \\(p_{k}\\) is initialized as 1 for all layers. We also clip the cloudy fraction \\(C_{i,k}\\) to ensure that \\(C_{i,k} \\in [0+\\epsilon, 1-\\epsilon]\\), where \\(\\epsilon = 0.001\\). Starting at the second highest layer, \\(k+1\\), we check if some \"cloudy\" conditions are met. These conditions are now arbitrarily defined by a cloudiness threshold of \\(\\epsilon\\) (i.e., \\(C_{i,k}>\\epsilon\\)) and a non-zero threshold on the total (both liquid and ice) droplet number concentration (i.e., \\(cQ_{i,k} + iQ_{i,k} > 0\\)). If the conditions are met, we estimate the cloud-top cloud fraction using an accumulative product following the maximum-random overlap assumption.
\\[c_{i} = 1 - \\prod_{k=2}^{K} p_{k} = 1 - \\prod_{k=2}^{K} \\frac{1 - \\max(C_{i,k}, C_{i,k-1})}{1-C_{i,k-1}}\\]In order to estimate cloud-top properties, we weight by the probability of \"remaining cloudiness\" or \\(p_{k-1} - p_{k}\\).
Type Equation cloud property \\(x_{i} = \\sum_{k=2}^{K} X_{i,k} \\Phi_{i,k} (p_{k-1} - p_{k})\\) cloud content \\(x_{i} = \\sum_{k=2}^{K} \\Phi_{i,k} (p_{k-1} - p_{k})\\) other property \\(x_{i} = \\sum_{k=2}^{K} X_{i,k} (p_{k-1} - p_{k})\\)In the above, \\(\\Phi_{i,k}\\) is the thermodynamic phase defined by the cloud droplet number concentration ratios.
\\[i\\Phi_{i,k} = \\frac{iQ_{i,k}}{iQ_{i,k} + cQ_{i,k}}\\] \\[c\\Phi_{i,k} = \\frac{cQ_{i,k}}{iQ_{i,k} + cQ_{i,k}}\\]The thermodynamic phase is used only for cloud properties (e.g., cloud-top cloud droplet number concentration) or cloud content (e.g., cloud liquid content). Further, \\(X_{i,k}\\) is the three-dimensional cloud property of interest which is needed if we are converting a property from three-dimensional (\\(X\\)) to its two-dimensional counterpart (\\(x\\)). \"Other\" properties here include temperature and pressure which are not dependent on the thermodynamic phase.
Most studies in this topic refer a technical report by Tiedtke et al. (1979)1. Another more recent general reference that may be of interest is that of R\u00e4is\u00e4nen et al. (2004)2.
M. Tiedtke, J.-F. Geleyn, A. Hollingsworth, and J.-F. Louis. ECMWF model parameterisation of sub-grid scale processes. Technical Report, ECMWF, Shinfield Park, Reading, January 1979. 10.\u00a0\u21a9
Petri R\u00e4is\u00e4nen, Howard W Barker, Marat F Khairoutdinov, Jiangnan Li, and David A Randall. Stochastic generation of subgrid-scale cloudy columns for large-scale models. Quarterly Journal of the Royal Meteorological Society: A journal of the atmospheric sciences, applied meteorology and physical oceanography, 130(601):2047\u20132067, 2004.\u00a0\u21a9
This section contains documentation on how to create, setup, and run CIME cases with EAMxx as the atmosphere component. It is assumed that the reader has a familiarity with CIME case control system. In particular, we assume that the user knows how to create a case, and what the case.setup, case.build, and case.submit commands do.
This user guide is still under construction. In the meantime, in case you can't find the information you need, you may visit our public confluence EAMxx user guide.
"},{"location":"user/clean_clear_sky/","title":"Clean- and clean-clear-sky diagnostics","text":"In order to decompose the aerosol effective radiative forcing, additional diagnostic radiation calls are needed. These extra diagnostics are optionally added to the main radiation call. The extra diagnostics are:
The extra calls are controlled by runtime flags extra_clnclrsky_diag and extra_clnsky_diag (they take either true or false as their values).
./atmchange extra_clnclrsky_diag=true\n ./atmchange extra_clnsky_diag=true\nBelow is an example output file to output the extra (clean and clean-clear-sky) radiation diagnostics atop the atmosphere.
%YAML 1.1\n---\nfilename_prefix: monthly.outputs\nAveraging Type: Average\nMax Snapshots Per File: 1\nFields:\n Physics PG2:\n Field Names:\n - SW_clnclrsky_flux_up_at_model_top\n - LW_clnclrsky_flux_up_at_model_top\n - SW_clnsky_flux_up_at_model_top\n - LW_clnsky_flux_up_at_model_top\noutput_control:\n Frequency: 1\n frequency_units: nmonths\n MPI Ranks in Filename: false\nCOSP is partially implemented and supported in EAMxx. Currently, minimal outputs from the ISCCP, MODIS, and MISR simulators have been enabled.
"},{"location":"user/cosp/#running-with-cosp","title":"Running with COSP","text":"Turning COSP on simply requires adding the cosp process to atm_procs_list via atmchange in a case directory:
./atmchange physics::atm_procs_list=\"mac_aero_mic,rrtmgp,cosp\"\natmchange: ./atmchange physics::cosp::cosp_frequency_units=\"steps\"\n./atmchange physics::cosp::cosp_frequency=1\nCOSP can be run with or without subcolumn sampling. This is configured by changing the cosp_subcolumns namelist variable via atmchange. A value of 1 implies no subcolumn sampling, while values greater than 1 specify the number of subcolumns to use for subcolumn sampling (assuming maximum-random overlap). E.g.,
./atmchange physics::cosp:cosp_subcolumns=1\n./atmchange physics::cosp::cosp_subcolumns=10\nSCOPS/PREC_SCOPS. The default for high resolution cases (e.g., ne1024) should be to not use subcolumns, while lower resolutions (e.g., ne30) should enable subcolumn sampling. Output streams need to be added manually. A minimal example:
./atmchange output_yaml_files=scream_daily_output.yaml\ncat << EOF > scream_cosp_daily_output.yaml\nAveraging Type: Average\nFields:\n Physics PG2:\n Field Names:\n - isccp_cldtot\n - isccp_ctptau\n - modis_ctptau\n - misr_cthtau\n - cosp_sunlit\nMax Snapshots Per File: 1\nfilename_prefix: eamxx\noutput_control:\n Frequency: 1\n frequency_units: ndays\nEOF\nThe following output fields are available:
ISCCP, MODIS, and MISR outputs are valid only for daytime/sunlit columns (to be consistent with available satellite retrievals). In order to aggregate only daytime columns in time averages, these outputs are multiplied by the sunlit flag (0 or 1) at each COSP calculation time. Time averages of these quantities are then aggregated, along with the cosp sunlit flag each time COSP is called. In order to back out the daytime-only time averages from the outputs, one needs to divide the output fields by cosp_sunlit. E.g.,
isccp_ctptau = mean(isccp_ctptau) / mean(cosp_sunlit)\nTo run the DP configuration of EAMxx (DP-EAMxx) please refer to the official DP resource page. At this location you will find full documentation of case descriptions and access to run scripts. Using these scripts, you should be able to get DP-EAMxx up and running in a matter of minutes on any machine that EAMxx currently runs on (CPU or GPU).
"},{"location":"user/eamxx_cases/","title":"Basics of EAMxx cases","text":"This section explains how to create a case which uses EAMxx as the atmosphere model, as well as what are the currently supported compsets and grids for EAMxx.
"},{"location":"user/model_input/","title":"Model inputs","text":"This section explains how input parameters are passed to EAMxx, and how the user can change their value. The full list of the currently configuraable runtime parameters for EAMxx can be found here.
The infrastructure for reading inputs into EAMxx involves a few scripts/files:
atmchange and atmquery: these scripts are located in SRCDIR/components/eamxx/scripts, and are soft-linked in the case folder. As their names suggest, they can be used to query and change the runtime configuration parameter of EAMxx. Since these two scripts are the only scripts that the average user needs to know and interact with, in the next sections we give a brief overview of how they can be used and how their output can be interpreted. Additionally, for both of the scripts a short help can also be obtained using the -h flag.buildnml: this script, located in SRCDIR/components/eamxx/cime_config, is called by CIME's case management scripts (case.setup, case.build, and case.submit), and is responsible for creating the input files that EAMxx will read to load its runtime parameters. Users should not have to modify this script, nor should they have to manually call it, but it is useful to know what it does. When buildnml runs, it creates a few files, containing EAMxx input parameters:
scream_input.yaml: this YAML file is located in the RUNDIR/data folder, and will be read by EAMxx at runtime to load all of its configuration parameters. More precisely, this file contains parameters that need to be used inside the EAMxx interfaces.namelist.nl: this namelist file is located in the RUNDIR/data folder, and will be parsed at runtime to get all the parameters for the HOMME dycore (ADD REF). This file only contains dycore-specific parameters that are only recognized inside HOMME, and does not contain any parameter pertaining EAMxx infrastructure.namelist_scream.xml: this XML file is located in the case directory, and contains all the runtime parameters that EAMxx will read in at runtime. buildnml uses this XML file as an intermediate file during the generation of scream_input.yaml and namelist.nl. More specifically, buildnml generates this file using case information to select the proper configurations from the file namelist_defaults_scream.xml, located in SRCDIR/components/eamxx/cime_config. Despite the fact that the only files that are needed at runtime are scream_input.yaml and namelist.nl, we generate and keep this XML file around to make the implementation of atmquery easier.Since these files are automatically generated when buildnml runs, users should not manually modify them. Any manual modification will be lost the next time buildnml runs (e.g., at case.submit time).
This script is the simplest way for the user to check the value and properties of EAMxx input parameters. A basic usage of the script is
$ ./atmquery my_param\nwhich will retrieve the value of the parameter called my_param, by locating the XML node \"my_param\" in the file namelist_scream.xml in the RUNDIR folder. Obviously, an XML file can have multiple nodes with the same tag, and the script is implemented to error out if multiple matches are found. In such a scenario, the user needs to provide also the parents nodes names, using enough parents to uniquely identify the node (in most cases, one parent is enough). To specify a parent, the user can prepend the parent name and :: to the node name:
$ ./atmquery foo::my_param\nThe output will contain the fully scoped parameter name, along with the value. E.g.,
$ ./atmquery foo::my_param\n namelist_defaults::node1::node2::foo::my_param: 10\nIt is sometimes desirable to query all the nodes that have a particular name, or that contain a particular string. We can do that by using the --grep flag:
$ ./atmquery --grep sub\n iop_options::iop_dosubsidence: false\n ctl_nl::hypervis_subcycle: 1\n ctl_nl::hypervis_subcycle_tom: 1\n ctl_nl::hypervis_subcycle_q: 6\n atmosphere_processes::number_of_subcycles: 1\n sc_import::number_of_subcycles: 1\n homme::number_of_subcycles: 1\n physics::number_of_subcycles: 1\nTODO: This difference between basic and --grep is not really intuitive: as pointed out in this issue, we should change this. If we do, don't forget to update this following part of the docs. Using the --grep option has another effect: if the match is not a leaf of the XML tree, all its subelements are printed:
$ ./atmquery --grep homme\n homme\n Moisture: moist\n BfbHash: 18\n number_of_subcycles: 1\n enable_precondition_checks: true\n enable_postcondition_checks: true\n repair_log_level: trace\n internal_diagnostics_level: 0\n compute_tendencies: None\nSimilarly to the CIME utility xmlchange, the options --value, --type, --valid-values, and --full can be used to respectively retrieve just the parameter value (useful for shell scripting), the parameter's type, a list of valid values for parameter (when applicable), or all of the above:
$ ./atmquery atm_log_level --value\n info\n$ ./atmquery atm_log_level --type\n namelist_defaults::driver_options::atm_log_level: string\n$ ./atmquery atm_log_level --valid-values\n namelist_defaults::driver_options::atm_log_level: ['trace', 'debug', 'info', 'warn', 'error']\n$ ./atmquery atm_log_level --full\n namelist_defaults::driver_options::atm_log_level\n value: info\n type: string\n valid values: ['trace', 'debug', 'info', 'warn', 'error']\nFinally, the option --listall can be used to list the whole content of the XML file, which will be displayed with each node indented in its parent scope:
$ ./atmquery --listall\n namelist_defaults\n grids_manager\n Type: Homme\n physics_grid_type: PG2\n physics_grid_rebalance: None\n dynamics_namelist_file_name: ./data/namelist.nl\n vertical_coordinate_filename: /some/path/to/coords/file.nc\n initial_conditions\n Filename: /some/path/to/ic/file.nc\n topography_filename: /some/path/to/topo/file.nc\n ...\nWhen buildnml runs, the model inputs are deduced from the case configuration settings (e.g., the grid, the compset, etc.) and the namelist_scream_defaults.xml file, located in the eamxx source tree. The user can change any of these parameters using the atmchange script. A basic usage of the script is
$ ./atmchange my_param=10\nAs for atmquery, if there are multiple matches for a given parameter name, the user must specify a unique scoped name, which allows atmchange to uniquely identify the XML node to modify:
$ ./atmquery homme::number_of_subcycles\n namelist_defaults::atmosphere_processes::homme::number_of_subcycles: 1\n$ ./atmchange number_of_subcycles=10\nERROR: internal_diagnostics_level is ambiguous. Use ANY in the node path to allow multiple matches. Matches:\n namelist_defaults::atmosphere_processes::number_of_subcycles\n namelist_defaults::atmosphere_processes::sc_import::number_of_subcycles\n namelist_defaults::atmosphere_processes::homme::number_of_subcycles\n namelist_defaults::atmosphere_processes::physics::number_of_subcycles\n namelist_defaults::atmosphere_processes::physics::mac_aero_mic::number_of_subcycles\n namelist_defaults::atmosphere_processes::physics::mac_aero_mic::tms::number_of_subcycles\n namelist_defaults::atmosphere_processes::physics::mac_aero_mic::shoc::number_of_subcycles\n namelist_defaults::atmosphere_processes::physics::mac_aero_mic::cldFraction::number_of_subcycles\n namelist_defaults::atmosphere_processes::physics::mac_aero_mic::spa::internal_diagnostics_level\n namelist_defaults::atmosphere_processes::physics::mac_aero_mic::p3::number_of_subcycles\n namelist_defaults::atmosphere_processes::physics::rrtmgp::number_of_subcycles\n namelist_defaults::atmosphere_processes::sc_export::number_of_subcycles\n$ ./atmchange homme::number_of_subcycles=10\nRegenerating /path/to/namelist_scream.xml. Manual edits will be lost.\n$ ./atmquery homme::number_of_subcycles\n namelist_defaults::atmosphere_processes::homme::number_of_subcycles: 10\nIn some cases, the user may be interested in changing all nodes with a given name. In that case, you can use 'ANY' as a node name:
$ ./atmquery --grep number_of_subcycles\n atmosphere_processes::number_of_subcycles: 1\n sc_import::number_of_subcycles: 1\n homme::number_of_subcycles: 1\n physics::number_of_subcycles: 1\n mac_aero_mic::number_of_subcycles: 24\n tms::number_of_subcycles: 1\n shoc::number_of_subcycles: 1\n cldFraction::number_of_subcycles: 1\n spa::number_of_subcycles: 1\n p3::number_of_subcycles: 1\n rrtmgp::number_of_subcycles: 1\n sc_export::number_of_subcycles: 1\n$ ./atmchange ANY::number_of_subcycles=3\nRegenerating /path/to/namelist_scream.xml. Manual edits will be lost.\n$ ./atmquery --grep number_of_subcycles\n atmosphere_processes::number_of_subcycles: 3\n sc_import::number_of_subcycles: 3\n homme::number_of_subcycles: 3\n physics::number_of_subcycles: 3\n mac_aero_mic::number_of_subcycles: 3\n tms::number_of_subcycles: 3\n shoc::number_of_subcycles: 3\n cldFraction::number_of_subcycles: 3\n spa::number_of_subcycles: 3\n p3::number_of_subcycles: 3\n rrtmgp::number_of_subcycles: 3\n sc_export::number_of_subcycles: 3\n$ ./atmchange mac_aero_mic::ANY::number_of_subcycles=1\nRegenerating /path/to/namelist_scream.xml. Manual edits will be lost.\n$ ./atmquery --grep number_of_subcycles\n atmosphere_processes::number_of_subcycles: 3\n sc_import::number_of_subcycles: 3\n homme::number_of_subcycles: 3\n physics::number_of_subcycles: 3\n mac_aero_mic::number_of_subcycles: 1\n tms::number_of_subcycles: 1\n shoc::number_of_subcycles: 1\n cldFraction::number_of_subcycles: 1\n spa::number_of_subcycles: 1\n p3::number_of_subcycles: 1\n rrtmgp::number_of_subcycles: 3\n sc_export::number_of_subcycles: 3\nSince the XML file stores constraints on the parameter value (like its type or valid values), attempting to use the wrong type will cause an error:
$ ./atmquery --type se_ne\n namelist_defaults::ctl_nl::se_ne: integer\n$ ./atmchange se_ne=hello\nERROR: Could not refine 'hello' as type 'integer':\nThere are three main types supported: integer, float, string, logical. When passing a string to atmchange, the script will try to interpret it acoording to the parameter type, and throw an error if that's not possible: for \"string\", anything works; for \"integer\", only digits are allowed, possibly with a negative sign in front; for \"float\", only digits are allowed, possibly with a negative sign in front and a decimal point; for \"logical\", only the strings \"true\" and \"false\" are allowed (case insensitive). There are two additional types supported: \"file\" and \"array(T)\", where \"T\" is any of the other supported types (but not another array): - \"file\" is used to inform CIME of the input files that have to be download from E3SM data servers, like initial conditions files, or certain lookup tables. - \"array(T)\" allows to specify a list of items (of the same type), which will be parsed inside EAMxx as a std::vector<T>.
For type \"string\" and \"array(T)\", it is also possible to append to the currently stored value
$ ./atmquery homme::compute_tendencies\n namelist_defaults::atmosphere_processes::homme::compute_tendencies:\n value: a, b\n type: array(string)\n valid values: []\n$ ./atmchange homme::compute_tendencies+=c\n$ ./atmquery homme::compute_tendencies --full\n namelist_defaults::atmosphere_processes::homme::compute_tendencies\n value: a, b, c\n type: array(string)\n valid values: []\nThe atmchange script can be used to change any of the runtime parameters of EAMxx. In particular, it can be used to add, remove, or reorder atmosphere processes. When adding an atmosphere process, we must first make sure that the defaults for that process are present in namelist_defaults_scream.xml. For instance, the default settings for the \"physics\" atmosphere process group include the following:
$ ./atmquery physics::atm_procs_list\n namelist_defaults::atmosphere_processes::physics::atm_procs_list: mac_aero_mic,rrtmgp\nwhere \"mac_aero_mic\" is itself an atmosphere process group, consisting of macrophysics, aerosols, and microphysics processes. If one wanted to add the \"cosp\" atmosphere process to this list, and change the number of its subcycles, it could do so via
$ ./atmchange physics::atm_procs_list+=cosp\n$ ./atmchange cosp::number_of_subcycles=3\nNotice that if we swapped the two commands, we would get an error, since the node \"cosp\" is not present in the XML generated from the defaults until we decide to add it.
It is also possible to declare a new (empty) atmosphere process group, which can then be filled with valid atmosphere processes via subsequent calls to atmchange. The syntax to trigger this behavior consists in specifying a process name that begins and ends with an underscore:
$ ./atmchange physics::atm_procs_list+=_my_group_\nThis adds a new process to the list of processes in \"physics\", called \"_my_group_\", and which is itself an atmosphere process group. Hence, we can then do
$ ./atmchange _my_group_::atm_procs_list+=A,B\nwhere A and B must be valid atmosphere process names (i.e., present in namelist_defaults_scream.xml) or be themselves new atmosphere process groups (i.e., beginning/ending with an underscore)
atmchange can also be used to completely change a list of atmosphere processes:
$ ./atmchange physics::atm_procs_list=A,B,C\nNotice that we used \"=\" instead of \"+=\", which means we will be overwriting the value, rather than appending. Any atmosphere process that was previously in the list but is no longer in it will be removed from the generated namelist_defaults.xml (and scream_input.yaml) files, along with all their nested parameters.
EAMxx allows the user to configure the desired model output via YAML files, with each YAML file associated to a different output file. In order to add an output stream, one needs to run atmchange output_yaml_files+=/path/to/my/output/yaml (more information on how to use atmchange can be found here). During the buildnml phase of the case management system, a copy of these YAML files will be copied into the RUNDIR/data folder. During this process, the files will be parsed, and any CIME-related variable will be resolved accordingly. Therefore, it is not advised to put the original YAML files in RUNDIR/data, since upon buildnml execution, all the CIME vars will no longer be available in the YAML file, making it harder to tweak it, and even harder to share with other users/cases. Another consequence of this is that the user should not modify the YAML files in RUNDIR/data, since any modification will be lost on the next run of buildnml.
The following is a basic example of an output request.
%YAML 1.1\n---\nfilename_prefix: my_output\nAveraging Type: Average\nMax Snapshots Per File: 10\nFields:\n Physics:\n Field Names:\n - T_mid\n - qv\n Dynamics:\n Field Names:\n - dp3d_dyn\n - omega_dyn\noutput_control:\n Frequency: 6\n frequency_units: nhours\nNotice that lists can be equivalently specified in YAML as Field Names: [f1, f2, f3]. The user can specify fields to be outputted from any of the grids used in the simulation. In the example above, we requested fields from both the Physics and Dynamics grid. The meaning of the other parameters is as follows:
Averaging Type: how the fields are integrated in time before being saved. Valid options are
output_control section. In the case above, each snapshot saved to file corresponds to an average of the output fields over 6h windows.filename_prefix: the prefix of the output file, which will be created in the run directory. The full filename will be $prefix.$avgtype.$frequnits_x$freq.$timestamp.nc, where $timestamp corresponds to the first snapshot saved in the file for Instant output, or the beginning of the first averaging window for the other averaging types
Max Snapshots Per File: specifies how many time snapshots can be put in a file. Once this number is reached, EAMxx will close the file and open a new one.Frequency: how many units of time are between two consecutive writes to file. For Instant output the fields are \"sampled\" at this frequency, while for other averaging types the fields are \"integrated\" in time over this windowfrequency_units: units of the output frequency. Valid options are nsteps (the number of atmosphere time steps), nsecs, nmins, nhours, ndays, nmonths, nyears.In addition to the fields computed by EAMxx as part of the timestep, the user can request to output derived quantities, which will be computed on the fly by the I/O interface of EAMxx. There are two types of diagnostic outputs:
quantities computed as a function of EAMxx fields. These are simply physical quantities that EAMxx does not keep in persistent storage. As of May 2024, the available derived quantities are (case sensitive):
PotentialTemperatureAtmosphereDensityExnerVirtualTemperaturez_intz_midgeopotential_intgeopotential_middzDryStaticEnergySeaLevelPressureLiqWaterPathIceWaterPathVapWaterPathRainWaterPathRimeWaterPathShortwaveCloudForcingLongwaveCloudForcingRelativeHumidityZonalVapFluxMeridionalVapFluxprecip_liq_surf_mass_fluxprecip_ice_surf_mass_fluxprecip_total_surf_mass_fluxsurface_upward_latent_heat_fluxwind_speedAerosolOpticalDepth550nmNumberPathAeroComCldTODO: add some information about what each diagnostic is, perhaps a formula
lower-dimensional slices of a field. These are hyperslices of an existing field or of another diagnostic output. As of August 2023, given a field X, the available options are:
X_at_lev_N: slice the field X at the N-th vertical level index. Recall that in EAMxx N=0 corresponds to the model top.X_at_model_bot, X_at_model_top: special case for top and bottom of the model.X_at_Ymb, X_at_YPa, X_at_YhPa: interpolates the field X at a vertical position specified by the give pressure Y. Available units are mb (millibar), Pa, and hPa.X_at_Ym_above_Z: interpolates the field X at a vertical height of Y meters above Z, with Z=surface or Z=sealevel.The following options can be used to to save fields on a different grid from the one they are computed on.
horiz_remap_file: a path to a map file (as produced by ncremap) between the grid where the fields are defined and a coarser grid. EAMxx will use this to remap fields on the fly, allowing to reduce the size of the output file. Note: with this feature, the user can only specify fields from a single grid.vertical_remap_file: similar to the previous option, this map file is used to refine/coarsen fields in the vertical direction.IOGrid: this parameter can be specified inside one of the grids sections, and will denote the grid (which must exist in the simulation) where the fields must be remapped before being saved to file. This feature is really only used to save fields on the dynamics grid without saving twice the DOFs at the interface of two spectral elements. E.g., for a scalar quantity defined only in the horizontal direction, native output from the Dynamics grid would produce arrays of length nelems*ngp*ngp, where ngp is the number of Gauss points along each axis in the 2d spectral element, and nelems is the number of horizontal elements. However, due to continuity, the values on the Gauss points on the element boundary must match the values on the neighboring element, resulting in duplicated data. By remapping to a \"unique\" version of the dynamics grid (which in EAMxx is referred to as \"Physics GLL\"), we can save roughly 45% of storage. Note: this feature cannot be used along with the horizontal/vertical remap.It is also possible to request tendencies of fields that are updated by atmosphere processes, on a per-process basis (here, \"updated\" means that the field is both an input as well as an output of the atmosphere process). Since the term \"tendency\" can be used with slightly different connotations, we clarify what we mean by that when it comes to model output: if process P updates field A, by the tendency of A from process P we mean (A_after_P - A_before_P) / dt, where dt is the atmosphere timestep.
As of May 2024, the user needs two things in order to get tendencies from a process. E.g., to get the tendencies of T_mid and horiz_winds from the process shoc, one needs:
atmchange shoc::compute_tendencies=T_mid,horiz_winds;shoc_T_mid_tend and shoc_horiz_winds_tend to the list of fields in the desired output YAML file.The YAML file shown at the top of this section, together with the remap options in the following section, covers most of the options used in a typical run. There are however particular use cases that require some less common options, which we list here (in parentheses, the location in the YAML file and the type of the parameter value).
flush_frequency (toplevel list, integer): this parameter can be used to specify how often the IO library should sync the in-memory data to file. If not specified, the IO library is free to decide when it should flush the data. This option can be helpful for debugging, in case a crash is occurring after a certain number of steps, but before the IO library would automatically flush to file.Floating Point Precision (toplevel list, string): this parameter specifies the precision to be used for floating point variables in the output file. By default, EAMxx uses single precision. Valid values are single, float, double, and real. The first two are synonyms, while the latter resolves to single or double depending on EAMxx cmake configuration parameter SCREAM_DOUBLE_PRECISION.file_max_storage_type (toplevel list, string): this parameter determines how the capacity of the file is specified. By default, it is set to num_snapshots, which makes EAMxx read Max Snapshots Per File (explained in the first section). However, the user can specify one_year or one_month, which will make EAMxx create one output file per year/month of simulation, fitting however many snapshots are needed in each file (depending on the output frequency). If one_year or one_month are used, the option Max Snapshots Per File is ignored.MPI Ranks in Filename (toplevel list, boolean): this option specifies whether the number of MPI ranks in the atm communicator should be put inside the output file name. By default, this is false, since it is usually not important. This option is mostly important for standalone unit testing, where several versions of the same test (corresponding to different numbers of MPI ranks) are running concurrently, so that different file names are needed to avoid resource contention.save_grid_data (output_control sublist, boolean): this option allows to specify whether grid data (such as lat/lon) should be added to the output stream. By default, it is true.iotype (toplevel list, string): this option allows the user to request a particular format for the output file. The possible values are default, netcdf, pnetcdf,adios,hdf5, wheredefault` means \"whatever is the PIO type from the case settings\".skip_t0_output (output_control sublist, boolean): this option is relevant only for Instant output, where fields are also outputed at the case start time (i.e., after initialization but before the beginning of the first timestep). By default it is set to false.output_yaml_files atm option (which can be queried via atmquery output_yaml_files). The user can specify a few options, in order to tweak the restart behavior:Perform Restart (Restart sublist, boolean): this parameter is true by default, but can be set to false to force the model to ignore any history restart files, and start the output stream from scratch, as if this was an initial run.filename_prefix (Restart sublist, string): by default, this parameter is set to match the value of filename_prefix from the toplevel list. It can be set to something else in case we want to restart a previous simulation that was using a different filename prefix.force_new_file (Restart sublist, boolean): this parameter allows to start a fresh new output file upon restarts. By default, is is set to false, so that EAMxx will attempt to resume filling the last produced output file (if any, and if it can accommodate more snapshots).Nudging is supported in EAMxx. Currently, it is possible to nudge EAMxx to the output from a different EAMxx run or to reanalysis. Nudging data can be on your model grid or an arbitrary coarser grid. Inline interpolating of finer-grid nudging data to a coarser model resolution isn't implemented yet but may be in the future.
"},{"location":"user/nudging/#data-to-nudge-towards","title":"Data to nudge towards","text":"The user is expected to prepapre (and then use atmchange to point to) nudging data files that are compliant with EAMxx specification. In practice, this means that the data files must contain variable names known to EAMxx (only U, V, T_mid, and qv are supported now). The files can be specified with an explicit list or via pattern matching. The files must contain an arbitary global attribute case_t0, and it is recommended to be the same as the time dimension unit (the files must be time-dimensioned). Finally, the dimension order must be such that lev is the last dimension, so most likely, the user must transpose the dimensions.
Pressure can be explicitly provided in the nudging data as time-varying p_mid corresponding to the option TIME_DEPENDENT_3D_PROFILE for source_pressure_type. Alternatively, the data can contain a time-invariant pressure variable p_lev corresponding to the option TIME_DEPENDENT_3D_PROFILE for source_pressure_type.
In regionally refined model applications, it is possible to use weighted nudging, for example, to avoid nudging the refined region. To achieve that, the user can use atmchange to set use_nudging_weights (boolean) and provide nudging_weights_file that has the weight to apply for nudging (for example, zeros in the refined region). Currently, weighted nudging is only supported if the user provides the nudging data at the target grid.
To enable nudging as a process, one must declare it in the atm_procs_list runtime parameter.
./atmchange physics::atm_procs_list=\"mac_aero_mic,rrtmgp,cosp,nudging\"\nThe following options are needed to specify the nudging.
./atmchange nudging::nudging_filenames_patterns=\"/pathto/nudging_files/*.nc\" # can provide file name explicitly here instead (or multiple patterns)\n./atmchange nudging::source_pressure_type=TIME_DEPENDENT_3D_PROFILE # see section on pressure\n./atmchange nudging::nudging_fields=U,V # can include qv, T_mid as well\n./atmchange nudging::nudging_timescale=21600 # in seconds\nTo gain a deeper understanding of these parameters and options, please refer to code implementation of the nudging process.
"},{"location":"user/rrm_eamxx/","title":"Running EAMxx with a Regionally Refined Mesh (RRM)","text":"Running EAMxx with a RRM allows you run a select region of the globe at high resolution (i.e. 3 km) with the remainder of the globe at a lower resolution (i.e. 25 or 100 km). This document will point you to the steps required and resources available to assist in developing and running a new RRM.
"},{"location":"user/rrm_eamxx/#choose-your-rrm","title":"Choose Your RRM","text":"What region of the globe do you want to refine? Your first step should be to check library of RRM grids/cases that have already been developed to potentially avoid duplicate work. If you found a RRM that suits your needs, you can skip the next step (\"Generate Your RRM\").
"},{"location":"user/rrm_eamxx/#generate-your-rrm","title":"Generate Your RRM","text":"Please refer to the offical E3SM guide for developing new atmosphere grids, which provides detailed guidance for developing your RRM.
After you have made all the necessary files for your RRM, you will need to configure your code branch so that it knows about your new grid. The steps required to do this are documented at the top of the library of RRM grids/cases page.
"},{"location":"user/rrm_eamxx/#make-your-initial-condition-file","title":"Make Your Initial Condition File","text":"The easiest way to generate an initial condition is to use the HICCUP tool, which is a set of flexible and robust python routines to streamline the task of generating a new atmospheric initial condition for E3SM. Otherwise, please see the step-by-step instuctions if you prefer to manually generate your initial condition.
"},{"location":"user/rrm_eamxx/#assemble-nudging-data-optional","title":"Assemble Nudging Data (Optional)","text":"If you wish to nudge your simulation, assemble your nudging data in the format required by EAMxx. Please refer to the nudging documentation.
In the event that you only want to nudge a portion of your domain, then you will need to generate a nudging weights file. A common use case for this is when you want the high-resolution region to remain free-running (unnudged) while nudging the coarse domain towards reanalysis or model data. Please use this script as an example of how to generate your nudging weights file.
"},{"location":"user/rrm_eamxx/#run-your-rrm","title":"Run your RRM","text":"Congratulations, you are now ready to run your EAMxx RRM. If you are running your RRM in free running mode (not using any nudging) then you simply need to modify an existing EAMxx script and change the resolution to match the one you created for your RRM.
If you are using nudging, then please see this example script of how to run a nudged EAMxx RRM run. This example script uses the California 3-km RRM, which is on the master branch.
"}]} \ No newline at end of file diff --git a/sitemap.xml.gz b/sitemap.xml.gz index 8109a5d0cfd..bd75785e9f5 100644 Binary files a/sitemap.xml.gz and b/sitemap.xml.gz differ