Merge branch 'development' into time_dependent_injection

.github/workflows/cuda.yml

@@ -110,7 +110,7 @@ jobs:
         which nvcc || echo "nvcc not in PATH!"
 
         git clone https://github.com/AMReX-Codes/amrex.git ../amrex
-        cd ../amrex && git checkout --detach 41a6700392400bab9a688ca6892a6d638483f6e0 && cd -
+        cd ../amrex && git checkout --detach d28b9ab1cd64e648372ae679a6be01c733899525 && cd -
         make COMP=gcc QED=FALSE USE_MPI=TRUE USE_GPU=TRUE USE_OMP=FALSE USE_PSATD=TRUE USE_CCACHE=TRUE -j 2
 
   build_nvhpc21-11-nvcc:

Docs/source/developers/python.rst

@@ -1,7 +1,89 @@
 .. _development-python:
 
-Python interface
-================
+Processing PICMI Input Options
+==============================
+
+The input parameters in a WarpX PICMI file are processed in two layers.
+The first layer is the Python level API, which mirrors the :ref:`C++ application input structure <running-cpp-parameters>`; the second is the translation from the PICMI input to the equivalent :ref:`app (AMReX) input file parameters <running-cpp-parameters>`.
+
+The two layers are described below.
+
+Input parameters
+----------------
+
+In a C++ input file, each of the parameters has a prefix, for example ``geometry`` in ``geometry.prob_lo``.
+For each of these prefixes, an instance of a Python class is created and the parameters saved as attributes.
+This construction is used since the lines in the input file look very much like a Python assignment statement,
+assigning attributes of class instances, for example ``geometry.dims = 3``.
+
+Many of the prefix instances are predefined, for instance ``geometry`` is created in the file ``Python/pywarpx/Geometry.py``.
+In that case, ``geometry`` is an instance of the class ``Bucket`` (specified in ``Python/pywarpx/Bucket.py``),
+the general class for prefixes.
+It is called ``Bucket`` since its main purpose is a place to hold attributes.
+Most of the instances are instances of the ``Bucket`` class.
+There are exceptions, such as ``constants`` and ``diagnostics`` where extra processing is needed.
+
+There can also be instances created as needed.
+For example, for the particle species, an instance is created for each species listed in ``particles.species_names``.
+This gives a place to hold the parameters for the species, e.g., ``electrons.mass``.
+
+The instances are then used to generate the input parameters.
+Each instance can generate a list of strings, one for each attribute.
+This happens in the ``Bucket.attrlist`` method.
+The strings will be the lines as in an input file, for example ``"electrons.mass = m_e"``.
+The lists for each instance are gathered into one long list in the ``warpx`` instance (of the class ``WarpX`` defined in
+``Python/pywarpx/WarpX.py``).
+This instance has access to all of the predefined instances as well as lists of the generated instances.
+
+In both of the ways that WarpX can be run with Python, that list of input parameter strings will be generated.
+This is done in the routine ``WarpX.create_argv_list`` in ``Python/pywarpx/WarpX.py``.
+If WarpX will be run directly in Python, that list will be sent to the ``amrex_init`` routine as the ``argv``.
+This is as if all of the input parameters had been specified on the command line.
+If Python is only used as a prepocessor to generate the input file, the list are the strings that are written out to create the
+input file.
+
+There are two input parameters that do not have prefixes, ``max_step`` and ``stop_time``.
+These are handled via keyword arguments in the ``WarpX.create_argv_list`` method.
+
+Conversion from PICMI
+---------------------
+
+In the PICMI implementation, defined in ``Python/pywarpx/picmi.py``, for each PICMI class, a class was written that
+inherits the PICMI class and does the processing of the input.
+Each of the WarpX classes has two methods, ``init`` and ``initialize_inputs``.
+The ``init`` method is called during the creation of the class instances that happens in the user's PICMI input file.
+This is part of the standard - each of the PICMI classes call the method ``handle_init`` from the constructor ``__init__`` routines.
+The main purpose is to process application specific keyword arguments (those that start with ``warpx_`` for example).
+These are then passed into the ``init`` methods.
+In the WarpX implementation, in the ``init``, each of the WarpX specific arguments are saved as attributes of the implementation
+class instancles.
+
+It is in the second method, ``initialize_inputs``, where the PICMI input parameters are translated into WarpX input parameters.
+This method is called later during the intialization.
+The prefix instances described above are all accessible in the implementation classes (via the ``pywarpx`` module).
+For each PICMI input quantity, the appropriate WarpX input parameters are set in the prefix classes.
+As needed, for example in the ``Species`` class, the dynamic prefix instances are created and the attributes set.
+
+Simulation class
+----------------
+
+The ``Simulation`` class ties it all together.
+In a PICMI input file, all information is passed into the ``Simulation`` class instance, either through the constructor
+or through ``add_`` methods.
+Its ``initialize_inputs`` routine initializes the input parameters it handles and also calls the ``initialize_inputs``
+methods of all of the PICMI class instances that have been passed in, such as the field solver, the particles species,
+and the diagnostics.
+As with other PICMI classes, the ``init`` routine is called by the constructor and ``initialize_inputs`` is called during
+initialization.
+The initialization happens when either the ``write_input_file`` method is called or the ``step`` method.
+After ``initialize_inputs`` is finished, the attributes of the prefix instances have been filled in, and the process described
+above happens, where the prefix instances are looped over to generate the list of input parameter strings (that is either written
+out to a file or passed in as ``argv``).
+The two parameters that do not have a prefix, ``max_step`` and ``stop_time``, are passed into the ``warpx`` method as keyword
+arguments.
+
+Python runtime interface
+========================
 
 The Python interface provides low and high level access to much of the data in WarpX.
 With the low level access, a user has direct access to the underlying memory contained

Docs/source/install/batch/lsf.rst

@@ -0,0 +1,39 @@
+Job Submission
+''''''''''''''
+
+* ``bsub your_job_script.bsub``
+
+
+Job Control
+'''''''''''
+
+* interactive job:
+
+  * ``bsub -P $proj -W 2:00 -nnodes 1 -Is /bin/bash``
+
+* `details for my jobs <https://docs.olcf.ornl.gov/systems/summit_user_guide.html#monitoring-jobs>`_:
+
+  * ``bjobs 12345`` all details for job with <job id> ``12345``
+  * ``bjobs [-l]`` all jobs under my user name
+  * ``jobstat -u $(whoami)`` job eligibility
+  * ``bjdepinfo 12345`` job dependencies on other jobs
+
+* details for queues:
+
+  * ``bqueues`` list queues
+
+* communicate with job:
+
+  * ``bkill <job id>`` abort job
+  * ``bpeek [-f] <job id>`` peek into ``stdout``/``stderr`` of a job
+  * ``bkill -s <signal number> <job id>`` send signal or signal name to job
+  * ``bchkpnt`` and ``brestart`` checkpoint and restart job (untested/unimplemented)
+  * ``bmod -W 1:30 12345`` change the walltime of a job (currently not allowed)
+  * ``bstop <job id>`` prevent the job from starting
+  * ``bresume <job id>`` release the job to be eligible for run (after it was set on hold)
+
+
+References
+''''''''''
+
+* https://www.ibm.com/docs/en/spectrum-lsf

Docs/source/install/batch/pbs.rst

@@ -0,0 +1,38 @@
+Job Submission
+''''''''''''''
+
+* ``qsub your_job_script.qsub``
+
+
+Job Control
+'''''''''''
+
+* interactive job:
+
+  * ``qsub -I``
+
+* details for my jobs:
+
+  * ``qstat -f 12345`` all details for job with <job id> ``12345``
+  * ``qstat -u $(whoami)`` all jobs under my user name
+
+* details for queues:
+
+  * ``qstat -a queueName`` show all jobs in a queue
+  * ``pbs_free -l`` compact view on free and busy nodes
+  * ``pbsnodes`` list all nodes and their detailed state (free, busy/job-exclusive, offline)
+
+* communicate with job:
+
+  * ``qdel <job id>`` abort job
+  * ``qsig -s <signal number> <job id>`` send signal or signal name to job
+  * ``qalter -lwalltime=12:00:00 <job id>`` change the walltime of a job
+  * ``qalter -Wdepend=afterany:54321 12345`` only start job ``12345`` after job with id ``54321`` has finished
+  * ``qhold <job id>`` prevent the job from starting
+  * ``qrls <job id>`` release the job to be eligible for run (after it was set on hold)
+
+
+References
+''''''''''
+
+* https://www.openpbs.org

Docs/source/install/batch/pjm.rst

@@ -0,0 +1,45 @@
+.. note::
+
+   This section is a stub and improvements to complete the ``(TODO)`` sections are welcome.
+
+Job Submission
+''''''''''''''
+
+* ``pjsub your_job_script.pjsub``
+
+
+Job Control
+'''''''''''
+
+* interactive job:
+
+  * ``pjsub --interact``
+
+* details for my jobs:
+
+  * ``pjstat`` status of all jobs
+  * (TODO) all details for job with <job id> ``12345``
+  * (TODO) all jobs under my user name
+
+* details for queues:
+
+  * (TODO) show all jobs in a queue
+  * (TODO) compact view on free and busy nodes
+  * (TODO) list all nodes and their detailed state (free, busy/job-exclusive, offline)
+
+* communicate with job:
+
+  * ``pjdel <job id>`` abort job
+  * (TODO) send signal or signal name to job
+  * (TODO) change the walltime of a job
+  * (TODO) only start job ``12345`` after job with id ``54321`` has finished
+  * ``pjhold <job id>`` prevent the job from starting
+  * ``pjrls <job id>`` release the job to be eligible for run (after it was set on hold)
+
+
+References
+''''''''''
+
+* https://www.bsc.es/user-support/arm.php#ToC-runningjobs
+* https://www.cc.kyushu-u.ac.jp/scp/eng/system/ITO/02-2_batch.html
+* https://www.r-ccs.riken.jp/en/fugaku/user-guide/

Docs/source/install/batch/slurm.rst

@@ -0,0 +1,44 @@
+Job Submission
+''''''''''''''
+
+* ``sbatch your_job_script.sbatch``
+
+
+Job Control
+'''''''''''
+
+* interactive job:
+
+  * ``salloc --time=1:00:00 --nodes=1 --ntasks-per-node=4 --cpus-per-task=8``
+
+    * e.g. ``srun "hostname"``
+  * GPU allocation on most machines require additional flags, e.g. ``--gpus-per-task=1`` or ``--gres=...``
+
+* details for my jobs:
+
+  * ``scontrol -d show job 12345`` all details for job with <job id> ``12345``
+  * ``squeue -u $(whoami) -l`` all jobs under my user name
+
+* details for queues:
+
+  * ``squeue -p queueName -l`` list full queue
+  * ``squeue -p queueName --start`` (show start times for pending jobs)
+  * ``squeue -p queueName -l -t R`` (only show running jobs in queue)
+  * ``sinfo -p queueName`` (show online/offline nodes in queue)
+  * ``sview`` (alternative on taurus: ``module load llview`` and ``llview``)
+  * ``scontrol show partition queueName``
+
+* communicate with job:
+
+  * ``scancel <job id>`` abort job
+  * ``scancel -s <signal number> <job id>`` send signal or signal name to job
+  * ``scontrol update timelimit=4:00:00 jobid=12345`` change the walltime of a job
+  * ``scontrol update jobid=12345 dependency=afterany:54321`` only start job ``12345`` after job with id ``54321`` has finished
+  * ``scontrol hold <job id>`` prevent the job from starting
+  * ``scontrol release <job id>`` release the job to be eligible for run (after it was set on hold)
+
+
+References
+''''''''''
+
+* https://slurm.schedmd.com/documentation.html

Docs/source/install/hpc.rst

@@ -6,6 +6,9 @@ HPC
 On selected high-performance computing (HPC) systems, WarpX has documented or even pre-build installation routines.
 Follow the guide here instead of the generic installation routines for optimal stability and best performance.
 
+
+.. _install-hpc-profile:
+
 warpx.profile
 -------------
 
@@ -18,30 +21,83 @@ Ideally, store that file directly in your ``$HOME/`` and source it after connect
 
 We list example ``warpx.profile`` files below, which can be used to set up WarpX on various HPC systems.
 
-HPC Systems
------------
+
+.. _install-hpc-machines:
+
+HPC Machines
+------------
+
+This section documents quick-start guides for a selection of supercomputers that WarpX users are active on.
 
 .. toctree::
    :maxdepth: 1
 
    hpc/adastra
    hpc/cori
-   hpc/perlmutter
-   hpc/summit
-   hpc/spock
    hpc/crusher
    hpc/frontier
    hpc/fugaku
+   hpc/hpc3
    hpc/juwels
    hpc/lassen
-   hpc/quartz
    hpc/lawrencium
-   hpc/ookami
-   hpc/lxplus
    hpc/lumi
+   hpc/lxplus
+   hpc/ookami
+   hpc/perlmutter
+   hpc/quartz
+   hpc/spock
+   hpc/summit
    hpc/taurus
 
 .. tip::
 
    Your HPC system is not in the list?
    `Open an issue <https://github.com/ECP-WarpX/WarpX/issues>`__ and together we can document it!
+
+
+.. _install-hpc-batch:
+
+Batch Systems
+-------------
+
+HPC systems use a scheduling ("batch") system for time sharing of computing resources.
+The batch system is used to request, queue, schedule and execute compute jobs asynchronously.
+The individual HPC machines above document job submission example scripts, as templates for your modifications.
+
+In this section, we document a quick reference guide (or cheat sheet) to interact in more detail with the various batch systems that you might encounter on different systems.
+
+Slurm
+"""""
+
+Slurm is a modern and very popular batch system.
+Slurm is used at NERSC, OLCF Frontier, among others.
+
+.. include:: batch/slurm.rst
+
+
+LSF
+"""
+
+LSF (for *Load Sharing Facility*) is an IBM batch system.
+It is used at OLCF Summit, LLNL Lassen, and other IBM systems.
+
+.. include:: batch/lsf.rst
+
+
+PBS
+"""
+
+PBS (for *Portable Batch System*) is a popular HPC batch system.
+The OpenPBS project is related to `PBS, PBS Pro and TORQUE <https://en.wikipedia.org/wiki/Portable_Batch_System>`__.
+
+.. include:: batch/pbs.rst
+
+
+PJM
+"""
+
+PJM (probably for *Parallel Job Manager*?) is a Fujitsu batch system
+It is used at RIKEN Fugaku and on other Fujitsu systems.
+
+.. include:: batch/pjm.rst

Docs/source/install/hpc/adastra.rst

@@ -63,7 +63,7 @@ And since Adastra does not yet provide a module for them, install c-blosc and AD
    # ADIOS2
    git clone -b v2.8.3 https://github.com/ornladios/ADIOS2.git src/adios2
    rm -rf src/adios2-pm-build
-   cmake -S src/adios2 -B src/adios2-pm-build -DADIOS2_USE_Blosc=ON -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DADIOS2_USE_ZeroMQ=OFF -DCMAKE_INSTALL_PREFIX=${HOME}/sw/adastra/gpu//adios2-2.8.3
+   cmake -S src/adios2 -B src/adios2-pm-build -DADIOS2_USE_Blosc=ON -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DADIOS2_USE_ZeroMQ=OFF -DCMAKE_INSTALL_PREFIX=${HOME}/sw/adastra/gpu/adios2-2.8.3
    cmake --build src/adios2-pm-build --target install -j 16
 
 Then, ``cd`` into the directory ``$HOME/src/warpx`` and use the following commands to compile:
@@ -73,7 +73,7 @@ Then, ``cd`` into the directory ``$HOME/src/warpx`` and use the following comman
    cd $HOME/src/warpx
    rm -rf build
 
-   cmake -S . -B build -DWarpX_COMPUTE=HIP
+   cmake -S . -B build -DWarpX_DIMS="1;2;3" -DWarpX_COMPUTE=HIP -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON
    cmake --build build -j 32
 
 The general :ref:`cmake compile-time options <building-cmake>` apply as usual.

Docs/source/install/hpc/crusher.rst

@@ -68,7 +68,7 @@ Then, ``cd`` into the directory ``$HOME/src/warpx`` and use the following comman
    cd $HOME/src/warpx
    rm -rf build
 
-   cmake -S . -B build -DWarpX_DIMS=3 -DWarpX_COMPUTE=HIP -DWarpX_PSATD=ON
+   cmake -S . -B build -DWarpX_DIMS="1;2;RZ;3" -DWarpX_COMPUTE=HIP -DWarpX_PSATD=ON
    cmake --build build -j 10
 
 The general :ref:`cmake compile-time options <building-cmake>` apply as usual.