update IDM markdown

utils/idmloader/IDM.md

@@ -1,111 +1,45 @@
-# MODFLOW 6 Input Data Model developer resource
-
+# MODFLOW 6 Input Data Model
 MODFLOW 6 reads simulation input from text and binary input files.  Traditionally, each model package has been responsible for reading its own input.  Although there are low-level utilities for reading arrays, and parsing input blocks, these utilities are called from dedicated read routines for each package.  To support new types of input, such as from NetCDF files or through the Application Programming Interface, a new effort is underway to implement a comprehensive Input Data Model (IDM) for MODFLOW 6.  Implementation of an Output Data Model may follow this effort.
 
-This document intends to describe, from a development perspective, the MODFLOW 6 IDM, and provide guidance for how to work with it. As the IDM is an active area of development this document should be expected to change frequently and will be updated to track with current state.
-
-* [Overview](#overview)
-* [Terminology](#terminology)
-* [Package Update Process](#package-update-process)
+- [Overview](#overview)
+- [Framework](#framework)
+- [Design](#design)
+- [Integration](#integration)
 
 ## Overview
-The MODFLOW 6 IDM is intended to provide a common entry point for simulation input data.  The approach is to read the input data first into memory and store the input data in the "input context."  Reading the simulation input into memory is performed by IDM load routines.  Model packages are then refactored so that they obtain their required information from the input context, rather than directly from the input files.  We refer to the process as "sourcing."  A model package "sources" its input from the input context.  The advantage here is that other readers can be implemented to load information into the input context, and then the model packages should work without modification.   This approach is built upon the MODFLOW 6 memory management infrastructure to define variable access paths that are scoped to a particular component/subcomponent. Input variables themselves are ultimately described by MODFLOW 6 definition (dfn) files.
+The MODFLOW 6 IDP (Input Data Processor) is a subsystem meant to generically read and store simulation input data from more than one type of input source.  It is built upon existing MODFLOW 6 capabilities, specifically input parameter descriptions currently stored in \*.dfn (definition) files, and the Memory Manager.  The parameter descriptions provide IDP with input block and parameter attributes needed to create memory and store input data.  The Memory Manager provides a globally accessible way to create, update, access and remove input (and other) data.  Existing components that read a traditional ASCII input file can be updated to source their input from the Memory Manager.  This type of update is a significant step towards supporting sources of input data other than the traditional ASCII files. 
 
-The MODFLOW 6 IDM is intended to decouple a simulation run from any particular, supported source of input that a user might construct. Currently,  MODFLOW 6 packages read input directly from proprietary MODFLOW-specific input files as required during a simulation run. The new design assigns responsibility for reading and loading data from the files to IDM proper, which builds the input context from the supported user input source (currently, MODFLOW 6 input files). Simulation components then access (or source) the input context to retrieve relevant input data.
+## Framework
+MODFLOW 6 \*.dfn files are used pre-compile time to generate fortran source files containing a subset of the parameter and block attribute information necessary to process input for the simulation. A single definition file is converted into one fortran source file that defines the component parameters and blocks and organizes them into lists.  This conversion from defintion file to fortran source is currently managed by the [dfn2f90.py](scripts/dfn2f90.py) script.  This script also creates framework source files with routines for generically accessing package definitions by component.  A package or other component intended to be updated and integrated with IDM must be added to the dfn2f90.py script.  This process is described below.
 
-## Terminology
-### Definition File (.dfn suffix)
-MODFLOW 6 dfn file formats are described in the [dfn readme.md](../../doc/mf6io/mf6ivar/readme.md). These definition files describe MODFLOW 6 input files and parameters.  They are used to generate the documentation in mf6io.pdf.  They are also used to auto-generate the FloPy classes for MODFLOW 6. Definition files are also read as input to the dfn2f90.py script, which is a new utility developed as part of this IDM effort.  This script generates Fortran source code that describes MODFLOW 6 input.  The Fortran source code generated by dfn2f90.py defines the input parameters and input file structure, and is used by IDM to load simulation input.
-### input definition
-The Fortran version of the Input definitions are generated from MODFLOW 6 dfn files and describe input parameter or blocks:
+## Design
+Input data stored in the Memory Manager use the special identifier ```__INPUT__``` as a prefix to memory path.  MODFLOW 6 refers to memory path prefixes such as these as a "context" and as such the collection of input data stored in the Memory Manager is referred to as the "input context".
 
-```fortran
-type(InputParamDefinitionType), parameter :: &
-  gwtdsp_opt_xt3d_off = InputParamDefinitionType &
-  ( &
-  'GWT', & ! component
-  'DSP', & ! subcomponent
-  'OPTIONS', & ! block
-  'XT3D_OFF', & ! tag name
-  'XT3D_OFF', & ! fortran variable
-  'KEYWORD', & ! type
-  '', & ! shape
-  .false., & ! required
-  .false., & ! multi-record
-  .false., & ! preserve case
-  .false. & ! layered
-  )
-```    
+IDM defines and implements loaders as objects to read input from some supported source and to allocate and store that input in a source independent way for the package or component.  Variations across sources are managed by the loaders.  Input retrieved by simulation packages is stuctured such that the source is not apparent or relevant to the package.
 
-These Fortran definitions are used by IDM input handlers to interpret user input and load input data into the input context.
+IDM distinguishes between static and dynamic loader objects.  Static loader objects load all pre-period input before any simulation objects are created and prepare for dynamic (stress period) data loads by creating the dynamic loader when relevant.  Static and dynamic loaders are defined per supported source by extending base types defined in InputLoadType.f90.  Top level IDM context (IdmLoad.f90) maintains pointers to loader base types and invoke the common deferred interfaces load() (static) and rp() (dynamic) for each component.  Each source can introduce as much complexity as needed to support variations in input by introducing source specific loader types.  For example, there are currently distinct loaders for dynamic ASCII list based and array-based inputs.  
 
-A related set of input definitions are contained in a Fortran file that is named from the related package file, e.g. [gwf3npf8idm.f90](../../src/Model/GroundWaterFlow/gwf3npf8idm.f90) contains input definitions relevant to [gwf3npf8.f90](../../src/Model/GroundWaterFlow/gwf3npf8.f90). A Fortran input definition file is organized into 3 lists, a parameter (input variable) list, an aggregate (similar to a numpy recarray) list, and a block list.  The Fortran input definitions also contain the definition for a found type object that can be used within packages to track what input variable paths were loaded into the input context by the IDM load routine:
+## Integration
+A simulation component (package, etc) can be integrated with IDM by following these steps:
+- Create dfn file
+- Add component to dfn2f90.py and run script
+- Update MODFLOW 6 build scripts (e.g. meson / msvs)
+- Use common intefaces to source input from input context
+
+The dfn file is already a requirement for MODFLOW 6 package integration and so is not an IDM specific requirement.  The dfn2f90.py update should be straightforward and currently amounts to specifying the input path to the package dfn file and an output path for the generated package \*idm.f90 file.  To run the script:
 
-```fortran
-type GwtDspParamFoundType
-  logical :: opt_xt3d_off = .false.
-  logical :: opt_xt3d_rhs = .false.
-  logical :: grid_diffc = .false.
-  logical :: grid_alh = .false.
-  logical :: grid_alv = .false.
-  logical :: grid_ath1 = .false.
-  logical :: grid_ath2 = .false.
-  logical :: grid_atv = .false.
-end type GwtDspParamFoundType`
-```
-## Package Update process
-###	Update [dfn2f90.py](scripts/dfn2f90.py)
-Add a new dfns entry in main, designating paths and names for the input dfn and output f90 files.
-### Run the dfn2f90.py script
 ```shell
 cd utils/idmloader/scripts
 python dfn2f90.py
 ```
-This will create the new IDM Fortran definition file at the location designated.  This will also automatically update IDM selector modules so that the newly generated definitions can be used.  If a new component (e.g. a new model) has been introduced, a new selector file will be generated in the src/Utitilites/Idm/selector directory.
-### Update [meson.build](../../src/meson.build) and [mf6core.vfproj](../../msvs/mf6core.vfproj)
-Add any newly generated fortran files to relevant build scripts to compile new definitions and definition select routines into MODFLOW 6 binaries.
-
-Note:  To simplify the update process, all necessary internal modifications are performed when dfn2f90.py is run.  Compiling with these changes immediately updates IDM to treat any newly added package as integrated.  This may not be the case if, for example, model package code has not been updated to source input from the input context.  GWF and GWT model code is fully integrated with IDM and as such either a valid unit number (for packages that are not IDM integrated) or a valid mempath (for packages that are IDM integrated) is passed into a package but not both.  Once a new fortran idm definition file has been compiled in, a valid unit number will no longer be provided to the package and parser operations will fail if attempted.
-### Update the package file
-#### Source package input data 
-To convert a package to use the new IDM approach, the read routine for the package must be replaced by a sourcing routine that accesses data from the input context.  The implementation of the source routine is dependent on the data itself but a common pattern is to use the MemoryManagerExtModule `mem_set_value()` interface to copy data from the input context to package paths.  A parameter found type, in the generated IDM definition file, should be used to pass a corresponding logical to `mem_set_value()`, which sets the logical to True if the input path was found and data was copied.  When sourcing has been completed, the found type parameter logicals can be checked to determine what other actions need to be taken in response to both found or not found input data.
-
-```fortran
-character(len=LENMEMPATH) :: idmMemoryPath
-type(GwtDspParamFoundType) :: found
-! ------------------------------------------------------------------------------
-!
-! -- set memory path
-idmMemoryPath = create_mem_path(this%name_model, 'DSP', idm_context)
-!
-! -- update defaults with idm sourced values
-call mem_set_value(this%ixt3doff, 'XT3D_OFF', idmMemoryPath, found%opt_xt3d_off)
-call mem_set_value(this%ixt3drhs, 'XT3D_RHS', idmMemoryPath, found%opt_xt3d_rhs)
-!
-! -- set xt3d state flag
-if (found%opt_xt3d_off) this%ixt3d = 0
-if (found%opt_xt3d_rhs) this%ixt3d = 2
-```
+Running this command will generate a new definition file for the package and update the IDM selector framework.  If the package also introduces a new model then a new selector framework file will also be created.  Update the MODFLOW 6 build system (meson and msvs) files with any newly generated files.
 
-#### Deallocate package input paths
-In deallocate, add a call to MemoryManagerExtModule `memorylist_remove()` for the package.  This call will search the subcomponent input path and deallocate all memory that was allocated as part of the load process.
+Once these files have been added to the build the package can be updated to source it's input from the input context.  In general, static input data is copied from the input context to the package (or "model") context by using the ```mem_set_value()``` interface:
 
-```fortran
-! -- Deallocate input memory
-call memorylist_remove(this%name_model, 'DSP', idm_context)
-```
-
-## Adding params to packages already using IDM
-
-Once a model package has been modified to use the IDM approach, adding a new package option or block requires two steps.  The steps are as follows. 
-
-### Update appropriate dfn file with new param(s)
-### Run the dfn2f90.py script
 ```shell
-cd utils/idmloader/scripts
-python dfn2f90.py
+call mem_set_value(this%xorigin, 'XORIGIN', this%input_mempath, found%xorigin)
 ```
 
-This will update the existing idm package f90 file to add the new parameter definition.  Compile to update the binaries.
-#### Source package input data
-Update the package source routine for the relevant block to copy the data from the input path to the package path. Take any necessary action depending on whether the data was found or not found. 
+The found instance is a convenience type generated by dfn2f90.py and should be complete and ready to use from the package \*idm.f90 module.  The inteface updates the data pointer (```this%xorigin``` in the example) only if the relevant data was provided as input and will set the logical ```found%xorigin``` to TRUE when this is the case.
+
+Dynamic data can also be copied between contexts by using the ```mem_set_value()``` interface or it can be accessed directly by setting a pointer to it.  In most cases, dynamic input memory is not reallocated during it's lifetime and such a pointer is valid for the duration of the simulation.  See CHD package code for an example (e.g. "HEAD")