Testing Group Tags and Attributes

FORTPY: Testing Groups

`<group>` Tag Attributes

The <group> tag with purpose="testing" can have any of the following tags: <test>, <global>, <assignment>, <mapping>, <prereq>, <target>. When these tags are at the <group> level, they apply to all unit <test>s defined within the group. In addition to child XML tags, <group> can also have these attributes, which reference relevant tags in the <globals> section of the XML file.

inputs a ;-separated list of <input> tag identifiers from the <globals> tag in the XML file to copy as children of the <test> tag. Since this attribute only looks at <globals> tags of type <input>, the global: prefix is not required for the identifiers. If the identifier starts with ^, then the tag is inserted at the start of the containing tag's sub-element list instead of being appended.
globals as for inputs, a ;-separated list of <global> tag identifiers from the <globals> tag in the XML file to copy as children. See also the note on that attribute about ^.
assignments as for inputs, a ;-separated list of <assignment> tag identifiers from the <globals> tag in the XML file to copy as children. See also the note on that attribute about ^.
mappings as for inputs, a ;-separated list of <mapping> tag identifiers from the <globals> tag in the XML file to copy as children. See also the note on that attribute about ^.

`<global>` Tag

The global tag gives definition and initialization information for those variables that will be declared as variables in the fortran program. Since the context of the program is a parent to all methods contexts called from the program, we call them "global" variables.

<global name="dFull" type="real(dp)" modifiers="pointer" 
        default="> null()" dimensions=":,:" ignore="true" />

default specifies the default value to assign to the variable as part of initialization. Can be the name of an existing variable or a valid fortran function. The contents of the attribute will be pasted directly into the program after the "=", e.g. int i =[default]. Any spaces etc. that need to appear after the equals should be in the attribute.
dimensions specifies that an array of the type should be defined. The number of elements in each dimension is specified separated by commas.
kind specifies the precision (kind) of int or real types, or the name of a derived type being defined.
modifiers lists definition modifiers that should appear after the type declaration. Possible values are: allocatable, pointer, optional etc. The dimension keyword should not appear here, rather the dimensions should be declared using dimensions.
name specifies the name to use when defining and referencing the variable within the program context. It should match the names of method parameters if auto-matching is used; otherwise the paramlist attribute of pre-requisite calls, or <mapping> tags for the method being tested, should be used.
type specifies a valid fortran type for the variable. If the variable is a derived type, the fortran type should be type or class and the name of the derived type should appear in kind.
ignore allows a variable in the call signature of the executable being unit tested to be ignored. This is useful for optional parameters for which the default values work well. If ignore="true", the parameter is left off the argument list; otherwise it is included.

`<assignment>` Tag

The <assignment> tag allows the values of variables to be changed between calls to pre-requisite/dependency methods and the actual executable being tested. The variable's value can be changed via direct assignment or a call to one of its methods if it is a fortran class. The following example would not be found in a real application, but shows all of the possible tags and attributes allowed inside of <assignment> tags.

  <assignment name="platTyp" value="identifier, id2, id3" allocate="true" constant="1" position="after">
    <conditionals>
      <if condition="LatDim==3" value="bulkconstant" />
      <elseif condition="LatDim==2" value="surfconstant" />
    </conditionals>
    <value identifier="bulkconstant" constant="'b'" />
    <value identifier="surfconstant" constant="'s'" />
    <value identifier="filevalue" folder="./tests" file="variable.in" repeats="true" 
           rename="othervar.in" />
    <value identifier="functionvalue" function="SIN(pi)" />
    <value identifier="embeddedmethod" embedded="embedded(params)" paramlist="a,b,c" 
           prereqs="true" />
    <value identifier="const_with_allocate" constant="1" allocate="size(arr, 1), 3" />
    <part identifier="arrpart" value="initfun" limits="1-mnseq_len" />
    <value identifier="wildfiles" folder="./tests" file="wild.*" suffix="$1" member="W" />
    <part identifier="wildcard" value="wildfiles" />
  </assignment>

As you can see in the example, the <assignment> tag only has three possible attributes:

name |required| the name of the variable that needs to have its value changed.
value if the variable will have its value set directly (i.e. not with a <conditional> block), set value to be the identifier of one of the nested <value> tags. You can leave this attribute off if you only want a variable to be allocated.
allocate is either "true" or "false" or the specific dimensions to allocate with. By default, a variable with modifier allocatable will be allocated by the framework as part of this <assignment>. If another subroutine or <prereq> will allocate the variable, set allocate="false" to prevent run-time reallocation errors. When the value is "true", no dimensions are passed to the allocate command; this works for derived-type variables with type pointer. Otherwise, it should be left out or specified explicitly.
constant specifies a constant value to assign to a variable. This is a shorthand for specifying a <value identifier="constant" constant="your_value" /> and setting this tag's value attribute to "constant".
position If a test specification includes assignments local to the test, then global <assignments> attached to the testing group are added either before/after the local assignments. This value specifies that position relative to the local ones.

The rest of the information needed to set the variable's value is in a <conditionals> block or a <value> or <part> tag. A <value> tag or <part> tag is always required when a value is assigned.

`<conditionals>` Block

This XML element is parent to <if>, <elseif> or <else> tags that specify logical tests to perform and the value to set if the test is true. <if> and <elseif> require a condition attribute to be set. All of them require a value attribute.

condition should be valid fortran code that would be inserted as part of an if or elseif statement. The value of this attribute is pasted directly into the *.f90 file; it is not checked for consistency/accuracy.
value should be set to the value of the identifier attribute from one of the <value> tags that appear in the same <assignment> group.

The order that the <if>, <elseif> and <else> tags appear inside of the <assignment> tag is the same order that they will be output in the *.f90 file.

`<value>` Tag

The value tag specifies how Fortpy should assign the value to the variable instance. The value can be changed by:

direct assignment to a constant; in this case use the constant attribute.
calling an embedded procedure in a derived type; for this case use embedded, paramlist and prereqs attributes.
reading the value in from an external file; use folder, file and rename attributes.
calling a function and assigning its return value to the variable; use the function attribute.
some combination of file, function and embedded assignments when using <part> tags or the ragged attribute.

All of the <value> tags have the following attributes in common:

identifier gives the value a unique name within its <assignment> block. Used by logical test tags or the main <assignment> tag to know how to assign the variable's value.
repeats can be either "true" or "false". When a test runs using constant-input mode, the call to the subroutine or function being unit-tested is repeated for each value in an input file. If repeats is set to "true", it means that this variable assignment should take place each time the main unit-testing method is run. Otherwise, the value is only set once at the start of the program.

The other attributes mentioned in the usage scenarios above are described now:

member specifies that the variable referenced in the <assignment> tag should have its member with this name set to the value of this <value> tag.
constant represents a constant value such as a string or number to set the variable to. This attribute's value must be valid fortran code as it gets pasted directly into the *.f90 program file.
embedded for a derived type (i.e. fortran class) specifies the method within the type that should be executed. If the method requires parameters, they should be explicitly noted in the paramlist attribute; otherwise, they are listed with the same names as coded call signature.
paramlist specifies the exact call signature to use when executing the embedded method. Useful for cases when a <global> variable has a different name but can be used as one of the parameters. If you are using the <part> tags for initializing elements in an array of derived types, then $i refers to the loop variable for the ith nested loop generated by the <part> tags (starting with 1). Thus, if I had a 1D array of derived types to initialize with values that were previously loaded from a file into some other variable, I could include varname(:,$1) in the paramlist value.
prereqs can be either "true" or "false". When "true", the embedded method's XML documentation is examined for testing declarations. If any are found that the embedded method depends on, they are run first in the correct order as if that method was being unit tested. WARNING when the pre-req chain is processed, if the names of the <global> variables or regular parameters are different, they are treated as different variables. Make sure your naming is consistent so that the correct variables are passed to functions.
folder is used when the variable's value should be set from the contents of a file. This attribute specifies the folder path relative to the code root where the file will be found. However, for autoclass mode, it represents the folder housing the variable's member data files. In that case, you can specify the case identifier the usual way (e.g. folder="../data/model/variable.{}").
file the name of the source file in folder to use for the variable's value. The file should contain only values. If the variable is a 2D array, the number of values in each row must be the same unless the ragged attribute is specified.
ragged specifies that the number of elements on each row of the file is different and that each row should be treated independently. In that case, fortpy creates a loop over rows in the file and produces the <value> assignment for each row value. The local variable holding the single row of data is called @file when it needs to be referenced in paramlist or function. In order for fortpy to declare the local, temporary variable, the attribute dtype is required whenever <part> tags or the ragged attribute is used.
dtype specifies the type of the data in a file for use with <part> tags or the ragged attribute. This allows fortpy to declare a local, re-usable, temporary variable for holding the file contents while in the loop.
kind specifies the kind for the dtype attribute specification. For example, if we have double-precision real values then dtype="real" and kind="dp".
filedim specifies the dimensionality of the data in a file for use with <part> or the ragged attribute. This is necessary because arrays of derived types have different dimensionality than that needed by the initializers using the file data. This attribute value is used in declaring the local, temporary variable the holds the file data during loop iterations.
rename is used to change the name of the file when it is copied to the execution directory.
function specifies the name and call list of a function to use for setting the value. The attribute's value must be valid Fortran code for the given context.
allocate specifies the dimensions that an allocatable variable should be initialized to before assigning its value to a constant or function. If allocate is not specified, fortpy assumes that the variable is ready to assign a value to.
suffix for wildcard filenames that get used in ragged and <part> assignments, this attribute overrides the default behavior of fortpy in generating the suffixes used for filenames. Normally, the loop variables are just period joined based on how many loops deep the value assignment is: for two nested loops you would get $1.$2. You can specify any combination of $i (period-joined). The files should then be named file.#1.#2.{case} for input (where #1 would be the value that loop variable $i has during its iterations).
autoclass specifies that the variable should get its value from an auto-class compatible folder. See the auto-class section below for details.
testsource specifies the module.executable.parameter name from another unit test. The output generated by that parameter will be used to set the folder and file attributes.

`<part>` Tag

Sometimes a developer may want to use assign a value to only certain dimensions or elements of an array. Fortpy automates that process with <part> tags. To use a <part> tag, just set the value attribute of the <assignment> tag to be the identifier of the <part>. When the driver file is generated, fortpy creates a loop for each of the <part> tags and puts the assignment code for the variables value inside the loop. The behavior of the loop and its limits can be controlled using the tags attributes.

For multi-dimensional arrays, part tags can be nested so that any dimension or element can have its value assigned individually using files, embedded methods or functions and based on the loop iteration variables. Let's review the attributes to clear up how it works.

<part identifier="outer" start="1" value="valuetag_id1, valuetag_id2" limits="start:stop" repeats="false" />

identifier is the name of the part tag that can be referenced by the <assignment> tag to have it take effect.
start specifies the starting index for the highest level <part> tag. For example, if I only want to iterate over columns to assign values, I would usually do variable(:,col)=.... To use only a single <part> tag, we choose start="2" to set the loop iteration over the second index. The index is 1-based.
value is a comma-separated list of identifiers of the <value> tags to generate inside the loop. Every <part> tag doesn't need to have a value attribute. If there are nested part tags and both tags have value specified, the first assignment would affect the variable as variable(first,:)=...; following that assignment, the next do loop would begin to iterate over the nested <part> specification. Its assignment would look like variable(first,second)=....
limits is a :-separated list of limits for the loop variable. Possible values are :b, a: or a:b where a and b can be constants or functions. If a is not specified, 1 is used. If b is not specified, size(variable, loop_depth) is used, where loop_depth is the depth of the nested loop starting with start. A comma-separated list of specific, integer indices can be hard-coded. In that case, no loop is created, rather the assignments are just listed explicitly for each value in the list.
repeats specifies whether the assignment via the loops is once-off or should be repeated before every execution of the unit-tested method (when running in simple mode with <input> tags in the test).

As mentioned in the <value> tag documentation, if you need to refer to a loop variable when specifying limits or other value assignments, $i will be replaced by the loop variable name by fortpy. Thus a possible limit could be limits="1:$1". Even if start is specified, $1 always refers to the outmost, nested <part> tag etc.

`<mapping>` Tag

Depending on how the method's parameters are defined, global variable names that are auto-generated by a pre-requisite may be different than the calling signature of the method being tested. In this case, you can specify a mapping to tell the framework that a parameter with name "A" should actually be specified by a variable with name "B".

<mapping name="latTyp" target="platTyp" />

name the name of the parameter of the method being unit tested.
target the name of the variable in the calling program's context.

`<test>` Tag

The <test> tag specifies which input to use when running the executable and which output files from the execution need to be tested for conformity. In cases where multiple tests have identical setup, cases can be specified to ease implementation. You can specify multiple <test> tags in a single testing group.

     <test identifier="original" runchecks="true" execute="true"
           description="Original enumerations tests of basic structures"
           cases="001, 002, 003, 004, 005, 006" timed="true" 
           inputs="struct_enum" globals="^seed; verbose" assignments="^seed">
	<input folder="./tests" file="struct_enum.in.{}" rename="struct_enum.in" />
	<output folder="./tests" file="struct_enum.out.{}" template="struct_enum.out.xml"
		identifier="struct_enum.out" />
	<target name="./struct_enum.out" compareto="struct_enum.out" />
     </test>

The following attributes describe a <test>:

identifier uniquely describes the test within the context of the module.method being tested. This identifier is used internally and as the name of the executable that gets generated.
runchecks specifies whether to check the output of the unit testing executable against the model output specified by <target> tags. If "false", fortpy won't generate errors for missing <target> and <output> tags. This is a quick way to generate drivers for specific use cases.
execute specifies whether to run the executable that gets built by the fortpy framework. If false, the executable will be compiled, but not run.
description is included in the <summary> tag of the *.f90 PROGRAM that gets generated.
cases a list of identifiers to specify multiple executions of this test. The file attributes of both <input> and <output> tags are formatted using each case value in turn for multiple executions. A separate execution folder gets created for each case in the main tests folder. You can specify multiple test cases using a range in square brackets. For example, cr[0-3],00[1-4] is equivalent to cr0,cr1,cr2,cr3,001,002,003,004. You can mix ranges and single values and also use text after the range, as in standard.a[0-25]n.
timed can be either "true" or "false". If "true", the execution time of the main method being unit tested is accumulated for each time it is run and reported along with the accuracy at the end of the test.
inputs a ;-separated list of <input> tag identifiers from the <globals> tag in the XML file to copy as children of the <test> tag. Since this attribute only looks at <globals> tags of type <input>, the global: prefix is not required for the identifiers. If the identifier starts with ^, then the tag is inserted at the start of the containing tag's sub-element list instead of being appended.
globals as for inputs, a ;-separated list of <global> tag identifiers from the <globals> tag in the XML file to copy as children. See also the note on that attribute about ^.
assignments as for inputs, a ;-separated list of <assignment> tag identifiers from the <globals> tag in the XML file to copy as children. See also the note on that attribute about ^.
mappings as for inputs, a ;-separated list of <mapping> tag identifiers from the <globals> tag in the XML file to copy as children. See also the note on that attribute about ^.

`<input>` Tag

A test can have multiple input tags. Each input tag specifies a file that needs to be copied to the execution directory before the test method can run. An optional <line> tag can be placed in the <input> tag to have it run in constant-input mode. When a test runs in constant-input mode, the main test to be executed is repeated for each line in the input file. The values of the <global> variables and regular parameters are updated using the information in the line template, which follows the same format as used by the input/output templates.

IMPORTANT: if you run a test in constant input mode:

When multiple <input> tags in the same test have <line> templates, all the files must have the same number of lines or the test will not run.
If a <prereq> also needs to be run for each line in the <input> file, it must have the repeats="true" attribute or it will only get run once.
If multiple test cases need to be run in constant input mode, you will need to create multiple <test> tags; you can't just use the cases attribute on the <test> tag.

The following attributes have meaning on an <input> tag:

folder is the path relative to the code root to the folder that houses the input file.
file is the name of the file inside of folder. If you are not running in constant-input mode, you can use '{}' to represent the case identifier from the cases attribute of the <test> tag for running multiple tests of the same type.
rename is a new name to give the file when it is copied over to the execution folder. This attribute is strongly encouraged when the test is running in cases-mode.
testsource specifies the module.executable.parameter name from another unit test. The output generated by that parameter will be used to set the folder and file attributes.

`<output>` Tag

In order to validate the results of a unit test, the framework needs model output files to validate against. You can specify multiple model outputs using <output> tags inside of the <test> tag. The presence of an <output> tag alone will not run any comparisons. Rather, <output> tags have identifier attributes that are referenced by other entities in the <test> group (e.g. <target>) which specify what to test and how to run the test. <output> only specifies the location of the data to use for the comparisons.

These attributes have meaning on an <output> tag:

identifier |required| is a unique identifier for this output data in the <test> group.
folder is the path relative to the code root to the folder that houses the output file. However, for autoclass mode, it represents the folder housing the variable's member data files. In that case, you can specify the case identifier the usual way (e.g. folder="../data/model/variable.{}").
file is the name of the file inside of folder. You can use '{}' to represent the case identifier from the cases attribute of the <test> tag for running multiple tests of the same type.
template is the name of an XML file with template information to use when comparing the outputs. This allows comparison of output files created by different versions of the code. The actual file can either be:
- a built-in template (such as integer.xml or float.xml) that ships with Fortpy.
- present in a folder called "templates" that lives in the code root.
- be in a folder specified in the script argument to ./runtests.py that executes the tests.
mode specifies the comparison mode/strictness for the output file comparisons. If not specified, it has the value "default".
value if the comparison is between a variable and a constant value, specify that constant value here. The value can be any valid python code and will be interpreted using eval() before its value is compared with the variable.
tolerance for finite precision calculations and comparisons, as long as the ratio of output to model values are within this percent, the test still passes. E.g. if tolerance=0.8, then a ratio of 0.92 is considered equivalent to 1.
autoclass for variables that have user-derived types, fortpy can save the state of the variable automatically using autoclass. For <output> it specifies that the folder specified in folder contains model files for all of the members of the derived types. Fortpy compares each of the files one-by-one and averages the percent error.
actolerance specifies the minimum common match that any individual file can have when running in auto-class mode. Any files comparisons that fall below this threshold are considered failures.

`<target>` Tag

A test will not perform any comparisons unless a <target> tag is specified. The <target> tells Fortpy what variable values or files need to be compared to model outputs as part of the test. In other words, a <target> links a variable value or output file generated by the unit test executable to an <output> tag.

name |required| is the name of a <global> variable (or regular parameter) or a file to compare to some <output> tag. To specify a file, the first characters of name should be "./"; otherwise it is assumed to be a variable.
compareto |required| is the identifier of the <output> tag that has data for the comparison.
varfile if a variable name is specified as the target, Fortpy automatically saves its value to a file for comparison after the executable terminates. This attribute controls the name of that file. If unspecified, the default name is name.fortpy where "name" is the name of the variable.
generator if the variable specified by name needs to be saved in a special, non-standard way, this attribute specifies the name of a subroutine to call. Unless varfile is set, the subroutine is called with only a single parameter, the name of the variable to save; it should create the file etc. internally. If varfile is specified, it is passed in as a second parameter.
when is one of ('begin', 'each', 'end') and specifies how often the target variable should be saved. 'Begin' saves the value of the variable before the main method has run. 'Each' saves the value of the variable directly after each call to the main method. 'End' saves the variable value once right before the executable terminates. By "saves the variable" we mean that either generator gets called or the generic Fortpy saving process is called to save the variable to varfile.
member specifies the name of a variable declared as a member of a derived type. That variable's value will be saved to the varfile and used for comparison.
autoclass tells fortpy to save the state of a variable with user-derived type automatically. See the next section on auto-class mode.

Auto-class Mode

For complicated user-derived types (and arrays of those types) an inordinate number of <target> and <outcome> tags would need to be specified to handle each member, especially if those members are also arrays of user-derived types. This complexity also applies to initializing the value of a user-derived type with complex structure in its members. Auto-class mode saves/loads the state of a user-derived type variable to/from a folder (having the name of the variable by default). Each member of the type has its value saved in a separate file.

This is accomplished by iterating recursively through each member. If the member's type is simple (i.e. a built-in type), then it is saved to a single file using pysave (no matter its dimension). If it is a user-derived type, then each of its members is examined. For arrays of user-derived type variables, do loops are constructed automatically and a file is created for each element of the array, with its position in the array included in the file name.

Example (see issue #59 for a more complete explanation): suppose I have a 2D array variable type(custom) :: var2d(:,:) of a user-type custom with member nl; then, _-1.2-nl would represent var2d(1,2)%nl.

File Naming Convention

In order for the auto-class to be auto- the files need to be named in a special way. Here are the conventions:

Each file name must start with _, which represents the variable being auto-classed from folder.
If any user-derived type variable is an array of custom types, the name of the variable will be followed by a period-separated list describing the position of the data in the parent array.
If the user-derived type's member is a built-in fortran type, it's lower case name is just appended.

The tree of variable names and array slices are - separated. See the ./tests/model/autoclass/c.* folders for a specific example. The accompanying fortran user types are in ./tests/unittests/autoclass.f90 and the XML specification showing the auto-class feature is at ./tests/unittests/autoclass.xml.

`<prereq>` Tag

Often, before a method can be tested, many other methods need to be called to read input files, pre-process data etc. These methods can be specified in a <prereq> tag. IMPORTANT! the framework assumes that you have put the <prereq> and <assignment> tags in the order that they need to be executed.

When a pre-req is specified, the framework will examine its testing groups to determine if it also has any pre-reqs specified. If it does, all of them are chained in the correct order so that they execute correctly. This also means that any auto-parameters specified by a pre-reqs testing group will be initialized automatically. If a parameter with the same name exists in multiple methods being executed, the framework assumes that they are the same variable. If this is not correct, the chaining should be prevented using the terminate attribute on the pre-req. In that case, all pre-reqs that are required should be specified explicitly and paramlist should be used to specify which variables go in which places in the calling signature.

  <prereq method="io_utils.read_input" terminate="true"
          paramlist="title, LatDim, parLV, nDFull" />

method a module.subroutine that should be executed before the method being tested.
paramlist a comma-separated list of variables in the calling program's context in the order that they should be passed to the subroutine call.
terminate if "true", the framework will not follow the dependency chain of the subroutine to determine if it has any pre-req or variable specifications for it to run correctly.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Testing Group Tags and Attributes

FORTPY: Testing Groups

`<group>` Tag Attributes

`<global>` Tag

`<assignment>` Tag

`<conditionals>` Block

`<value>` Tag

`<part>` Tag

`<mapping>` Tag

`<test>` Tag

`<input>` Tag

`<output>` Tag

`<target>` Tag

Auto-class Mode

File Naming Convention

`<prereq>` Tag

Clone this wiki locally

Testing Group Tags and Attributes

FORTPY: Testing Groups

<group> Tag Attributes

<global> Tag

<assignment> Tag

<conditionals> Block

<value> Tag

<part> Tag

<mapping> Tag

<test> Tag

<input> Tag

<output> Tag

<target> Tag

Auto-class Mode

File Naming Convention

<prereq> Tag

Clone this wiki locally

`<group>` Tag Attributes

`<global>` Tag

`<assignment>` Tag

`<conditionals>` Block

`<value>` Tag

`<part>` Tag

`<mapping>` Tag

`<test>` Tag

`<input>` Tag

`<output>` Tag

`<target>` Tag

`<prereq>` Tag