FX variables saga

[valeriupredoi]

The fx variables are and have been a thorn in the foot of ESMValTool and now it's gone to esmvalcore. I will try to summarize the standing problems that the fx files/variables pose to us pointing to the already existing or planned solutions.

Fx file discovery
fx files have a slightly different standard DRS path than regular CMIP5/6 variables files and up until noew we have used bespoke structures for fx input dirs and fx file formats in config-developer.yml. We have realized that by slightly altering the CMIP dir and file structures we can use common structures for both CMIP variables and fx variables: #21 and #22 have this implemented

Use fx variables as part of the standard variables list
#21 and #22 now allow the user to specify any fx variable as part of the diagnostics/variables list

Coupling fx variables and standard variables in the preprocessor
This is tricky and has different aspects to it. For short-hand descriptive purposes I will say that a certain standard variable VAR needs a certain set of fx variables FX to be used during different steps of preprocessing. The details below explain first #21 (which is a simplified approach) and then #22 (which is more complex).

PR #21

• for VAR to use FX during preprocessing (and have the files listed in metadata.yml for diag purposes) one has to add eg fx_files: [{short_name: areacello, mip: Ofx}] as argument for VAR (this is an extension of the previous implementation eg fx_files: [areacello] to account for fx mip variability for CMIP6; in CMIP5 all fx variables had mip fx; no need to add the mip for CMIP5 data, just use fx_files: [{short_name: areacello}]);
• apart from the above, all aspects of previous implementation stand (no CMOR checks to FX, no preprocessing for FX, unless decalred as a separate variable but then VAR will not know about it)

PR #22

• this PR introduces the first practical coupling between VAR and FX: as above, by specifying an FX eg fx_files: [{short_name: areacello, mip: Ofx}] as argument to VAR, the implementation will do the following:
  • assemble the FX dictionary as a sub-variable of VAR: find the FX data files but not use them as VAR[fx_files] files, instead assigning a priority task before the VAR task to: use the FX data files and run a default preprocessor with them (load, fix, CMOR check and fix, save) and populate the VAR[fx_files] dictionary with the already default-preprocessed FX files (ancestor task to VAR task via _get_fx_tasks); these files live in the diagnostic/preproc/VAR directory and will be used just as usual for derive, mask and area or volume statistics as assigned by _recipe.py/_update_fx_settings
  • there is good potential to allow _get_fx_tasks to set up a more complex preprocessor for FX, since stuff like area and volume statistics need a certain level of preprocessing (if VAR needs that); note that derive and mask (landsea or landseaice) need the FX in its native form;

Standing issues

• The need to eliminate specifying VAR arguments eg fx_files: [{short_name: areacello, mip: Ofx}] if the FX is present in the variables list: this is very tricky since the coupling VAR-FX is impossible to realize since VAR and FX are independent variables that have independent threaded tasks;
• The need to preprocess FX to a certain level as VAR (but not using all the preprocessing done to VAR): this can be done by using a custom preprocessor for FX and passing it as argument to _get_fx_tasks

Any other ideas? 🍺

[bouweandela]

The need to preprocess FX to a the same level as VAR

This can probably be done by adding both normal variable and fx variable files as input files in the same preprocessor task, similar to how it is done for derived variables that require more than one input variable. This would probably require some changes to certain preprocessor functions, e.g. if they do things that do not make sense for an fx variable, they should ignore those.

[valeriupredoi]

#170