Development cmip6fx conventions with cmor fixes to fx vars

[valeriupredoi]

Related issue #109
ported ESMValGroup/ESMValTool#1037 to ESMValCore/version2_development

[zklaus]

This contains a lot of stuff about different preprocessors that has nothing to do with fx vars. Did something in the porting go wrong or am I missing something?

[valeriupredoi]

Yeah, that's the stuff the branch version2_development has ahead of development, merging that in development will rid us of all those extra stuff Dr Valeriu Predoi. Computational scientist NCAS-CMS University of Reading Department of Meteorology Reading RG6 6BB United Kingdom

…

On Tue, 11 Jun 2019, 13:48 Klaus Zimmermann, ***@***.***> wrote: This contains a lot of stuff about different preprocessors that has nothing to do with fx vars. Did something in the porting go wrong or am I missing something? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#22?email_source=notifications&email_token=AG5EFI5CLRGFXYSVBEVUH7LPZ6NI7A5CNFSM4HWUVGNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXM745Y#issuecomment-500825719>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AG5EFI74DMN5Z66TZX6T7ZLPZ6NI7ANCNFSM4HWUVGNA> .

[zklaus]

Ok, I will wait with the review then until this is done 'cause at the moment it is a bit difficult to see what should actually be commented upon.

[valeriupredoi]

Sure thing man, I am away until Friday but @mattiarighi or @bouweandela can update development with version2_development latest 🍺 Dr Valeriu Predoi. Computational scientist NCAS-CMS University of Reading Department of Meteorology Reading RG6 6BB United Kingdom

…

On Tue, 11 Jun 2019, 13:53 Klaus Zimmermann, ***@***.***> wrote: Ok, I will wait with the review then until this is done 'cause at the moment it is a bit difficult to see what should actually be commented upon. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#22?email_source=notifications&email_token=AG5EFI67Q5HFIKRD35O3LJDPZ6N4JA5CNFSM4HWUVGNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXNALIQ#issuecomment-500827554>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AG5EFI6UPCXVSYUAD63RNU3PZ6N4JANCNFSM4HWUVGNA> .

[bouweandela]

Can you please merge the development branch into this branch, so we can see what's new?

[valeriupredoi]

Can you please merge the development branch into this branch, so we can see what's new?

le done 🍺

[valeriupredoi]

@ledm @zklaus @bouweandela I have had another look at this PR and I think it is very important to have it examined, suggestions plugged in and merged on an asap level: this is helping with a bunch of rather important standing issues. Here is an example for the recipe bit:

    variables:
      tos:
        preprocessor: pp_rad
        mip: Omon
        grid: gn
        fx_files: [{short_name: areacello, mip: Ofx}]
      areacello:
        preprocessor: pp_rad
        grid: gn
        mip: Ofx

the fx_files from tos are found, priority-based (preprocessed before tos) CMOR-checked and fixed and then fed to the preprocessor functions that need them (derive, mask and statistics); they are also included in the fx_files dictionary in metadata.yml for diag purposes. NOTE that the preprocessing of fx_files variables is limited to a basic (default) preprocessor because the derive and mask preproc functions need them in their basic form (and not preprocessed); only the area and volume statistics functions would need them preprocessed to various levels.

Also, part of this PR allows for specifying the fx variable like a regular variable, together with any sort of preprocessor for it. This can be used later on in the diagnostic.

NOTE that due to the internal structure of the current workflow, inter-using stand-alone variables with other variables is an absolute nightmare (the parallel nature of running each variable task makes things very tricky). Unless we seriously change the framework, the coupling var-fxvar (or for the matter, var-var) in the preprocessor can not be done unless the fxvar(fx_files) is a subsection of the variable.

[ledm]

only the area and volume statistics functions would need them preprocessed to various levels.

1. Any preprocessor that includes regridding at an early stage would likely need to regrid the area/volume as well.
2. Aside from regridding, we can solve some of this problem by changing the extract_region preprocessor from a slice to a mask. (NB It already applies mask if the grid is irregular, it's only regular grids that get the slice!)

[zklaus]

@valeriupredoi, nice to see this moving forward!

I can't speak for the others, but what makes the review difficult for me is that it is really unclear what this PR is trying to achieve, what other PRs on the topic are around, and how they relate to each other.
It certainly is difficult to read the history in this case.

How about this: Let's make one issue for fx stuff (or designate an existing one) and collect links to all the bits and pieces there. Then we can clarify what does what.

What do you think?

[valeriupredoi]

@zklaus sounds like a very good idea! This PR is too large to be reviewed just using the (bits of) information here. I'll create the issue right now, and ask you guys to contribute 🍺

[valeriupredoi]

#109

[valeriupredoi]

superseded for now by #170