Earth System Data Cube (ESDC) cmorizer

[bsolino]

Description

• Closes Earth System Data Cube (ESDC) cmorizer #2798
• Link to documentation: https://esd.copernicus.org/articles/11/201/2020/

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🛠 This pull request has a descriptive title
• 🛠 Code is written according to the code quality guidelines
• 🛠 Documentation is available
• 🛠 Tests run successfully
• 🛠 The list of authors is up to date
• 🛠 Any changed dependencies have been added or removed correctly
• 🛠 All checks below this pull request were successful

New or updated data reformatting script

• 🛠 Documentation is available
• 🛠 The dataset has been added to the CMOR check recipe
• 🧪 Numbers and units of the data look physically meaningful

---

To help with the number of pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[bsolino]

The implementation is ready and can be reviewed already. There are a few details missing that don't affect the code:
- I classified the dataset as Tier 2, as it is publicly available without login. However, it includes data from Tier 3 datasets, like ERA5.
- I am unsure about the appropriate name of the dataset, I will consult with the providers in case they prefer a different name

[remi-kazeroni]

Thanks for starting this work @bsolino! I did some testing of the CMORizer without checking the code in depth as you wanted some early feedback on this PR. The CMORizer seems to work fine, including reading data from the cloud store. I think the handling of metadata is not fully compete, see suggestion. Also, you may want to check with the provider which original metadata to keep.

The main issue I see is that this cmorizer only starts looking for cloud data if a ~/RAWOBS/Tier2/ESDC directory already exists (even an empty directory is enough). Otherwise, the Tool crashes because no data are found. I'm not sure this is something to address in this cmorizer script or more broadly in cmorizer.py.

[bsolino]

The main issue I see is that this cmorizer only starts looking for cloud data if a ~/RAWOBS/Tier2/ESDC directory already exists (even an empty directory is enough). Otherwise, the Tool crashes because no data are found. I'm not sure this is something to address in this cmorizer script or more broadly in cmorizer.py.

Good catch! I've done some testing, but there is nothing I can do from within the cmorizer. The detection happens in the function def _get_dataset_tier(self, dataset) from cmorizer.py. The only solution I can think to that issue is to have an entirely different detection system, perhaps from the documentation data (datasets.yml or cmor_config/DATASET.yml). However, it seems that will require a bigger discussion, as I don't know why the tool takes the tier from the folder and not the folder from the tier.

[remi-kazeroni]

Thanks for the changes @bsolino! Everything looks good to me and works as expected 👍

[remi-kazeroni]

@valeriupredoi, could you please take a look at this PR whenever you have time for that? And maybe advise @bsolino on how the best way to add dependencies to the environment*.yml and setup.py files? Cheers 🍻

[valeriupredoi]

just a couple optional wiggles from me, looking good, good work, guys! I'll have a look at the environment files now 👍

[valeriupredoi]

setup.py looks fine apart from that aiohttp should be at the top of the list, but for the two environment yml files I'd put zarr and aiohttp in the main list of dependencies (since they are actively used by the main code), so zarr would be at the bottom of the dependencies list and aiohttp at the top after pip and python. Where is it used (aiohttp) anyway, I've not seen it imported anywhere? If it's not, please add it only when it will be, last thing we want is yet another dependency 😁

[valeriupredoi]

just temp holding this for the unused aiohttp dependency

[bsolino]

Where is it used (aiohttp) anyway, I've not seen it imported anywhere? If it's not, please add it only when it will be, last thing we want is yet another dependency 😁

It is not used directly by us (neither is zarr, for that matter 😉). It is a necessary module to open http filesystems, without it the program terminates when trying to open the zarr datacube in the cloud.

ImportError: HTTPFileSystem requires "requests" and "aiohttp" to be installed

[valeriupredoi]

Where is it used (aiohttp) anyway, I've not seen it imported anywhere? If it's not, please add it only when it will be, last thing we want is yet another dependency grin

It is not used directly by us (neither is zarr, for that matter wink). It is a necessary module to open http filesystems, without it the program terminates when trying to open the zarr datacube in the cloud.

ImportError: HTTPFileSystem requires "requests" and "aiohttp" to be installed

aha gotcha! zarr is used via the xarray engine, and that too should be a dependency of xarray only it isn't (had this convo with the kerchunk devs that want to do deps like Xarray's - optional depending what engine/module one needs - never agreed with this but it's a thing). Cheers for explaining it, mate 👍

[valeriupredoi]

all nice and green and am happy too, awesome work @bsolino 🍺 Cheers to @remi-kazeroni for testing as well!