Example pipeline for SWOT-Xover #14
Comments
|
Can you provide more details about the input files? How big are they? What URLs will we use to download them? |
|
I was hoping that each modelling group could upload their zarrified data to the Wasabi cloud storage... |
Then this is not a Pangeo Forge pipeline. The point of Pangeo Forge is to automatically put together the Zarr in the cloud. What you propose is fine--it's just not part of Pangeo Forge. Let's leave this open for now as we figure out the best path forward. |
I think we're going to try the Pangeo Forge pipeline for the eNATL60 data. Pending on how this goes, we may recommend other modelling centers to follow the pipeline. |
|
Great! To move forward, we need some more details about exactly where to find the data and how it is formatted. Please edit your original issue to conform to the template (https://github.com/pangeo-forge/staged-recipes/issues). |
|
Yes, I'm still working on extracting the cross-over regions (which surprisingly takes time dealing with massive netCDF files) but I will update the details as soon as I get this hashed out. |
If only there were a better format! 🤣 😉 |
|
This is getting a bit ahead of ourselves but in the case we would ask the modelling groups to provide their data via ftp or opendap links for the pangeo-forge pipeline, would the "computation" costs to upload them to the cloud come out from the payments we'll be making to 2i2c? I'm only asking because I think it would be best if we could reduce the amount of hassle each modelling group goes through. The idea I had in mind was to develop the pipeline on the SWOT-AdAC Jupyterhub. |
|
@rabernat I added a bit more detail in the |
Can you provide an actual working FTP link to one of the datasets?
No, they will be supported by pangeo forge and our NSF grant. 2i2c is for the jupyterhub.
This will require pangeo-forge/pangeo-forge-recipes#49, a feature that is not yet implemented. We are working on it. |
Sorry for the lagged response. Here is a working link: |
|
Here is another working ftp link for lNALT60: https://data.geomar.de/downloads/20.500.12085/0e95d316-f1ba-47e3-b667-fc800afafe22/data/ |
|
The eNATL60 regional outputs for regions 1-3 are now all available here: https://ige-meom-opendap.univ-grenoble-alpes.fr/thredds/catalog/meomopendap/extract/SWOT-Adac/catalog.html |
|
FYI, that server is giving HTTP certificate errors. Would it be possible to get this fixed? |
@auraoupa Do you know why this is happening...? |
|
Yes it is a know issue of our opendap (expired certificate), we get by it by adding --no-check-certificate to our wget commands, it would be --insecure for curl (did not try it), but it maybe more efficient (and cleaner) to have it fixed ... I'll try to make it happen ! |
|
We need to get the files via fsspec and unfortunately I don't (yet) know how to work around the certificate error...but there must be a way! I'll try to dig deeper on my end too. |
|
Now it looks like the server https://ige-meom-opendap.univ-grenoble-alpes.fr/ is down completely? This is making it hard to develop the recipe. |
|
Sorry about that, it should be ok now. About the certificate, the University should be fixing it soon they say ! I keep you posted |
|
The certificate is now valid, I hope it helps for the development of the recipe ! |
|
Success! url = 'https://ige-meom-opendap.univ-grenoble-alpes.fr/thredds/fileServer/meomopendap/extract/SWOT-Adac/Surface/eNATL60/Region03-surface-hourly_2010-04.nc'
with fsspec.open(url) as fp:
ds = xr.open_dataset(fp)
display(ds) |
Sorry, I missed this. This is great news! Could you let us know what the status is regarding the data storage on OSN @rabernat ?? |
|
The status is that I'm still working on it. I hope to be able to start ingesting data soon (next week). I'm deeply sorry for the delays and I thank you for your patience. |
|
thanks for all your work with this @rabernat ! |
|
I started a PR #24 for the recipe. |
|
@rabernat @cisaacstern I've started analyzing the SWOT-AdAC data (#24 #26 #29 ) on a Google Cloud based Jupyterhub but does the OSN storage also support storing of analysis data? |
No, we cannot provide write access to OSN. Can you explain more about the use case you have in mind? How much data do you imagine needing to write? Does it need to be shared across users? For writing data, you have a few options:
|
|
I believe all of the surface datasets are now on OSN. Returning to this main thread to provide an high-level "flyover" of how it's organized. Note that below, import s3fs
endpoint_url = 'https://ncsa.osn.xsede.org'
fs_osn = s3fs.S3FileSystem(anon=True, client_kwargs={'endpoint_url': endpoint_url},)
swot = "Pangeo/pangeo-forge/swot_adac"💺 Fasten your seatbelt, this will be a long one! INALT60 #26fs_osn.ls(f"{swot}/INALT60")We currently a single zarr store for each surface dataset. The time dimension for these data is non-contiguous as seen in the recipe here. If it's useful, I can separate each of these surface datasets into separate seasonal stores, as demonstrated in the other recipes below. GIGATL #27fs_osn.ls(f"{swot}/GIGATL")@roxyboy, unless you need it for something, I will delete For each region's surface data, there are both fs_osn.ls(f"{swot}/GIGATL/Region01/surf")fs_osn.ls(f"{swot}/GIGATL/Region02/surf")The HYCOM50 #29fs_osn.ls(f"{swot}/HYCOM50")For each region defined in the recipe, there are both fs_osn.ls(f"{swot}/HYCOM50/Region01_GS/surf")fs_osn.ls(f"{swot}/HYCOM50/Region02_GE/surf")fs_osn.ls(f"{swot}/HYCOM50/Region03_MD/surf")@roxyboy, `surf_01.zarr`, `surf_02.zarr`, and `surf_03.zarr` are the earlier drafts where non-contiguous data is concatenated together. Do you have any use for them now that the seasonal stores are up? If not, I'll delete. eNATL60 #24fs_osn.ls(f"{swot}/eNATL60")For each of the regions, fs_osn.ls(f"{swot}/eNATL60/Region01/surface_hourly")fs_osn.ls(f"{swot}/eNATL60/Region02/surface_hourly")fs_osn.ls(f"{swot}/eNATL60/Region03/surface_hourly")Next steps@roxyboy, please let me know if you run into any issues with any of the above. Also, what should we work on next? Adding the interior data? |
|
This is great! Thanks @cisaacstern .
The time metadata for INALT60 is in Gregorian so I think it's fine that we keep it as it currently is because it's much easier to parse out the seasons.
Yes, please delete
Please feel free to delete
Yes, fluxing the interior data to the cloud would be greatly appreciated :) |
|
@cisaacstern Model outputs from FESOM are in the making but can the recipes handle netcdf4 files compressed with tar? |
It would be ideal if we could avoid tarring inputs. But if this is unavoidable, we will find a way to deal with it. |
|
@roxyboy - Today @cisaacstern and I met to discuss this. It will introduce significant complexity in Pangeo Forge to handle the tarred files. We are not sure this effort is worth it since there is an easy workaround--can we just ask the data provider to un-tar the files before putting them online? That is a reasonable request, no? |
Yes, I've asked them to untar it. Will be making a PR for FESOM soon. |
|
We've (@lesommer and I) decided that hosting regional extracts from LLC4320 on OSN is probably better than us pulling the data from ECCO portal for analysis. Dimitris asked if he could directly push the data himself from where LLC4320 sits after the extraction but is this possible? Otherwise, I can ask him to (temporarily) put it on an ftp server. |
|
@roxyboy, as far as I'm aware, we're not able to provide write access to OSN. If you point me to the files on a temporary ftp server, however, I can write them to the |
|
Brief progress report below. Simulations with no emojis mean that we haven't started a recipe yet.
Edit (July 2): As mentioned in #29 (comment), updated table to reflect that HYCOM50 int data is online. Edit (July 3): Updated table to reflect that GIGATL int data is online; xref #27 (comment). Edit (July 19): FESOM surface data added to project catalog: pangeo-data/swot_adac_ogcms#2 Edit (July 20): eNATL60 surface data added to catalog: pangeo-data/swot_adac_ogcms#3 |
|
@rabernat @cisaacstern What do the Pangeo folks think about hosting the global 1/25 HYCOM surface data of u, v, and SSH developed by Brian Arbic's group on OSN? The storage will likely be on the order of 8Tb. The idea is that it'll benefit the SWOT-AdAC community by having global access to both LLC4320 and HYCOM25. As an example, we/I can work on hosting a Jupyter notebook showing the transition scales on the Pangeo gallery. |
|
YES to global HYCOM on OSN. |
|
Standing by to assist with the recipe once @roxyboy and/or @lesommer points us to the source files. This will be a good test for our new Google Cloud Bakery once it comes online. |
Source Dataset
SWOT-Xover is a subset of a few basin-scale model outputs with the resolution of ~1/50° surface hourly and interior daily data. The subsets will cover the cross-over regions of the SWOT fast-sampling phase.
Transformation / Alignment / Merging
Files should be concatenated along the time dimension.
Output Dataset
The zarrification of data should be automated via the pangeo-forge pipeline following the pangeo-forge recipe. In order to facilitate the automation, we would ask each modelling group to have the outputs in netcdf4 format and make it available via an ftp server.
A single monthly file of daily-averaged 3D data of
u, v, w, T & Sin one region is ~30Gb. With the four regions, six months and five models, this would sum up to ~3.6Tb in total on the cloud storage. The chunks of the zarr dataset will be on the order of{'time':30, 'z':5, 'y':100, 'x':100}.For the surface, a single daily file of hourly averaged data of
SST, SSS, SSH, wind stress & buoyancy fluxesin one region is ~380Mb. With the regions, months and models, this sums up to ~45Gb. The chunks of the zarr dataset will be on the order of{'time':100, 'y':100, 'x':100}The text was updated successfully, but these errors were encountered: