Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS NOAA WHOI #221

Merged
merged 7 commits into from
Dec 13, 2022
Merged

AWS NOAA WHOI #221

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
a279651
add file pattern
kathrynberger Nov 9, 2022
eb843a9
replace with ref_hdf_zarr recipe and add meta yaml
kathrynberger Nov 16, 2022
c1af95c
additions to meta file
kathrynberger Nov 16, 2022
9d21dcf
cleaned up recipe
kathrynberger Nov 16, 2022
7f85f66
Update recipes/aws-noaa-whoi/meta.yaml
kathrynberger Nov 17, 2022
d87c7ea
revise with deterministic file pattern
kathrynberger Nov 17, 2022
166954e
add s3 prefix and allow for future years
kathrynberger Nov 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions recipes/aws-noaa-whoi/meta.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
title: 'AWS NOAA WHOI SST'
description: 'Analysis-ready datasets derived from AWS NOAA WHOI NetCDF'
pangeo_forge_version: '0.9.2'
pangeo_notebook_version: '2021.07.17'
recipes:
- id: aws-noaa-sea-surface-temp-whoi
object: 'recipe:recipe'
provenance:
providers:
- name: 'AWS NOAA Oceanic CDR'
description: 'Registry of Open Data on AWS National Oceanographic & Atmospheric Administration National Centers for Environmental Information'
roles:
- producer
- licensor
url: s3://noaa-cdr-sea-surface-temp-whoi-pds/
license: 'Open Data'
maintainers:
- name: 'Kathryn Berger'
orcid: '0000-0001-9731-6519'
github: kathrynberger
bakery:
id: 'pangeo-ldeo-nsf-earthcube'
20 changes: 20 additions & 0 deletions recipes/aws-noaa-whoi/recipe.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
import s3fs

from pangeo_forge_recipes.patterns import pattern_from_file_sequence
from pangeo_forge_recipes.recipes.reference_hdf_zarr import HDFReferenceRecipe

url_base = 's3://noaa-cdr-sea-surface-temp-whoi-pds/data/'

years = range(1988, 2022)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we might want to handle years differently, given that when data is available in 2023, this won't find the 2023 data.

years could instead be defined like so?

Suggested change
years = range(1988, 2022)
years_folders = fs.ls(join(url_base))
years = list(map(lambda x: os.path.basename(x), years_folders))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good suggestion @rbavery just tested this to verify, great way to consider ingesting future years. I'll add this to the revisions

file_list = []

fs = s3fs.S3FileSystem(anon=True)

for year in years:
file_list += sorted(
filter(lambda x: x.endswith('.nc'), fs.ls(url_base + str(year), detail=False))
)

pattern = pattern_from_file_sequence(file_list, 'time', nitems_per_file=1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My guess is that the files in file_list do not start with s3://, so that fsspec is looking for local files instead.

Copy link
Contributor

@rbavery rbavery Nov 18, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm working on a separate recipe and found this is the issue. adding a map to add the missing s3:// solved it.

from os.path import join

fs = s3fs.S3FileSystem(anon=True)
is_nc = lambda x: x.endswith('.nc')
add_s3 = lambda x: "s3://" + x
for year in years:
    file_list += sorted(
        filter(is_nc, map(add_s3, fs.ls(join(url_base, str(year)), detail=False)))
    )

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @cisaacstern @rabernat and @rbavery for catching this, I see the error here and will revise with the suggestion above. 👍


recipe = HDFReferenceRecipe(pattern, netcdf_storage_options={'anon': True})