Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for USGS 3DEP (formerly NED) items and collections #81

Merged
merged 10 commits into from
Apr 6, 2021
Merged

Add support for USGS 3DEP (formerly NED) items and collections #81

merged 10 commits into from
Apr 6, 2021

Conversation

gadomski
Copy link
Member

3DEP is the 3D Elevation Program from USGS. This commit provides support for creating items and collections for 3DEP data hosted at ftp://rockyftp.cr.usgs.gov/vdelivery/Datasets/Staged/Elevation.

Also includes two linty commits relating to a cgls_lc100 dangling symlink and old version string.

@gadomski
Copy link
Member Author

Per discussion w/ @lossyrob we're going to treat #83 as a predecessor of this PR, so I've rebased onto feature/landsat-aster w/o any conflicts (yay).

@kylebarron
Copy link
Contributor

This commit provides support for creating items and collections for 3DEP data hosted at ftp://rockyftp.cr.usgs.gov/vdelivery/Datasets/Staged/Elevation.

Does it also work with the same data (I think) in the prd-tnm S3 bucket? Everything under StagedProducts/Elevation/

https://s3.console.aws.amazon.com/s3/buckets/prd-tnm?region=us-west-2&prefix=StagedProducts/Elevation/&showversions=false

@gadomski
Copy link
Member Author

gadomski commented Mar 31, 2021

Does it also work with the same data (I think) in the prd-tnm S3 bucket?

Yep, 4ff1abb 8b6837e switches to using the AWS_BASE (https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation) as the default. It's built so that AWS, the USGS FTP, and arbitrary other bases are hopefully interchangeable.

@rsignell-usgs
Copy link

rsignell-usgs commented Apr 1, 2021

@gadomski and @kylebarron , just discovered this, thanks to @TomAugspurger !
The USGS is eager to create STAC items that will work with stackstac and stac-vrt for this dataset, so just let us know what the next step should be on our end!

@gadomski
Copy link
Member Author

gadomski commented Apr 1, 2021

@rsignell-usgs right on!

create STAC items that will work with stackstac and stac-vrt for this dataset

stac-vrt has some reccommended attributes which we are including. I'm not very familiar with stackstac so am less sure what STAC attributes are required/recommended there, and wasn't able to find a quick reference w/ that info. @gjoseph92 would you be able to provide guidance on any optional attributes to include in our STAC objects to best support stackstac?

gadomski added 5 commits April 2, 2021 07:08
3DEP is the 3D Elevation Program from USGS. This commit provides
support for creating items and collections for 3DEP data hosted
at ftp://rockyftp.cr.usgs.gov/vdelivery/Datasets/Staged/Elevation.
You can still use a different base (e.g.
`stactools.threedep.constants.USGS_FTP_BASE) when creating an item
through the `base` argument.
This uses AWS and boto3 to fetch all ids for the given product.
To maintain architecture consistancy, this commit moves item creation
logic from `Metadata.to_item` to `stac.create_item`. Well actually
the logic really is in `stac.create_item_from_metadata`, because
we basically moved the `to_item` code straight over, but now the
basic api call `stactools.threedep.stac.create_item("metadata.xml")`
will produce a stac Item, matching other subpackages.
@gjoseph92
Copy link

@gadomski FYI there's some info on this buried in the documentation for the resolution argument https://stackstac.readthedocs.io/en/latest/api/main.html, but basically stackstac just wants the same things as stac-vrt (the important fields from the proj extension).

gadomski added 5 commits April 5, 2021 15:55
Previously, if items were created with `None` explicitly passed in
for the base, they just didn't have a base at all, which is not really
what you want. This fixes to use the DEFAULT_BASE.
Adds threedep:region to the item for creating subcatalogs.
This geopackage contains sub-tile information, such as the instrument
used to collect subsets of the tile's data.
@gadomski gadomski merged commit 0c66a85 into stac-utils:master Apr 6, 2021
@gadomski gadomski deleted the feature/threedep branch April 6, 2021 18:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants